Class imbalance is a common problem in medical diagnosis, causing a standard classifier to be biased towards majority classes and ignore the importance of the rest. This is especially true for dermatology, a specialty with thousands of skin conditions but many of which rarely occur in the wild. Buoyed by recent advances, we explore meta-learning based few-shot learning approaches in skin condition recognition problem and propose an evaluation setup to fairly assess the real-world impact of such approaches. When compared to conventional class imbalance techniques, we find that the state-of-the-art few-shot learning methods are not as performant, but combining the two approaches using a novel ensemble leads to improvement in all-way classification, especially the rare classes. We conclude that the ensemble can be useful to address the class imbalance problem, yet progress here can further be accelerated by the use of real-world evaluation setups for benchmarking new methods.