Research into how AI can help users understand skin conditions

More than half of adults use the Internet for health information, and one-third turn to artificial intelligence (AI). However, access to information does not mean that it is easy to understand or correctly interpreted. In short, the human component of AI for health information remains important to research to help people benefit from better health information.

Specifically, this is important in the space of dermatology (skin, hair, nails; henceforth “skin” for brevity) because people have trouble looking for the right information online related to their skin concern. For instance, you may notice “red dots on legs,” but not have the background knowledge to specifically search for “palpable purpura”.

Over the years, we have built a technical foundation in this area, including developing AI models to inform differential diagnoses, performing validation of model generalization, and releasing datasets like SCIN to help clinicians and researchers. However, the most significant impact can only be realized by supporting the decision-making of people who have skin concerns through providing high-quality information.

To do this right, understanding how humans engage with AI to inform their decisions is critical. Previous studies evaluating non-AI tools have shown that while people might get better at identifying a condition using the internet, they don't necessarily get better at deciding what next steps to take. We need to ensure that as AI tools become available, we carefully study and improve upon the human factors to support people in making better decisions.

With the above in mind, today we share some of our recent and past research on consumer understanding of AI tools for their dermatology-related questions. These include a recent large-scale quantitative paper that demonstrates increased ability to name conditions with AI assistance, as well as some benefits in determining what next steps to take. It also includes an in-depth mixed-methods study addressing how people use these tools on their own skin concerns, and how the understanding they gain compares to that from conversation with doctors.

Evaluating consumer understanding at scale

In “Consumer Understanding of Skin Concerns With an AI-Powered Informational Tool,” published this week in JAMA Dermatology, we investigated how structured AI assistance changes a user's ability to identify a condition and determine their next steps. We showed 2,345 survey participants retrospective, de-identified skin condition cases — complete with images and structured medical history — and asked them to imagine the cases were their own.

Participants were randomized into three groups to research the cases:

• (Negative) control: Participants used existing tools they were familiar with, such as standard text-based web searches.
• AI: Participants used a prototype AI (user interface in figure above) that provided a scrollable carousel of 3 to 7 matching conditions based on an AI model's predictions, complete with textbook images and details about symptoms and treatments.
• “Wizard of Oz” (positive control): Participants used the same AI interface, but the predicted conditions were actually the "ground truth" differentials provided by a panel of dermatologists, mimicking an AI that always matches the ground truth.

We found that AI assistance provided a statistically significant improvement for consumer understanding. When using the AI tool, participants were more willing to attempt to name the condition shown (over 62%) compared to the control group using standard search tools (41%).

More importantly, participants’ condition name guessing accuracy improved dramatically. Accuracy was nearly three times higher in the AI arm (23%) compared to the unassisted control arm (8%). In the "Wizard of Oz" arm, accuracy was about four times higher (36%), but still not near perfect. Having AI "cards" to display matching conditions also imparted significantly higher confidence in their condition guesses, and greater overall satisfaction with their search results and the time spent searching.

The challenge of defining next steps

To avoid being prescriptive, the AI in our study was designed to focus on matching images to possible conditions and relying on the user to interpret what should be done. Our goal was to enable users to search efficiently and not to be prescriptive or diagnostic. In addition, the treatment and information given was written by dermatologists with access to authoritative sources, based purely on the condition name and not tailored to the specific severity of the condition in that case.

Perhaps because of the generality of information provided, deciding on the appropriate medical next steps, such as using a home remedy versus scheduling an urgent clinic visit, remained challenging for users. Our study found that while next-step accuracy increased by a small amount in the "Wizard of Oz" arm (63.5% vs 60% in control), the standard AI arm did not show a statistically significant improvement. Furthermore, participants in the AI arm were slightly more likely to suggest a less urgent next step than a dermatologist would, compared to the control group (30% vs 27%).

This reinforces that simply identifying the condition is not always enough. There is still progress to be made in designing tools that better inform laypeople about the safest and most appropriate next steps.

Engaging communities for richer feedback

While large-scale survey studies are invaluable for understanding general trends, we also recognized the need to understand how people interpret information when it is directly relevant to their own concerns, rather than interpreting pictures of others’ conditions. To get this richer, more nuanced feedback, we sought deep, qualitative insights directly from the communities who stand to benefit most from these tools.

In "Navigating Skin Concerns with AI: A Human-Centered Investigation of a Dermatology App in a Diverse Community," published in the ACM Computer-Human Interaction (CHI) conference last year, we collaborated with the Stanford Healthcare AI Applied Research Team (HEA₃RT) and the Santa Clara Family Health Plan (SCFHP). SCFHP serves members of the surrounding community, many of whom rely on a healthcare safety net, Medi-Cal. Our goal was to study how diverse, consented participants with active skin concerns actually used and reacted to information from a skin AI system in a real-world setting.

Crucially, we wanted to ensure we were building for this community; since the participants spoke four primary languages, the AI application was translated into their respective languages. Volunteers or staff fluent in the respective language were also present to facilitate communication.

In this real-world study, 110 consented participants used the app (and consulted with a clinician immediately after to clarify any concerns). Similar to the survey study above, using the app increased these participants’ ability to name their condition (an increase of 260%, though the correct guess rate was overall low). Participants heavily relied on visual matching of the textbook images to their condition, highlighting the importance of having images from a spectrum of skin tones, condition severities, and body parts to help them “pattern match”.

The clinicians in the study felt the app’s predictions were generally (86%) consistent with their own assessments of the condition. Because the participants could open the app during the clinician consultation, the clinicians were also able to use it as a shared reference point for discussion and facilitate patient-doctor conversation. The clinicians reported the app as a helpful tool 92% of the time.

Conclusions

Our studies above focused on the use of image-based AI to help individuals with diverse backgrounds better understand skin conditions. Key findings for possible improvements include providing more "textbook" examples to guide user understanding and pattern matching, and including actionable information more specific to the actual user query (as opposed to the conditions). Additionally, our research using image similarity based tools support that an image and text (i.e., multimodal) approach to AI-based skin condition information search is preferred by laypersons over using either alone.

When we look at all of these studies collectively, a potential picture of the future of searching for skin condition information emerges. Providing a visual start lowers the barrier to entry, and more personalized AI guidance may help navigate complex medical information. However, building highly effective tools requires continuous, human-centered research to ensure that everyone can effectively interpret this information to help support healthcare journeys.

Acknowledgements

We thank Elyse Bagley, Trevor Crowell, Bhavna Daryani, Huy Doan, Morgan Du, Madison Elliott, Bea Erickson, Mat Fleck, Zoe Gan, Tammi Huynh, Yetunde Ibitoye, Yejin Jeong, Sergio Marquez, Jay Nayar, Kira Nguyen, Trang Nguyen, Javier Perez, Carola Ponce, Uriel Rivera, Sunny Virmani, Renee Wong, and Allan Ysunza for their contributions to the execution of the mixed-methods studies; and Michael Howell, Naama Hammel, Rajeev Rikhye, Abi Jones and Dave Steiner for valuable feedback on the papers.