Overview
MIT researchers and collaborators ran parallel experiments testing explainable AI assistance for dermatological diagnosis on lay users and primary care physicians, finding that non-experts improved accuracy but did so by deferring to the model even when it was wrong, while clinicians resisted incorrect AI explanations and performed best with a bare prediction and no explanation at all. The study, published in Nature Medicine, demonstrates that AI assistance design cannot be one-size-fits-all across user expertise levels.
Brief
Artificial intelligence is moving into dermatology from two directions at once: as a clinical decision-support tool inside physician workflows and as a consumer-facing feature inside AI search products that patients query directly about a worrying mole. Both pathways rest on an unexamined assumption — that explaining an AI system's reasoning, through a heat map, a visual-similarity display, or a plain-language write-up from a large language model, makes the human using it better at the underlying task. A team led by MIT associate professor Marzyeh Ghassemi set out to test that assumption directly, rather than take it on faith.
The design split participants into two populations doing two different tasks. Lay users were asked to make a binary call on a photo of a skin mole: cancerous or not. Primary care physicians were given a harder job, producing a full differential diagnosis of a dermatological condition rather than a yes-or-no screen. Each group was randomized across four explanation conditions — a bare prediction with confidence score, a GradCAM heat map, a content-based visual-similarity retrieval, or a multimodal LLM-generated text explanation — and the researchers deliberately fed both correct and incorrect AI outputs into the mix so they could isolate what happened when the model was wrong, not just when it was right. The underlying model itself was trained with a fairness constraint aimed at closing known performance gaps on darker skin tones, since dermatology AI has a well-documented history of underperforming on Fitzpatrick IV-VI skin.
The results split cleanly along expertise lines. Every explanation method improved lay users' accuracy overall, and the fairness-constrained model measurably reduced skin-tone-based diagnostic disparities. But the mechanism behind the lay-user improvement was not better reasoning — it was deference. Non-experts leaned on the model's answer, and when that answer was correct, their accuracy rose; when it was wrong, their accuracy fell further than the gain from a correct call, and they grew more confident, not less, when an LLM had supplied the wrong explanation in plain language. Clinicians ran the opposite pattern. Physicians weren't fooled by incorrect AI-generated explanations, and their diagnostic accuracy peaked when they received only the bare model prediction, with no explanatory narrative attached at all — the explanation methods that helped lay users the most helped clinicians the least.
The study's own framing places this inside a broader concern about automation bias: the tendency for an explanation, especially a fluent, natural-language one from an LLM, to function less as a check on the model and more as a script that talks the human out of their own judgment. That risk compounds as AI-powered search and consumer health apps put diagnostic-flavored AI output directly in front of patients with no clinical training to catch an error, a distribution channel the researchers explicitly note is expanding.
The practical resolution the researchers point toward is not to abandon explainability but to stop treating it as a universal good. For a novice screening their own mole, a system that visibly reduces skin-tone bias and nudges toward correct answers has real value, provided the deference it produces is closely monitored, because a wrong call in that setting compounds already-known disparities in melanoma outcomes for underdiagnosed populations. For a primary care physician doing differential diagnosis, the same LLM narrative that helps a layperson can crowd out the physician's own signal detection, meaning the safer default in that setting is a plainer prediction the clinician can independently interrogate rather than a persuasive narrative they have to consciously resist.
Key Actors (3)
Marzyeh Ghassemi
MIT associate professor in EECS and principal investigator at the Abdul Latif Jameel Clinic for Machine Learning in Health who led the study and its interpretation of the automation-bias findings.
623 lay participants
Non-expert study cohort tasked with binary cancerous/non-cancerous mole classification, whose accuracy gains from AI assistance were driven by deference to the model rather than independent judgment.
153 primary care physicians
Clinician cohort tasked with full differential diagnosis, who resisted incorrect AI explanations and performed best with a bare prediction and no explanatory narrative.
What Happened (5)
1Researchers built a fairness-constrained dermatology AI model
The team trained a diagnostic model specifically constrained to reduce known performance disparities across skin tones, addressing a documented weakness in prior dermatology AI systems on darker Fitzpatrick skin types.
Marzyeh GhassemiMIT research team
2Two populations were randomized across four explanation types
623 lay people and 153 primary care physicians were each randomly assigned to receive a bare prediction, a GradCAM heat map, a visual-similarity retrieval, or an LLM-generated text explanation, with both correct and incorrect AI outputs deliberately included.
623 lay participants153 primary care physicians
3Non-experts improved accuracy but mainly by deferring
All explanation methods raised lay-user accuracy, largely by helping them correctly rule out non-cancerous moles, but the improvement traced to reliance on the model's answer rather than better independent reasoning.
623 lay participants
4Wrong AI answers hurt non-experts more than right answers helped
When the AI's prediction was incorrect, non-expert accuracy dropped by more than it had gained from correct predictions, and the effect was strongest with LLM explanations, where wrong answers were held with higher confidence.
623 lay participants
5Clinicians resisted incorrect explanations and did best unaided
Primary care physicians were not misled by incorrect AI-generated explanations and reached their highest diagnostic accuracy when given only the bare model prediction with no explanation attached.
153 primary care physicians
Lessons (3)
Accuracy gains from AI assistance can be a byproduct of deference rather than of improved reasoning, and the two are only distinguishable by testing incorrect AI outputs, not just correct ones.
Why it transfers: Any human-AI accuracy study that only measures performance when the AI is right will systematically overstate the tool's value; the true test is what happens to decision quality when the AI is wrong, which applies well beyond dermatology to any assistive-AI deployment in medicine, finance, or law.
The explanation format that most increases user confidence, in this case LLM-generated natural-language narratives, is not the same as the format that most increases user accuracy, and the gap between the two widens for less expert users.
Why it transfers: Persuasive, fluent explanations can function as a script that overrides independent judgment rather than a check on it; this generalizes to any setting where an LLM narrates its reasoning to a non-expert, but the finding rests on one study population and task type and has not yet been replicated across other clinical domains.
AI assistance design should be matched to the user's expertise level rather than deployed as a single uniform interface across both expert and novice audiences.
Why it transfers: The same product decision, whether to show an explanation and in what form, has opposite effects depending on who is using it, which argues for expertise-aware interface design in any dual-audience AI deployment; the limit is that this study tested primary care physicians specifically, not specialists like dermatologists, so the clinician-side finding may not hold uniformly across all levels of medical expertise.
Facts & Figures (6)
The claims behind this analysis, each with its verification status — including what is contested, unverified, or could not be established.
The study tested 623 lay people and 153 primary care physicians in two large-scale randomized experiments using a fairness-constrained dermatological AI model paired with four explanation types.
Establishes the actual sample sizes and design, distinguishing this from a small pilot or single-arm study.
✓ GROUNDED
Lay users were tasked with a binary cancerous/non-cancerous call on mole images, while primary care physicians were given the harder task of producing a full differential diagnosis of dermatological disease.
The two groups did not perform the same task, so their accuracy gains are not directly comparable in magnitude, only in direction and mechanism.
✓ GROUNDED
All four explainability methods improved non-expert accuracy, primarily because the tools helped users correctly identify non-cancerous moles, and the fairness-constrained model reduced diagnostic disparities tied to skin tone.
Shows the benefit for non-experts was real but concentrated on the easier side of the diagnostic task (ruling out cancer), not broadly across all lesion types.
✓ GROUNDED
The deference effect was largest with LLM-generated text explanations, and non-expert users reported higher confidence in wrong answers when an LLM explanation was involved.
Identifies the specific explanation modality that most amplifies automation bias, which is also the modality most likely to reach patients via consumer AI search tools.
✓ GROUNDED
Clinicians were not misled by incorrect AI explanations and achieved their best diagnostic performance when given only the bare AI prediction with no accompanying explanation.
This is the direct inverse of the non-expert pattern and is the core evidence for the study's central claim that assistance design must differ by user expertise.
✓ GROUNDED
The study, titled 'Divergent impacts of explainable AI for dermatological diagnosis on clinicians versus lay people,' was published in Nature Medicine and was funded in part by the National Science Foundation, Schmidt Sciences, the National Bureau of Economic Research, and Columbia University.
Confirms the peer-reviewed publication venue and funding sources, establishing this as published research rather than a preliminary press release.
✓ GROUNDED