Make this research yours. Add it to a free WorldbyFlow workbench to run follow-ups, ask questions, and re-check it as events move.

By the Numbers

When AI diagnostic advice was incorrect, physician diagnostic accuracy collapsed to 23.6%–26.1% (vs. 85.3%–92.8% when AI advice was correct) — a same-study, same-clinician swing showing AI correctness, not clinician skill, is the dominant variable in outcomes (ScienceDaily/Johns Hopkins study, reported Nov. 2024).

This compiles sourced benchmark figures comparing AI diagnostic accuracy to human-clinician performance, contrasted against materially lower AI treatment-recommendation accuracy, alongside documented rates of clinician over-reliance (automation bias) when AI outputs are wrong. The gap between diagnosis and treatment-planning accuracy is the load-bearing finding: AI systems cluster near or above physician-level accuracy on discrete diagnostic tasks but degrade substantially on multi-step treatment recommendations, while clinicians exposed to incorrect AI advice show measurable, replicated drops in their own accuracy.

The accuracy picture for AI in medicine bifurcates sharply between diagnosis and treatment recommendation, and the split is consistent across multiple independent study designs. On discrete diagnostic tasks, general-purpose large language models now perform at or above published human benchmarks in several settings: GPT-4V reached 88.7% accuracy vs. 77.8% for human readers on NEJM Image Challenge cases, and 73.3% vs. 63.6% on JAMA cases, with GPT-4V also outperforming physicians on the subset of cases physicians answered incorrectly, though the same study flagged that GPT-4V's stated diagnostic rationale was flawed in a notable share of even its correct answers. A separate NEJM AI analysis found GPT-4 correctly diagnosed 57% of cases, outperforming 99.98% of simulated human readers generated from online answers, though the authors themselves cautioned this used a poorly characterized comparator population and likely represents a best-case read for the model. A real-world, non-benchmark comparison at a Cedars-Sinai virtual urgent-care clinic found AI recommendations were rated as optimal more often (77%) than those of physicians (67%) and were less frequently potentially harmful, though the study's authors noted a limitation: they could not measure how much physicians actually relied on the AI output in that setting.

Treatment recommendation accuracy is consistently lower and more brittle than diagnostic accuracy within the same models and same studies — the central asymmetry this brief is built to surface. In a fundus-imaging study cited within a neuroradiology accuracy paper, GPT-4 achieved 55.8% accuracy on a diagnosis task using fundus images and 42.7% on a treatment recommendation task — a roughly 13-point same-model, same-study drop moving from diagnosis to treatment planning. The Doctorina MedBench evaluation of agent-based medical AI found a similar directional pattern but with an added complication: AI Doctor reached a Diagnosis Accuracy of 89.3%, surpassing the 84.6% achieved by GPT-5 and Treatment Accuracy at 53.0%, compared to 38.0% for the baseline — meaning treatment accuracy for the better-performing system (53.0%) was still roughly 36 points below its own diagnostic accuracy (89.3%). Neuroradiology-specific work found comparable degradation: GPT-4 Turbo achieved a baseline diagnostic accuracy of 55.1% on primary imaging diagnosis, in a domain where the same source describes current misdiagnosis rates ranging from 30–50% for LLM-assisted imaging generally — a range wide enough that it should be read as a documented span across studies, not a single point estimate. A dental-medicine exploratory study similarly found AI-generated treatment recommendations aligning with the evidence base at up to 75% agreement, with the best-performing tool (GPT-4) reaching a Cohen's kappa of 0.42 against clinical guidelines — a moderate, not strong, agreement level by standard interrater-reliability thresholds.

The second load-bearing figure set concerns clinician deference to incorrect AI output — automation bias — which is now documented across multiple independent clinical domains with converging but not identical magnitudes. In pathology, a study found in 7 out of every 100 cases where a pathologist had initially reached the correct conclusion, the introduction of an erroneous AI recommendation caused them to abandon that correct answer, and the source explicitly frames this as a patient-safety metric, not statistical noise, given the stakes of missed cancer staging. A related internal-medicine study spanning residents, faculty, and students found the study, encompassing 216 physicians... showed improved diagnostic accuracy, with a more significant increase observed for the students than for senior residents and faculty. However, the study noted that in 6% of cases, clinicians changed their own correct diagnosis in favor of the inaccurate recommendation from the DSS. A wound-maceration simulation with 223 physicians and nurses generating 1,338 decisions found that correct AI assistance proved to be the most influential predictor, with participants having tenfold higher odds of making the correct diagnostic decision when receiving a correct recommendation (OR = 10) — an odds ratio, not a probability, meaning the underlying baseline accuracy rate still governs the absolute risk. In musculoskeletal MRI reading, one laboratory study found that 45.5% of the total mistakes made by clinicians in the AI-assisted round were due to following incorrect AI recommendations, and the same review noted this effect held across all levels of clinician seniority. A 2026 randomized trial registered as NCT06963957 tested whether AI training itself protects against this bias, and found it does not eliminate it: physicians exposed to flawed LLM advice scored worse on overall diagnostic reasoning: 73.3% versus 84.9% in the error-free group, with an adjusted difference of -14.0 percentage points, and top-choice diagnosis accuracy also dropped, from 90.5% in the control group to 76.1% in the flawed-advice group, an adjusted difference of -18.3 percentage points. The consistent finding across pathology, radiology, internal medicine, and now AI-literacy-trained physicians is that formal training reduces but does not eliminate deference to wrong AI output — a structural finding rather than a single-study artifact, replicated across at least five independent research groups and clinical domains cited here.

The regulatory backdrop for interpreting these accuracy figures is itself quantifiable and dated. As of early 2026, the FDA had authorized over 1,350 AI-enabled devices (about double the number in 2022), with the FDA database [showing] a total of 1451 devices (up from 1250 last year)... As of March, 2026, no device has been authorized that uses generative AI or is powered by large language models. This is a critical scope caveat for the entire benchmark literature above: nearly all of the GPT-4/GPT-5/LLM diagnostic and treatment-accuracy figures cited in this brief describe research or investigational use, not any FDA-cleared clinical product — the cleared device base remains dominated by narrower, task-specific machine learning (imaging triage, detection, measurement), concentrated in radiology, which accounted for 74.4% of 2024's newly authorized Class II ML devices. Separately, transparency in the cleared-device evidence base is thin: among 2024 authorizations, only 49 devices (29.2%) reported both sensitivity and specificity; 15.5% provided demographic data — meaning that for roughly seven in ten newly cleared AI devices in 2024, the FDA summary itself does not disclose the sensitivity/specificity figures a clinician would need to calibrate trust in the tool's diagnostic output.

Two structural caveats bound all of the above. First, nearly every diagnostic-accuracy figure here comes from retrospective benchmark testing against curated case sets (NEJM/JAMA quiz archives, radiology report databases, simulated vignettes) rather than prospective, real-world clinical deployment with patient outcomes — these are test-set accuracy rates, not validated clinical-outcome improvements, and Phase II-style benchmark performance historically overstates real-world performance. Second, the automation-bias figures (6%, 7%, 45.5%, the 14–18 percentage-point degradations) come from distinct study designs, patient populations, and AI systems (CNNs, DSSs, and LLMs are not interchangeable failure modes) — they should be read as a consistent directional finding across the literature, not as a single unified error rate.

The claims behind this analysis, each with its verification status — including what is contested, unverified, or could not be established.

August 13, 2026

August 13, 2026

August 13, 2026