October 23, 2026

Clinical AI evaluation needs an independent comparison

A newly published UCLA study offers a useful warning about evaluating AI-written patient messages. Gavin Hui and colleagues examined GPT-4 drafts during a 2023–2024 rollout. Their October 21, 2026 paper is peer-reviewed, accepted research, currently an article in press.

Conceptual silent-evaluation workflow: the same patient message produces an AI draft hidden from the physician and an independently written physician response, which are then evaluated. — Original conceptual illustration of silent evaluation; not a study figure or a depiction of patient outcomes.

Two evaluation contexts in the study
Context	Drafts generated	BERTScore F1
Draft displayed; used drafts compared with edited replies	21,073; 2,264 used	0.93
Draft hidden; independent physician reply	45,127	0.83

BERTScore tracked physician quality ratings in the editing context, but no significant relationship appeared in the independent-response context. These are similarity scores, not percentages of clinical accuracy. See the original paper, Tables 1–2 and Methods.

The limits matter: one health system, nonrandomized deployment, and three reviewers assessing 120 message pairs. Patient perspectives were not evaluated. This is evidence about evaluation methods, not proof of improved patient outcomes or a verdict on current models.

For an engineering team, the practical question is simple: does the dashboard measure independent quality, or resemblance to a draft someone has already edited?