Noticing is not applying
A new paper on arXiv argues that language models reliably notice when a clinical guideline has been revised and then fail to act on the revision. Min Zeng, Zaifu Zhan, Kai Yu and Rui Zhang posted it on 22 October.
The benchmark behind it is called ChronoEvo. It holds 1,709 recommendation-level revisions, covering twelve types of change to medical knowledge — the ways guidance actually moves, rather than a general test of medical recall. The question it puts to a model is not whether it knows the guideline, but whether its answer reflects the current version.
In what the paper calls the primary model, 90.8% of responses recognised that the recommendation had been revised. 80.2% were clinically correct. Only 58.5% fully incorporated the revised guidance. That is a 32.3-percentage-point gap between seeing the change and acting on it. The paper does not name the primary model, and names none of the fourteen it ran. The same dissociation, it says, showed up across all of them.
What clinicians found
The automated scoring was checked against human judgement. Two clinicians adjudicated 300 responses, and in that sample the recognition–incorporation gap came out at 40.3 percentage points — wider than the benchmark’s own measure, not narrower.
That direction matters. A benchmark that flatters the models it scores is a familiar problem. This one appears to be doing the opposite, which makes the headline number a floor rather than a ceiling.
Which changes break it
The paper identifies the kinds of revision the models handle worst. Changes in the strength of a recommendation — a treatment moving from “should” to “may be considered” — and changes in which patients are eligible for it were both associated with models retaining the superseded advice. Those are the revisions that leave the underlying intervention in place and alter the conditions around it, which is also the kind of change a reader skimming for the drug name would miss.
The intervention that half worked
The authors tried making the revision more prominent in what the model was given. It reduced outdated content and improved clinical correctness. It did not significantly improve full incorporation.
That is the most useful finding in the paper for anyone deploying this. The cheap fix — surface the update, flag the change, put the new text first — moves the metrics that are easy to move. It does not close the gap that matters, which is whether the answer a clinician reads is built on the current recommendation or the one it replaced.
What to watch
Retrieval-augmented systems are the standard answer to stale medical knowledge, and this result does not disprove that approach so much as describe its failure mode: the current document is retrieved, the model acknowledges it, and the answer still reflects what the model learned in training.
The test worth running next is the same benchmark against production clinical tools rather than base models, and with the model names published. Fourteen anonymous models make a research point. Named ones would make a procurement decision.
