Data quality in clinical NLP: what enterprises get wrong

After running clinical annotation programs across 12 hospital systems, the same three mistakes show up almost everywhere — and none of them are about the model.

Healthcare systems come to us after an internal NLP project stalls — usually one built on EHR free-text extraction or an ambient scribe pilot. The model itself is rarely the problem. In program after program, we trace the stall back to one of three data issues.

1. Templates get mistaken for signal

Clinical notes are full of templated boilerplate — standard review-of-systems language, copy-forwarded assessment sections. A generic annotation pass often labels this boilerplate as if it were meaningful clinical content, which quietly teaches a model that repetition equals importance. We now run a templating-detection pass before any clinical annotation begins, so labelers are working on the actual clinical signal, not the scaffolding around it.

2. Negation and uncertainty get flattened

"No evidence of pneumonia" and "pneumonia" look identical to a naive keyword extractor, and surprisingly often to an undertrained annotator too. We require every clinical annotator to pass a negation/uncertainty calibration test before touching a live batch — it's a five-minute check that prevents the single most common labeling error we see in intake-note structuring.

3. Nobody owns inter-annotator disagreement

When two labelers disagree on a clinical field, most programs just take the majority label and move on. We route every disagreement above a threshold to a licensed clinician for adjudication, and — critically — we log why the call was made. That log becomes the seed of the next guideline revision, so the guidelines actually improve over the life of a program instead of staying frozen at kickoff.

These aren't exotic fixes. They're disciplined process, applied by people who understand the domain. That combination is what our annotation & labeling program is built around for healthcare clients specifically.