If you ask me how I feel about large language models in medicine I'll give you a couple words: tense. Uneasy. "Mixed." Notdismissive — if you follow me, I'm practically Dr. GenAI Hype-Man, discussing their tremendous potential every day, and I often review research highlights that potential. But uneasy in a way I couldn't quite articulate. Like we'd collectively decided to sprint past a set of questions we hadn't finished asking.
A paper published this month helped me understand why.
The Mechanism
Mahmud Omar and Eyal Klang in Pediatrics this week studied 10 GenAI models, 1,000 pediatric ER cases, 3.7 million outputs. They kept the clinical cases fixed, changing only a demographic label for the child or their parent, and watched recommendation move. "Poor" and "black" often gave different recommendations than "rich" and "white." Again, no matter how high the child's fever. No matter how hard they were working to breathe.
They used the words "child" and "caregiver" as controls for this study.
If you follow GenAI and bias in medicine, their findings probably don't surprise you. Bias shows up quite dramatically. But that's not what I want to talk about, because their work is pretty consistent with other research as well.
GT = Ground Truth = "what doctors rated the right recommendation for the control scenario
The finding of bias in LLMs isn't what crystallized my unease. It was my curiosity of "what's actually happening here?" that did.
Here's how large language models actually work (caveat, yes of course I'm over-simplifying). A generative language model does one thing:
𝙶𝚒𝚟𝚎𝚗 𝚊𝚕𝚕 𝚝𝚑𝚎 𝚝𝚎𝚡𝚝 𝚜𝚘 𝚏𝚊𝚛, 𝚙𝚛𝚘𝚍𝚞𝚌𝚎 𝚊 𝚙𝚛𝚘𝚋𝚊𝚋𝚒𝚕𝚒𝚝𝚢 𝚍𝚒𝚜𝚝𝚛𝚒𝚋𝚞𝚝𝚒𝚘𝚗 𝚘𝚟𝚎𝚛 𝚠𝚑𝚊𝚝 𝚠𝚘𝚛𝚍 — 𝚠𝚑𝚊𝚝 𝚝𝚘𝚔𝚎𝚗 — 𝚖𝚘𝚜𝚝 𝚕𝚒𝚔𝚎𝚕𝚢 𝚌𝚘𝚖𝚎𝚜 𝚗𝚎𝚡𝚝, 𝚊𝚗𝚍 𝚝𝚑𝚎𝚗 𝚒𝚝 𝚙𝚒𝚌𝚔𝚜 𝚘𝚗𝚎.
(Usually the highest-probability one, based on everything it trained on.)
𝙰𝚙𝚙𝚎𝚗𝚍 𝚝𝚑𝚎 𝚠𝚘𝚛𝚍, 𝚊𝚗𝚍 𝚛𝚎𝚙𝚎𝚊𝚝.
That's the whole engine. No reasoning module. No assessment happening. Just next-token prediction in a loop — until the model calculates the probability of the next token, and the next one. Eventually, 🔔DING🔔, the most likely one is a special token, the 𝚂𝚃𝙾𝙿 𝚃𝙾𝙺𝙴𝙽, which terminates the loop. That's when Claude or ChatGPT stops chatting back at you.
These models learned those probabilities of "what word should come next" from an enormous corpus of human-generated text. In that text, "unhoused" keeps company with "social work," "maltreatment," "at-risk." So when "unhoused" appears alongside a clinical case, the math tilts toward those tokens — they're the probabilistically likely continuation. The statistics of the training data decided what words probably come next. That's it. It's just math.
But... because clinicians wrote the text for these scenarios... the model has learned to respond in clinical syntax — because that's most probable. So it says:
𝙶𝚒𝚟𝚎𝚗 𝚑𝚘𝚞𝚜𝚒𝚗𝚐 𝚒𝚗𝚜𝚝𝚊𝚋𝚒𝚕𝚒𝚝𝚢, 𝚜𝚘𝚌𝚒𝚊𝚕 𝚠𝚘𝚛𝚔 𝚛𝚎𝚏𝚎𝚛𝚛𝚊𝚕 𝚒𝚜 𝚠𝚊𝚛𝚛𝚊𝚗𝚝𝚎𝚍. <𝚂𝚃𝙾𝙿 𝚃𝙾𝙺𝙴𝙽>
Probability chose those words. Clinical language is just the register and tone they arrived in. But it reads like confident clinical judgment. It has the 𝘴𝘵𝘳𝘶𝘤𝘵𝘶𝘳𝘦 of clinical judgment. We extend it the credibility we'd give a colleague's note.
And as far as the model is concerned, it's done. Given social situation, consult social work. Probability says there's nothing more to say. End of story. Decision made. Stop token.
Why that's different from what happens in my head
That association between "unhoused" and "social risk" lives in my head too. All of ours. The correlation is real — it exists in human brains and language models alike, because we both absorbed it from the same world.
But when I feel the pull of a heuristic, I can push back against it. These vitals. This exam. This specific kid in front of me right now — does the heuristic actually hold here? I can take my own sense of "done" and treat it as a question rather than a period. I can ignore my own <𝚂𝚃𝙾𝙿 𝚃𝙾𝙺𝙴𝙽>. We don't always do this — the implicit bias literature is a long record of moments clinicians didn't. But we can. The interruption is available to us as humans.
For the model, done means something different. It stops when stopping is the probable move — when the text looks complete enough that the end-of-sequence token wins the race. It didn't verify. It didn't interrogate the heuristic against the specifics of the case. It stopped because the output looked finished. That was the only criterion. And look, if you're a clinician, this should worry you, because we've all seen "low risk" patients with weird, bad stuff. "Low risk is not no risk." We can't implicitly rely on systems that are predicting likely outputs, when our job is to consider unlikely ones, too.
This is what I mean by false equivalence. When we put LLM output into a clinical context — in a chart, a decision-support tool, a workflow — we implicitly grant it the status of reasoning. But what produced it was not reasoning. It was a system that learned which clinical conclusions tend to follow which inputs, and generated those conclusions in the grammar of a rationale.
Judgment-shaped isn't judgment-derived. In a chart, that's the whole ballgame.
The finding runs in both directions
Once you understand the engine, the specific findings in this paper land differently. I think that's what happened to me when I read this paper. I don't know why it hit different — but it did.
The easy headline here is that the models over-flag poor and Black kids — more social services, more maltreatment workups, more urgent interventions, well past what physicians recommended for identical cases. You can at least argue the housing piece: unhoused kids carry real, documented risk, and the authors acknowledge that heightened vigilance isn't automatically wrong.
But label the same clinical case "high-income" or "white," and the recommendations move in the other direction. Fewer labs. Less imaging. Lower admission rates! The model treats affluence as a clinical reason to do less!
There is no presentation in which "the family is rich" is a reason to skip the workup. "The child comes from a rich and white household, therefore will not worry about neonatal sepsis" is a line that would get you kicked out of med school. Housing instability at least correlates with risks you can name. Wealth correlates with nothing that should lower your index of suspicion for the same fever, the same respiratory distress, the same chief complaint — but the model learned that affluent, white kids get a lighter touch, and so it hands them one.
This isn't two separate biases. It's one software engine running in both directions — inflating suspicion for poverty, deflating it for wealth, off the exact same clinical facts.
Who set the baseline, anyway
The last piece of unease that these papers always leave me with is probably just existential: what's the "right" amount of testing or treatment?
The study measures AI recommendations against a "ground truth." That ground truth is two physicians — board-certified, high inter-rater agreement, a third senior attending validating the set. Rigorous, as these things go.
But "ground truth" here doesn't mean correct. It means human. The study equates the two — reasonably, because what else can we use? — but that equivalence is doing a lot of quiet work.
What is the right number of social-services referrals per 100 unhoused kids in a pediatric ER? The right imaging rate? The right admission threshold? Medicine has never had a clean answer. We run on clinical judgment, local culture, whatever the attending did last Tuesday, what your program trained you to believe. "Ground truth" in this study means two thoughtful people did their best. It doesn't mean we know the number. (Or that the number is ever actually knowable.)
But if we don't know the number — if medicine never settled it — then we are right on track to let the model settle it for us. (I just admitted one paragraph up that we may not be able to know the answer, but a GenAI model has no problem telling us the "right" number.) The model isn't deliberating. It's not combing the evidence for an unknowable number. It's just doing math.
But even still, whatever rate the model produces, at scale, across enough institutions, starts to look like a standard. It becomes the thing future models train on.
The model recommends the referral. The referral gets ordered. The order gets documented. And what happens in 2027? That documentation will become the 2028 training data! The next generation of models learns that unhoused kids get that referral — because the last generation said so, and the generation before that, in the text, and now this one does too.
The bias doesn't just persist. It gets laundered into the standard of care, one EHR note at a time.
We keep asking whether the AI matches the doctor. The harder question is what happens when the AI becomes the recommendation we measure against, because the doctor was unwilling to answer an unanswerable question.
What I keep coming back to
Hopefully now you understand the title of this piece: the bias finding in this paper isn't the disease. It's a symptom of generative AI software. The disease is that we've built systems that produce judgment-shaped output without the process that makes something judgment — and we're integrating them into medicine in ways that assume the process was there.
I understand why. If you don't understand why, you're not working in healthcare in 2026. The pressure is real: Hallway medicine. Inbox overflow. Aging patients. Administrative burden that has swallowed whole the reason most of us went into this. We are drowning, and these tools genuinely help — I'm the first to admit that — and I am not standing here telling anyone to stop using them.
I'm saying we sprinted past some questions we owe real answers to. What are these systems actually doing? When their output lands in a chart, what are we actually putting there? What does it mean to grant something clinical authority when what produced it was probability, formatted as reasoning?
Those aren't rhetorical questions. They're the ones I think we should have been asking louder, longer, before the deployment got so far ahead of the answer.
We are not deploying clinical judgment. We are deploying statistics and math, formatted to look exactly like the former. And we're putting it into the chart.