They’re called models in AI because they’re trying to represent something else. Just like you might build a model train, the whole point is it’s representing the full-sized one. One of the things you do when you see a model train is compare it to the real thing: look at the fine details, check the scale. Maybe there's even flashing lights to increase the realism.

AI models are no different: in medical LLMs, we’re using a model to try to represent “the practice of medicine” or “the best doctor’s mind.” And so one of the ways you might assess the model is by running some tests to see how well it performs against a long list of standard medical questions that you know the answers to. If it does really well on the test, then perhaps it represents medicine quite well.

The Benchmarks are Not Benchmarking

The problem today in the medical LLM space is that the tests are bad — like, really bad — in numerous ways:

  • they have wrong or outdated answers,
  • inconsistent answers,
  • typos and misspellings, and
  • lots of information that has nothing to do with the practice of medicine.

And so if you build generative AI models for healthcare — and then try to make your model optimized at scoring highly on a faulty test — you end up with a mis-calibrated compass or altimeter, and you end up flying in the wrong direction or even worse: crashing into a mountain. ⛰️

So I’ll give a brief summary of the “state of the art” (or lack thereof) when it comes to these medical test questions (evaluation data sets or benchmarks), and I’ll also include some signs of hope and optimism from Google’s Med-Gemini paper published yesterday where they actually tried to clean up the mess!

🙄I mean srsly who cares

A lot of my passion around this subject came when some AI researchers decided to create a medical LLM leaderboard and I thought, “What an interesting idea to see how these models rank in their quality — I should look at the underlying data to see what’s being measured.” So I downloaded the medical test questions (they’re seemingly open source and freely available online) and was appalled skimming through the questions. As a physician, it was immediately obvious to me how poor the quality was, and it made me lose a ton of faith in some extremely respected and highly resourced companies and individuals who have promoted the hell out of these medical models for PR despite underlying abhorrent benchmarks (calling you out OpenAI, Microsoft, Meta, and Google).

Some people have said, “Graham, you’re over-reacting, this is just how progress works!” To that I would say: outside of Google’s team who redeemed themselves yesterday and acknowledged this issue (and threw resources at addressing it), we’re wasting a ton of energy by seeing how these models perform with bad data, and these models are promoted as tools that will soon be helping doctors, or can interact with patients, or can summarize medical information.

If you’re training them and optimizing them for 2+2=5, that will fundamentally create danger and harm for patients through errors of omission or commission. Medicine will not and cannot stand for that. A quick benchmark summary:

Article content

I’d like to applaud Google for investing the time and energy to improving the MedQA dataset by using actual human doctors to review the questions and throw out the ones that had agreement about low quality. This is expensive, tedious, and slow work but is critical if we want to get anywhere. (I strongly hope that Google releases their improved MedQA data set as a new benchmark.) Here Google describes how they cleaned up the MedQA data set using at least 3 primary care physicians (18 total were used):

Article content

They ended up removing 7.4% of questions due to poor quality/disagreement/vague answer options, which arguably should make the MedQA dataset better. One interesting tidbit that should make you, wise reader, skeptical about if MedQA is actually a good measure of a "doctor's knowledge":

Article content

Over 4 minutes to answer a single multiple choice question???

There are three other issues that are worth pointing out about MedQA:

  • MedQA is based on US Medical Licensing Exam (USMLE) test questions that all medical students take as part of their training. They are generally structured to be "two steps away" from the actual answer. So for example, if a question might ask "the action of morphine," they have to know 1️⃣ what is morphine and then 2️⃣ what does morphine do at a cellular level to get the question correct. (The 2nd step is usually not very important for actual clinical medicine.)
  • Second, Google wisely didn't touch this with a 10 foot pole, but when evaluating "quality" in medicine, it starts to get murky very quickly. There probably never will be a way to say the "right" way to approach many problems in medicine, because there are so many ways to skin a cat. So just the concept of using multiple choice questions to assess if someone is a proficient doctor (or LLM) is probably impossible and the wrong way to ask the question. (Real world testing is much better, as patients don't come to you with 4 options of what could be wrong with them.)
  • Third — to me there seems to be some copyright issues around MedQA ; if you download the entire data set, you can download the plain text of 18 medical textbooks that are most definitely copywritten.

Article content

If you want to see for yourself, here are links to all of them:

Other Thoughts

A massive thanks to everyone I've talked to about this topic in the past few days, especially Maxime Griot, MD who's become my new Belgian colleague, Cyril Zakka, MD who I really trust in the medical LLM space, Karl Swanson who always keeps up to date and lives and breathes AI, and Morgan Cheatham who is extremely thoughtful when it comes to emphasizing and painting the vision of the future we want for healthcare.