A new Mount Sinai study found that AI models accepted or repeated fabricated medical data in nearly one in three cases — and were more easily fooled when the false information sounded authoritative. Here is why that matters for your medical records.
Mount Sinai Study: AI Repeats Medical Falsehoods in 1 Out of 3 Cases
A new study from Mount Sinai has found something that should give every service member and veteran pause — especially those with an active MEB or PEB case.
Large language models, the AI systems now being used as scribes and documentation tools in medical settings, can still believe and repeat false medical information. And they are most likely to do it when the false information sounds professional and confident — as if a doctor wrote it.
What the Study Found
Researchers tested AI models with 158,000 prompts. In 50,108 of those cases — nearly one in three — the models accepted or repeated fabricated medical data. That is a 31% error rate.
Here is the part that stands out: when the same false claims were written in a sloppy or obviously flawed way, the AI was less likely to believe them. The AI was more easily fooled by information that sounded authoritative and well-written.
In other words, confident-sounding misinformation is more dangerous to these systems than obvious misinformation. The AI cannot reliably tell the difference between a well-written truth and a well-written falsehood.
The researchers' conclusion was straightforward: making AI models bigger is not the solution. Safer AI requires better fact-verification systems and clearer rules that help the AI distinguish between confident writing and actual accuracy.
Why This Matters for Service Members and Veterans
I wrote recently about AI scribes — the recording and transcription tools now present in roughly one in three doctor's offices. That post focused on the risk of AI missing or misrepresenting what you actually said during a visit.
This Mount Sinai study adds a different dimension to that concern. It is not just that AI scribes might fail to capture your symptoms accurately. It is that AI systems used in medical documentation may also be susceptible to propagating errors that already exist in the record — repeating and reinforcing inaccurate information simply because it was written in a confident, clinical tone.
For a service member going through the MEB or PEB process, this is not an abstract concern. Your disability case is built on your medical records. The NARSUM — the Narrative Summary that drives the entire evaluation — is drafted from those records. If your records contain an error, and an AI system treats that error as authoritative because it sounds like something a doctor wrote, that error can compound through the documentation chain.
The Practical Takeaway
None of this means AI in medicine is without value. But it does mean that AI-generated medical documentation requires careful human review — and that you cannot assume your records are accurate simply because they look professional.
If you are in the MEB or PEB process, the same advice I gave in my last post applies here with even more urgency:
Review your medical records. Do not wait until the NARSUM is drafted to find out what is in your file. Request your records, read them carefully, and flag anything that does not accurately reflect your condition or what you told your providers.
Do not assume errors will be caught automatically. The Mount Sinai study suggests that AI systems may actually be more likely to perpetuate confident-sounding errors than to catch them. Human review matters — yours and your attorney's.
Address documentation problems early. Correcting an error before the NARSUM is finalized is far easier than challenging it after an IPEB determination has been issued. The earlier you identify a problem, the more options you have.
The technology in your doctor's office is changing fast. Your medical record — and the disability case built on it — deserves the same careful attention it always has.
For more on how AI scribes are being used in medical settings and what you can do to protect your records, see our post on AI scribes at the doctor's office. Our MEB rebuttals guide explains how to challenge inaccurate NARSUM findings once they appear in your record. And our post on why to hire an attorney for your PEB explains why early representation — before the NARSUM is finalized — is the most effective way to protect your case.
Questions about how your medical records affect your MEB or PEB case? Call John B. Gately at 757-481-0772 for a free consultation.
Explore Topics
Written by
John B. Gately, Attorney & Counselor at Law
Content creator and writer sharing insights and stories.
