Four signs an AI health answer is actually useful (and three that mean it isn't)

Why I wrote this

About 40 million people open ChatGPT every day with a health question (NPR, March 2026). Roughly 1 in 3 US adults now use a chatbot for medical advice (Harvard Gazette, May 2026). For people who are uninsured, live far from a clinic, or cannot get a quick appointment, that makes sense. It is often free. It is always open.

But the same tools give wrong or misleading answers more often than most people realize. This guide helps you tell the difference between an answer worth using and one to throw out.

What "AI health advice" means in this guide

By AI I mean a general chatbot you can talk to in writing: ChatGPT, Gemini, Claude, Microsoft Copilot, Perplexity, Pi. These are not doctors. They are not apps cleared by the FDA. They are computer programs that guess the next word based on patterns in billions of pages of text.

They sometimes get it right. They sometimes get it confidently wrong. The hard part is that the wrong answers look just as smooth as the right ones.

What the research actually shows

A 2026 Oxford study tested chatbots against real medical scenarios. The bots themselves answered correctly about 95% of the time when researchers gave them clean information. But when 1,300 regular people used those same bots, only about 35% got the right answer, and only about 43% picked the right next step like "go to the ER" versus "stay home" (BBC, April 2026).

In emergencies, AI tools under-triaged 52% of the time, meaning they treated something serious like it was minor (Mount Sinai study, via NPR).

Across 22 ChatGPT versions tested in Nature, the best one was right about 74% of the time, and newer versions did not get more accurate (Nature Communications Medicine, February 2026).

The gap is not the chatbot's knowledge. The gap is what happens when a real person, with partial information and worry, asks a real question.

The 4 signs an AI health answer is actually useful

A useful AI health answer does all four of these. If even one is missing, slow down before you act on it.

1. It names real sources you can click and read. A good answer cites a real journal, the FDA, the CDC, or a known professional society, and the links actually work. AI tools sometimes invent citations that sound real but do not exist (one study found about a third of ChatGPT's medical references were fabricated, PsyPost, April 2024). Click at least one source before you trust the answer.
2. It asks you follow-up questions before recommending anything. A safe tool will ask about your age, your medications, when the symptom started, and what makes it better or worse. If it gives you advice without asking, the advice does not actually fit you.
3. It gives a range, not a single certain answer. A useful AI response says, "this could be A, B, or C; here is how to tell them apart; here is when to be worried." That kind of answer respects how medicine really works.
4. It tells you when to call a real person, with a clear threshold. Look for specifics: "if your fever is above 102 for more than 3 days," "any chest pain or shortness of breath," "if you cannot keep fluids down." A useful AI answer always ends with a line that sends you to a human if things change.

If you get all four, the answer is at least a useful starting point. It still does not replace your doctor.

The 3 signs you should throw the answer out

Stop using an AI health answer if any of these are true:

1. It is too confident. Phrases like "you definitely have," "this is certainly," or "there is no reason to worry" are red flags. Medicine is rarely that certain, especially without an exam.
2. It tells you to skip the doctor, skip the ER, or stop a prescribed medication. No general-purpose AI should override a medical referral or change your treatment plan. If it does, close the app.
3. It will not show its work. Ask, "Where did this come from?" or "What is the strongest argument against this answer?" If the tool refuses, dodges, or makes up sources, the answer is not safe to act on.

How to ask a better question

The Oxford study found the wording of your question changes the answer a lot. Two people described the same situation. One said "the worst headache I've ever had." The bot told them to go to the ER. The other person did not say "worst." The bot told them to take aspirin and rest. It turned out to be life-threatening (NPR, March 2026).

Give the AI what a doctor would ask for:

• Your age, sex, and any conditions you have (diabetes, high blood pressure, pregnancy)
• All medications you take, including over-the-counter and supplements
• When the symptom started and how it has changed
• What makes it better or worse
• Anything new (a new med, a new injury, a recent trip, a recent illness in your family)

Then ask the AI to list possible causes, the most serious one to rule out, and what would tell you to seek care now. That phrasing tends to get safer answers.

A simple test before you act

Step	What to do
1. Verify	Pick the most important claim. Search it on a trusted site: medlineplus.gov, cdc.gov, mayoclinic.org, or healthfinder.gov. If you cannot find it on any of these, do not act on it.
2. Reverse	Ask the same chatbot, "What might be wrong with that answer? What is the strongest argument against it?" If it cannot answer, the original answer is shakier than it looked.
3. Run it by a person	Call your doctor's office, a nurse line, or a pharmacist before changing any medication, starting a new supplement, or skipping urgent care. Most pharmacies will answer simple drug questions for free.

When to stop using AI and call a real person

Stop and call your doctor, a nurse line, or 911 if:

• You have chest pain, shortness of breath, sudden weakness on one side, slurred speech, or the worst headache of your life
• You are bleeding that does not stop, or are dehydrated and cannot keep fluids down
• A child has a high fever, will not eat or drink, or is hard to wake up
• You are pregnant and have new symptoms
• Your mental health is getting worse, or you are thinking of hurting yourself or someone else (call or text 988)
• A medication is doing something unexpected
• You just feel something is very wrong. Trust that.

You do not need insurance to call 988 or 911. You do not have to give your name.

If you do not have a doctor

• Find a Federally Qualified Health Center near you at findahealthcenter.hrsa.gov. Many use sliding-scale fees.
• Use a free nurse line. Some health departments and hospital systems offer one 24/7. Search "[your county] nurse line."
• Call your insurance (if you have it) and ask for the nurse advice line on the back of your card.
• Free clinics are listed at freeclinics.com.
• For prescriptions you cannot afford, try needymeds.org or your pharmacy's discount programs.

What to tell your doctor about AI use

If you are using AI for health questions, tell your doctor. Bring:

• The name of the tool you are using (ChatGPT, Gemini, etc.)
• What you asked it and what it said
• Any decisions you made because of it (starting a supplement, skipping a medication, delaying care)

Your doctor will not judge you. They want to know because an AI suggestion may interact with your actual treatment.

What this guide is, and is not

This is education. It is not medical advice. It does not create a doctor-patient relationship.

AI tools can be useful, especially if a doctor visit is hard to get. They are not a substitute for one. If something in this guide does not match what your doctor or therapist tells you, trust them. They know you. I do not.

You can share this guide. You can print it for your clinic. Please do not change it or sell it.

Dr. Maia