Dr. Sina Bali, a practicing surgeon and AI healthcare leader at data company iMerit, has seen firsthand how ChatGPT can mislead patients with incorrect medical advice.
“I recently had a patient come in and recommend a medication, and a dialog printed from ChatGPT said that this medication had a 45% chance of pulmonary embolism,” Dr. Bari told TechCrunch.
Upon further investigation, Dr. Bari discovered that the statistics were from a paper about the drug’s effects in a niche subgroup of tuberculosis patients and did not apply to his patients.
Still, when OpenAI announced its purpose-built ChatGPT Health chatbot last week, Dr. Bari felt more excited than concerned.
ChatGPT Health, rolling out in the coming weeks, will allow users to have conversations with chatbots about their health in a more private setting, and their messages won’t be used as training data for the underlying AI models.
“I think it’s great,” Dr. Bari says. “This is already happening, so we need to formalize it to protect patient information and put safeguards in place.” […] There will be even more powerful features available to patients. ”
Users can get more personalized guidance from ChatGPT Health by uploading their medical records and syncing them with apps like Apple Health and MyFitnessPal. For anyone who cares about security, this immediately raises red flags.
tech crunch event
san francisco
|
October 13-15, 2026
“Suddenly, health data is being transferred from HIPAA-compliant organizations to non-HIPAA-compliant vendors,” Itai Schwartz, co-founder of data loss prevention company MIND, told TechCrunch. “So I’m interested to see how regulators approach this.”
But in the view of some industry experts, the cat is already out of the bag. Instead of searching Google for cold symptoms, people are now talking to AI chatbots. Over 230 million people already consult ChatGPT about their health every week.
“This was one of the biggest use cases for ChatGPT,” Andrew Brackin, a partner at health technology investor Gradient, told TechCrunch. “So it makes a lot of sense that they would want to build a more private, secure and optimized version of ChatGPT for medical questions.”
AI chatbots have a persistent problem of hallucinations, which is a particularly sensitive issue in the medical field. According to Vectara’s fact-consistency rating model, OpenAI’s GPT-5 is more prone to hallucinations than many Google and Anthropic models. But AI companies see the potential to fix inefficiencies in healthcare (Anthropic also announced a healthcare product this week).
For Dr. Nigam Shah, a professor of medicine at Stanford University and chief data scientist at Stanford Health Care, the threat of ChatGPT providing inappropriate advice is more urgent than the threat of American patients receiving treatment.
“Right now, at any medical facility you go to, you’re going to want to see your primary care physician. The wait time is going to be three to six months,” Dr. Shah said. “If you had a choice between waiting six months for a real doctor or seeing someone who is not a doctor but can do things for you, which would you choose?”
Dr. Shah believes a clearer path to bringing AI into the healthcare system lies with healthcare providers rather than patients.
Medical journals frequently report that about half of a primary care physician’s time can be spent on administrative tasks, thereby significantly reducing the number of patients he or she can see in a day. If this kind of work could be automated, doctors would be able to see more patients and perhaps reduce the need to use tools like ChatGPT Health without additional input from actual doctors.
Dr. Shah leads the team developing ChatEHR at Stanford University. ChatEHR is software that is integrated into electronic health record (EHR) systems and allows clinicians to interact with patient medical records in a more streamlined and efficient manner.
“Making electronic health records easier to use means doctors can spend less time combing through electronic health records for the information they need,” said Dr. Sneha Jain, an early tester of ChatEHR, in a Stanford Medicine article. “ChatEHR gives you proactive information so you can spend your time doing what matters: talking to patients and understanding what’s going on.”
Anthropic is working on AI products that can be used by clinicians and insurance companies as well as the Claude chatbot for the public. This week, Anthropic announced Claude for Healthcare and explained how it can be used to reduce time spent on tedious administrative tasks, such as submitting pre-authorization requests to insurance companies.
“Some people are seeing hundreds or even thousands of these pre-authorization cases a week,” Mike Krieger, Anthropic’s chief financial officer, said in a recent presentation at JPMorgan’s Healthcare Conference. “Now imagine cutting each task down by 20 to 30 minutes. That’s a dramatic time savings.”
As AI and healthcare become more intertwined, there is an inevitable tension between these two worlds. While doctors’ primary motivation is to help their patients, technology companies are ultimately accountable to their shareholders, no matter how noble their intentions may be.
“I think tension is important,” Dr. Bari says. “Patients trust us to be cynical and conservative to protect them.”
