AI medical tools found to downplay symptoms of women, ethnic minorities

Artificial intelligence tools used by doctors risk leading to worse health outcomes for women and ethnic minorities, as a growing body of research shows that many large language models downplay the symptoms of these patients.

A series of recent studies have found that the uptake of AI models across the healthcare sector could lead to biased medical decisions, reinforcing patterns of under-treatment that already exist across different groups in Western societies.

The findings by researchers at leading US and UK universities suggest that medical AI tools powered by LLMs have a tendency to not reflect the severity of symptoms among female patients, while also displaying less “empathy” toward Black and Asian ones.

The warnings come as the world’s top AI groups such as Microsoft, Amazon, OpenAI, and Google rush to develop products that aim to reduce physicians’ workloads and speed up treatment, all in an effort to help overstretched health systems around the world.

Many hospitals and doctors globally are using LLMs such as Gemini and ChatGPT as well as AI medical note-taking apps from start-ups including Nabla and Heidi to auto-generate transcripts of patient visits, highlight medically relevant details, and create clinical summaries.

In June, Microsoft revealed it had built an AI-powered medical tool it claims is four times more successful than human doctors at diagnosing complex ailments.

But research by the MIT’s Jameel Clinic in June found that AI models, such as OpenAI’s GPT-4, Meta’s Llama 3, and Palmyra-Med—a healthcare-focused LLM—recommended a much lower level of care for female patients, and suggested some patients self-treat at home instead of seeking help.

A separate study by the MIT team showed that OpenAI’s GPT-4 and other models also displayed answers that had less compassion towards Black and Asian people seeking support for mental health problems.]

That suggests “some patients could receive much less supportive guidance based purely on their perceived race by the model,” said Marzyeh Ghassemi, associate professor at MIT’s Jameel Clinic.

Similarly, research by the London School of Economics found that Google’s Gemma model, which is used by more than half the local authorities in the UK to support social workers, downplayed women’s physical and mental issues in comparison with men’s when used to generate and summarize case notes.