AI in Medical Decision-Making: An Emerging Ethical Challenge
The increasing use of artificial intelligence (AI) in healthcare is raising critical ethical questions. Doctors warn that without a robust ethical framework, AI systems may end up serving the interests of insurers, administrators, and algorithms rather than patients.
A recent case study highlighted this dilemma. When OpenAI’s GPT-4 was asked to assess a young boy with low growth hormone levels from the perspective of a pediatric endocrinologist, it recommended treatment with human growth hormone. However, when prompted to take the perspective of a health insurance representative, GPT-4 provided a scientifically grounded rationale for denying care. The facts of the case remained unchanged; only the frame of reference shifted, altering the AI’s moral and clinical judgment.
Isaac Kohane, Nelson professor of bioinformatics at Harvard Medical School, shared this case at Mass General Brigham’s 2025 Medically Engineered Solutions in Healthcare (MESH) Core Incubator. Kohane emphasized that AI systems are not neutral arbiters of knowledge but are shaped by the values embedded in their design, deployment, and prompting.
The Problem of Perspective in AI Decision-Making
Kohane conducted an experiment using 1,000 simulated patient cases, comparing his decisions with those of three leading AI large language models (LLMs): OpenAI’s GPT-4, Google’s Gemini, and Anthropic’s Claude. While all models performed well on straightforward cases, their agreement with Kohane dropped significantly in more complex scenarios, especially when both patients had acute medical demands. Notably, Gemini came closest to Kohane’s decisions, while GPT-4 showed concerning variability, sometimes contradicting itself.
The experiment revealed that AI models don’t always behave as expected, even when provided with seemingly helpful information. For instance, when Kohane provided sample clinical decisions to guide the AI models, Claude’s performance degraded, becoming less aligned with both Kohane’s decisions and its own prior outputs. In contrast, GPT-4 became more aligned and consistent.
Ethical Implications and the Need for Leadership
The ethical implications of AI in healthcare extend beyond biased datasets to how human feedback teaches AI systems to respond and whose perspectives to prioritize. Already, health insurers are using AI to automate care authorization decisions, prompting lawsuits from patients and clinicians who argue that opaque algorithms deny care unjustly.
Kohane stresses that while regulation is important, ethical leadership from within the healthcare profession is crucial. The structural weaknesses in healthcare, such as increasingly inaccessible primary care and razor-thin margins for academic medical centers, may drive hospitals toward AI automation that enhances billing or scales high-revenue specialties.
Toward Ethical AI in Healthcare
Kohane’s Human Values Project aims to test LLMs across various global settings to document how well AI aligns with different communities’ decisions and whether models can be adjusted to reflect local clinical values. The goal is to fine-tune AI models to reflect different laws, regulations, social norms, and socioeconomic realities.
The current moment in medicine is likened to the invention of battlefield triage during the Napoleonic era, which revolutionized medicine by prioritizing survival over military rank. Today, medicine has a similar opportunity to use AI effectively while keeping patients at the center of healthcare. This requires developing an ethical framework for AI’s decision-making processes.
