Understanding the Impact of Large Language Models on Clinical Reasoning

BACKGROUND

Existing studies show that large language models (LLMs) can match or exceed clinicians on diagnostic accuracy, but they tell us little about how AI changes the process of cognitive engagement, cognitive load, clinical reasoning, and patient engagement.

PROJECT DESCRIPTION

The AI Learning Lab is a collaborative project of the IC² Institute and Dell Medical School. The project examines how access to LLMs influences clinical reasoning in medical education and practice. Comparing patterns across medical students, residents, and faculty clinicians, the study explores whether the use of LLMs contributes to new forms of skill development, changes existing clinical practices, or introduces new challenges into the practice of medicine. Rather than focusing only on diagnostic accuracy, this study examines the reasoning process itself, including cognitive engagement, critical thinking, and facility with external cognitive tools as learners and clinicians work through clinical tasks.

METHODS

The research team has designed complex clinical vignettes for standardized patient encounters (actors portraying patients in simulated hospital rooms). The simulated patient encounters allow us to observe and capture communication, bedside manner, and real-time problem-solving — in other words, clinical reasoning in action. Some participants are receiving LLM access; others have only traditional resources with which to work through the same case. From the patient encounters, we are collecting multimodal data. We are also collecting participant reflections and post-task interviews to understand the perceived effects of external tools on reasoning, confidence and decision-making.

PROJECT GOALS

Our broadest goal is to generate empirical evidence to inform the responsible integration of AI into the delivery of health care.

Outcomes may include:

Generating an evidence-based account of how LLMs shape clinical reasoning behaviors, cognitive engagement, and performance across experience levels.
Identifying new clinician skills that grow out of AI-augmented clinical reasoning.
Generating a taxonomy of LLM engagement strategies.
Outlining recommendations for integration of our learnings into medical school curriculum.