Aseel Bahakim

Artificial Intelligence Specialization · Cohort 2025

Evaluating Bias and Sycophancy in Clinical Large Language Model Summaries

Large language models (LLMs) have opened up new frontiers for reducing administrative burdens in health systems. Healthcare institutions around the world have already begun piloting the use of automated scribes and other tools aimed at summarizing patients’ clinical records. Aside from these institutional endeavors, there is also evidence that independent care providers are increasingly utilizing large-language models to support care delivery, despite the lack of guidelines and oversight mechanisms. In light of these recent trends, there is a critical need to better understand the prevalence, types, and impacts of bias that risk being perpetuated by LLMs. Social biases such as racial and gender stereotypes, as well as systematic discrepancies in clinical LLM summaries, pose a risk of exacerbating health disparities. Relatedly, biases may also stem from sycophancy, a phenomenon where LLMs generate outputs that reflect the user’s anticipated preferences or assumptions. The goal of this project is to evaluate the risk of social bias and the effects of sycophancy on several publicly-available LLMs, and summarise findings in a whitepaper or research report. To support these evaluations, we will use anonymized clinical notes from the MIMIC-IV dataset, which have already been annotated for patients’ language, race, and ethnicity.

Mentor: Zahra Shakeri

Read the final report (PDF)

See it in the program report (PDF)