Uncertainty-Aware Medical LLMs: Quantifying Doubt in the Face of Counterfactuals

TL;DR: Teach LLMs to say “I’m not sure” when the evidence looks weird or unsafe, instead of answering confidently. The initial experiment would compare baseline and uncertainty-augmented LLMs on MedCounterFact, scoring for cautiousness and reduction in unsafe completions.

Research Question: Can integrating advanced uncertainty quantification methods into LLMs help mitigate overconfident acceptance of counterfactual medical evidence?

Hypothesis: LLMs equipped with predictive and semantic uncertainty estimation will be less likely to provide confident, unsafe answers when presented with implausible or dangerous medical evidence.

Experiment Plan: - Implement uncertainty quantification (e.g., Bayesian inference, deep ensembles, Monte Carlo dropout) in LLM outputs for medical QA.

• Augment MedCounterFact with labels for plausible, implausible, and hazardous contexts.
• Prompt LLMs to indicate confidence or uncertainty in their answers.
• Evaluate: correlation between uncertainty and evidence plausibility; rate of caveated/hedged responses; reduction in harmful outputs.
• User study: Do clinicians prefer uncertainty-aware model outputs in high-risk scenarios?

References:

• Mo, K. et al. (2026). Faithfulness vs. Safety: Evaluating LLM Behavior Under Counterfactual Medical Evidence.
• Atf, Z., Safavi-Naini, S. A. A., Lewis, P. R., Mahjoubfar, A., Naderi, N., Savage, T., & Soroush, A. (2025). The challenge of uncertainty quantification of large language models in medicine. arXiv.org.