Dynamic Safety Overrides: When Should LLMs Reject Harmful Contexts in Medical QA?

TL;DR: Teach language models to know when to say “no” or flag suspicious medical info, even if the context looks official. The initial experiment would test rule-based and learning-based override mechanisms that trigger when counterfactual or unsafe medical evidence is detected, measuring reductions in dangerous completions.

Research Question: Under what circumstances should LLMs override the provided medical context in favor of safety, and how can such overrides be reliably triggered?

Hypothesis: Integrating explicit safety override protocols—either via rule-based checks (e.g., toxic substance detection) or learned skepticism—will significantly reduce the rate at which LLMs accept and propagate harmful counterfactual evidence, without unduly suppressing legitimate novel findings.

Experiment Plan: - Develop a set of explicit safety triggers (e.g., toxic substance recognition, implausible interventions, or semantic anomaly detection).

• Implement both deterministic (rule-based) and probabilistic (learned) override modules within LLM inference pipelines.
• Run models on MedCounterFact and related datasets, with and without overrides.
• Measure: frequency of unsafe completions, false positive/negative override rates, impact on legitimate, rare-but-true findings.
• Analyze trade-offs between faithfulness, safety, and model utility.

References:

• Mo, K., Venkatayogi, S., Shaib, C., Kouzy, R., Xu, W., Wallace, B. C., & Li, J. J. (2026). Faithfulness vs. Safety: Evaluating LLM Behavior Under Counterfactual Medical Evidence.
• Han, T., Kumar, A., Agarwal, C., & Lakkaraju, H. (2024). MedSafetyBench: Evaluating and Improving the Medical Safety of Large Language Models. Neural Information Processing Systems.