Detecting and Explaining Anomalous Belief Shifts in Language Models

Research Question: What kinds of anomalous or unexpected belief shift patterns emerge in language models as context accumulates, and how can these be detected and explained to users?

Hypothesis: Specific interaction patterns or content types (e.g., adversarial, ambiguous, or emotionally charged contexts) will lead to belief shifts that are either disproportionate or surprisingly stable, potentially exposing latent model weaknesses or alignment issues.

Experiment Plan: Design interaction scripts across domains (moral, political, factual) with expected belief shift trajectories. Automatically flag sessions where model beliefs remain stubbornly constant or shift abruptly, counter to expectations based on prior data. Use XAI techniques (e.g., SHAP, attention/gradient analysis) to identify input features driving anomalous shifts. Generate and test user-facing explanations for these anomalies, measuring interpretability and user trust. Validate on real-world dialog datasets (e.g., Bard Intelligence and Dialogue Dataset).

References: 1. Tripathi, A., Jadhav, S., Singh, S., Nandan, S. K., Vyas, R., & Vyas, O. P. (2024). ProM-Ex: An Explainable Framework for Anomaly Detection in Process Mining Using Large Language Models. Conference Information and Communication Technology. 2. Geng, J., Chen, H., Liu, R., Horta Ribeiro, M., Willer, R., Neubig, G., & Griffiths, T. L. (2025). Accumulating Context Changes the Beliefs of Language Models.