Cognitive-Inspired Monitorability Metrics: Blending AI Supervision with Human Factors Engineering

TL;DR: Let’s borrow tricks from human factors and cognitive science to design smarter, more human-aligned ways to supervise AI models! The initial study might adapt clinical workload and operator trust metrics (from healthcare and autonomy) to build new composite monitorability metrics for LLMs.

Research Question: Can cognitive workload, trust calibration, and transparency metrics from human-AI collaboration domains improve our ability to supervise and interpret advanced model behaviors?

Hypothesis: Metrics that capture cognitive load, trust, and situational awareness—when transferred and adapted to model monitoring—can reveal new dimensions of monitorability that standard CoT-based metrics miss, especially as models scale.

Experiment Plan: - Review and adapt human factors metrics (Hu & Shi, 2025; Zhang & Xu, 2025) and trust/situation awareness measures (Setyowati et al., 2025) for AI model supervision.

• Implement operator-in-the-loop studies where humans monitor model outputs, tracking their cognitive workload and trust in the model’s reasoning.
• Compare new composite metrics (that blend behavioral, ergonomic, and transparency factors) against traditional monitorability metrics on diverse LLM tasks.
• Analyze whether these metrics better predict breakdowns, anomalies, or “unsafe” model behaviors as model size and complexity increase.

References:

• Hu, C., & Shi, X. (2025). Optimizing clinical workflow through human factors and ergonomics: A comprehensive review of methodologies, applications and future directions. Healthcare Engineering.
• Zhang, Z., & Xu, H. (2025). Explainable AI for Maritime Autonomous Surface Ships (MASS): Adaptive Interfaces and Trustworthy Human-AI Collaboration. arXiv.org.
• Setyowati, D. L., Perdana, A. S. D., Latif, A., & Widyarto, W. (2025). Research Trends in Road Safety (2013-2023): A bibliometric Review Using Science Mapping Techniques on Human and Technological Factors. Media Publikasi Promosi Kesehatan Indonesia (MPPKI).