Prompt Robustness Vectors: A Multi-Dimensional Framework for Fine-Grained Model Evaluation

Papers like Mizrahi et al. (2023) and Polo et al. (2024) have shown the inadequacy of single-metric, single-prompt evaluations for LLMs. However, current multi-prompt methods still often collapse nuanced failures into a single score. Inspired by the persona-centric metamorphic evaluation (Chen et al., 2024) and the multi-axis annotation (Chang et al., 2025), this idea proposes a “robustness vector” for each prompt-model pair, with axes corresponding to distinct error types (e.g., hallucination, safety, bias, consistency, privacy). By tracking and analyzing these vectors, researchers can identify not just which prompts are problematic, but how and why—revealing targeted weaknesses that broad averages miss. This approach could be integrated into leaderboard reporting and model documentation, fundamentally changing how robustness is quantified and compared across models.

References:

1. Persona-centric Metamorphic Relation guided Robustness Evaluation for Multi-turn Dialogue Modelling. Yanbing Chen, Lin Li, Xiaohui Tao, Dong Zhou (2024). arXiv.org.
2. Efficient multi-prompt evaluation of LLMs. Felipe Maia Polo, Ronald Xu, Lucas Weber, M'irian Silva, Onkar Bhardwaj, Leshem Choshen, Allysson Flavio Melo de Oliveira, Yuekai Sun, M. Yurochkin (2024). Neural Information Processing Systems.
3. State of What Art? A Call for Multi-Prompt LLM Evaluation. Moran Mizrahi, Guy Kaplan, Daniel Malkin, Rotem Dror, Dafna Shahaf, Gabriel Stanovsky (2023). Transactions of the Association for Computational Linguistics.