SimUSER-Bench: Simulated Personas for Stress-Testing LLM Personalization in Evaluation Frameworks

TL;DR: What if we could create fake users with realistic quirks and see how LLMs adapt (or fail) to them? Use simulated user agents with diverse, self-consistent personas to systematically probe LLM behavior and its evaluation robustness.

Research Question: Can simulated user agents with realistic, diverse personas be used to systematically and scalably assess how personalization affects LLM evaluation outcomes, compared to real users?

Hypothesis: Simulated personas can surface edge cases and non-obvious personalization effects in LLM evaluation, revealing both strengths and blind spots not captured by traditional offline or even limited real-user testing.

Experiment Plan: Adapt the SimUSER framework to language model evaluation: develop a suite of simulated personas (with varying preferences, expertise, communication styles). Have these agents interact with LLMs through a range of benchmark and open-ended tasks, maintaining dialogue history. Compare outputs to both stateless responses and a sample of real user interactions. Analyze where simulated agents uncover failure modes or behavioral shifts unseen in offline or small-scale real-user evaluations.

References:

• Bougie, N., & Watanabe, N. (2025). SimUSER: Simulating User Behavior with Large Language Models for Recommender System Evaluation. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 6: Industry Track).
• Cheng, M., Zhang, H., Yang, J., Liu, Q., Li, L., Huang, X., Song, L., Li, Z., Huang, Z., & Chen, E. (2024). Towards Personalized Evaluation of Large Language Models with An Anonymous Crowd-Sourcing Platform. The Web Conference.