Echoes of the Past: Quantifying the Influence of Interaction History on LLM Personalization and Evaluation Outcomes

TL;DR: Imagine if a chatbot remembers your old questions and changes its answers because of them—how much does this really matter for what we measure? Start by systematically varying the length and type of user interaction history before asking identical benchmark questions, then analyze how these histories quantitatively shift LLM responses and evaluation metrics.

Research Question: How does the accumulation and nature of user interaction history quantitatively affect the output of LLMs in benchmark evaluations, and which types of history most strongly drive deviations from offline, stateless results?

Hypothesis: The longer and more contextually relevant the prior interaction history, the greater the divergence from offline evaluation outcomes, with certain types of content (e.g., emotionally charged or preference-expressive exchanges) producing outsized effects.

Experiment Plan: Recruit participants to interact with an LLM-based interface, each following scripted interaction histories varying in length and content (neutral, preference-rich, emotional, technical, etc.). At fixed intervals, insert standard benchmark questions and log the responses. Compare these to the same questions asked in a stateless, offline fashion. Use both automated and human evaluations to measure semantic, stylistic, and factual deviations. Employ regression analyses to identify which history features most strongly predict divergence.

References:

• Li, S. S., Bose, A., Brahman, F., Du, S. S., Koh, P. W., Fazel, M., & Tsvetkov, Y. (2025). Personalized Reasoning: Just-In-Time Personalization and Why LLMs Fail At It. arXiv.org.
• Sun, C., Yang, K., Reddy, R., Fung, Y., Chan, H. P., Zhai, C., & Ji, H. (2024). Persona-DB: Efficient Large Language Model Personalization for Response Prediction with Collaborative Data Refinement. International Conference on Computational Linguistics.