Personalization Leakage: Evaluating Privacy and Identifiability Risks in Personalized LLM Evaluations

TL;DR: If models adapt to you, can they accidentally reveal who you are? Test if and how personalized LLM behavior can inadvertently leak user identity or sensitive attributes, especially when evaluation data is shared or analyzed.

Research Question: To what extent can personalized LLM responses be used to infer or re-identify user attributes, and what are the privacy implications for evaluation frameworks that incorporate personalization?

Hypothesis: Personalized LLM outputs, especially when based on rich interaction histories, can be reverse-engineered to reveal sensitive user information, posing privacy risks that are invisible in stateless offline evaluations.

Experiment Plan: Collect personalized LLM interaction logs using anonymized but attribute-rich user histories. Task adversarial models with re-identifying user attributes (e.g., age, location, interests) from anonymized outputs. Compare privacy leakage rates between stateless and personalized evaluation setups. Test mitigation strategies, such as distributional user embeddings or local profile generation (as in Ding et al., 2025). Recommend privacy-preserving evaluation protocols.

References:

• Ding, Y., Tan, Y., Liu, X., Niu, C., Meng, F., Zhou, J., Liu, N., Wu, F., & Chen, G. (2025). Personalized Language Model Learning on Text Data Without User Identifiers. Knowledge Discovery and Data Mining.
• Mendoza, R., Cruz, I., Liu, R., Deshmukh, A., Williams, D., Peng, J., & Iyer, R. (2024). Adaptive Self-Supervised Learning Strategies for Dynamic On-Device LLM Personalization. arXiv.org.