Preference Shapley: Mapping the Influence of Minority Preferences on Language Model Output Diversity

Research Question: How do individual or minority annotator preferences shape the diversity of LMs’ open-ended outputs, and can this influence be systematically quantified and enhanced?

Hypothesis: Minority or idiosyncratic preferences, though often drowned out by majority trends, meaningfully contribute to increased diversity in LM outputs. By explicitly quantifying these contributions (e.g., using Shapley value analysis), we can design new reward signals or sampling strategies that promote richer, more pluralistic generations.

Experiment Plan: Use the Infinity-Chat dataset with its 25 annotators per prompt. Adapt Shapley value or similar cooperative game-theoretic tools to measure each annotator’s marginal contribution to diversity in ratings and output space. Fine-tune an LM with reward signals weighted to amplify high-Shapley (minority influence) annotations. Compare output diversity, human-judged creativity, and minority preference satisfaction before and after intervention. Key metrics: output distinctiveness, annotator agreement rates, and minority preference alignment.

References:

• Jiang, L., Chai, Y., Li, M., Liu, M., Fok, R., Dziri, N., Tsvetkov, Y., Sap, M., Albalak, A., & Choi, Y. (2025). Artificial Hivemind: The Open-Ended Homogeneity of Language Models (and Beyond). arXiv.org.