Calibrating the Judges: Building Human Preference Clusters to Guide Reward Model Diversity

Research Question: Can explicitly modeling and calibrating to clusters of human preferences (rather than a global average) yield reward models that better incentivize diverse, human-aligned LM outputs?

Hypothesis: Reward models trained on clustered or personalized human preferences will more accurately capture the spread of acceptable, diverse outputs and reduce homogenization compared to models trained on aggregated ratings.

Experiment Plan: Use the 31,250 human annotations from Infinity-Chat to cluster annotators based on preference profiles (e.g., using unsupervised methods on ratings). Train separate reward models for each cluster. During LM training or inference, sample from or ensemble the cluster-specific reward models to select outputs. Compare intra- and inter-model diversity, as well as cluster-specific satisfaction, to standard reward model training.

References:

• Jiang, L., Chai, Y., Li, M., Liu, M., Fok, R., Dziri, N., Tsvetkov, Y., Sap, M., Albalak, A., & Choi, Y. (2025). Artificial Hivemind: The Open-Ended Homogeneity of Language Models (and Beyond). arXiv.org.