Challenging Protected Attribute Assumptions: Auditing LLM Fairness Without Explicit Attribute Labels

A core assumption in most fairness auditing (e.g., Haim et al., 2024; Chu et al., 2024) is the availability of explicit demographic or protected attribute labels (race, gender, etc.)—but these are often unavailable, contested, or context-dependent. This project proposes a paradigm shift: develop auditing techniques that infer latent groupings or bias axes using unsupervised clustering (e.g., topic modeling, representation learning) and proxy variables, mapping these emergent groups to observed disparities in LLM outputs. The method would allow auditing of fairness in settings where attributes are unobserved or sensitive (privacy-preserving), and could reveal hidden forms of group bias missed by conventional audits. This directly challenges prevailing assumptions and supports more universal, context-agnostic fairness analysis—critical for global LLM deployment and compliance with emerging privacy standards.

References:

1. Fairness in Large Language Models: A Taxonomic Survey. Zhibo Chu, Zichong Wang, Wenbin Zhang (2024). SIGKDD Explorations.
2. What's in a Name? Auditing Large Language Models for Race and Gender Bias. Amit Haim, Alejandro Salinas, Julian Nyarko (2024). arXiv.org.