Cross-Cultural Counterfactuals: A Dynamic Test Bed for LLM Fairness in Low-Resource and Culturally Diverse Settings

Most debiasing and auditing work is concentrated in resource-rich, Western contexts (Yogarajan et al., 2024; Wasi et al., 2025). Inspired by the gaps identified in debiasing LLMs for Thai (Sermsri & Panboonyuen, 2025) and Bengali dialects (Wasi et al., 2025), this idea proposes a toolkit for constructing “cross-cultural counterfactuals”—parallel test cases that systematically vary cultural, regional, and linguistic factors. The toolkit would enable the automatic injection and evaluation of fairness scenarios across a spectrum of cultural identities, dialects, and value systems, using both human annotation and LLM-assisted data generation. By benchmarking LLMs against these dynamic, culturally sensitive test beds, researchers can uncover locality-specific biases and develop generalizable debiasing strategies. This would bridge the fairness audit divide between high- and low-resource settings, supporting ethical, inclusive LLM deployment worldwide.

References:

1. Debiasing Large Language Models in Thai Political Stance Detection via Counterfactual Calibration. Kasidit Sermsri, Teerapong Panboonyuen (2025).
2. Debiasing large language models: research opportunities*. Vithya Yogarajan, Gillian Dobbie, Te Taka Keegan (2024). Journal of the Royal Society of New Zealand.
3. Debiasing large language models: research opportunities*. Vithya Yogarajan, Gillian Dobbie, Te Taka Keegan (2024). Journal of the Royal Society of New Zealand.
4. Dialectal Bias in Bengali: An Evaluation of Multilingual Large Language Models Across Cultural Variations. Azmine Toushik Wasi, Raima Islam, Mst Rafia Islam, Farig Sadeque, Taki Hasan Rafi, Dong-Kyu Chae (2025). The Web Conference.