Meta-Judgment and Self-Reflection: Can LLMs Explain and Evaluate Their Own Interestingness Criteria?

Research Question: Can prompting LLMs for meta-cognitive self-critique improve the transparency and alignment of their interestingness judgments with humans?

Hypothesis: Structured self-reflection will expose previously opaque biases and allow for targeted fine-tuning, leading to better alignment with human rationales and distributions.

Experiment Plan: For each math problem, prompt LLMs to rate interestingness, explain their rationale, and then critique the explanation for completeness, bias, or missing factors. Compare pre- and post-critique rationales for depth and alignment with human explanations. Optionally, use meta-rewarding (Wu et al., 2024) to iteratively refine judgment capability.

References: 1. Mishra, S., et al. (2025). A Matter of Interest: Understanding Interestingness of Math Problems in Humans and Language Models.
2. Wu, T., Yuan, W., Golovneva, O., Xu, J., Tian, Y., Jiao, J., Weston, J. E., & Sukhbaatar, S. (2024). Meta-Rewarding Language Models: Self-Improving Alignment with LLM-as-a-Meta-Judge. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing.