Dataset-Centric OPSDC: Benchmarking and Diagnosing Compression on Reasoning Complexity and Overthinking

TL;DR: Let’s build new datasets that target reasoning “noise” and complexity, so we can see exactly when OPSDC helps or hurts. Design tasks with varying reasoning steps, distractors, and “trap” redundancy to stress-test concise reasoning.

Research Question: How does OPSDC perform across datasets engineered to vary in reasoning complexity, noise, and redundancy, and can such datasets reveal failure modes or inspire improved compression strategies?

Hypothesis: Dataset-driven evaluation will uncover specific types of reasoning noise (e.g., distractor steps, ambiguous logic) where OPSDC’s compression is most beneficial or where it risks harmful under-explanation.

Experiment Plan: Develop synthetic and semi-natural datasets with controlled complexity (steps required), redundant/noisy reasoning, and noise patterns, inspired by FDA/THR [4] and MatVQA [2]. Benchmark OPSDC and baselines, measuring accuracy, compression, and error types as a function of dataset properties. Use findings to propose enhancements (dynamic compression, error detection) or new evaluation metrics for reasoning compression. Release new datasets as open benchmarks for the community.

References:

• 1. Sang, H., Xu, Y., Zhou, Z., He, R., Wang, Z., & Sun, J. (2026). On-Policy Self-Distillation for Reasoning Compression.
• 2. Du, Y., Mondorf, P., Casola, S., Yao, Y., Litschko, R., & Plank, B. (2025). Reason to Rote: Rethinking Memorization in Reasoning. Conference on Empirical Methods in Natural Language Processing.
• 3. Wu, S., Zhang, H., Li, Y., Effaty, F., Ataei, A., & Liu, B. (2025). Seeing Beyond Words: MatVQA for Challenging Visual-Scientific Reasoning in Materials Science. arXiv.org.