Procedural Overfitting: Does RL-Induced Traversal Skill Generalize Beyond Training Hierarchies?

Research Question: Do RL-enhanced LLMs generalize their procedural traversal skills to novel hierarchical structures, or do they overfit to the specific hierarchies seen during training?

Hypothesis: RL fine-tuning improves general procedural navigation strategies that transfer to new hierarchical domains, provided the new structures share some abstract properties with the training data.

Experiment Plan: Train models (SFT and RL) on a specific hierarchical dataset (e.g., ICD-10 codes). Construct novel but structurally similar hierarchies (e.g., taxonomy of animal species, synthetic trees). Evaluate zero-shot traversal and deep recall accuracy in these unseen domains. Compare performance drops and analyze procedural path-following in the new hierarchies. If RL models generalize, this supports the “procedural skill” hypothesis; if not, it suggests procedural overfitting.

References: ['Zhang, R., Kaniselvan, M., & Mireshghallah, N. (2025). Reinforcement Learning Improves Traversal of Hierarchical Knowledge in LLMs.', 'Wang, D., Li, B., Song, B., Chen, C., & Yu, F. (2024). HSMH: A Hierarchical Sequence Multi-Hop Reasoning Model With Reinforcement Learning. IEEE Transactions on Knowledge and Data Engineering.']