Neurosymbolic Internal RL: Bridging Abstract Reasoning and Latent Action Generation

TL;DR: Mix logic with intuition! This research fuses symbolic reasoning modules (e.g., logic programs or graph-based planning) with internal RL in autoregressive models. The goal is to see if explicit, high-level reasoning can guide or constrain the discovery/composition of latent, temporally abstract controllers, yielding more interpretable and generalizable behaviors. The experiment tests this hybrid approach on tasks requiring both symbolic planning and low-level execution.

Research Question: Can the integration of symbolic reasoning modules with internal RL in autoregressive models facilitate the emergence of more interpretable, generalizable, and efficient hierarchical controllers?

Hypothesis: Neurosymbolic hybrids will outperform purely neural internal RL on tasks that require long-horizon planning, compositional generalization, or adherence to explicit rules, and will yield more transparent decision-making pipelines.

Experiment Plan: - Embed a symbolic planner or logic module alongside the higher-order controller in the internal RL setup.

• Use the symbolic module to suggest or constrain high-level latent actions/abstractions.
• Evaluate on hybrid tasks (e.g., planning with explicit constraints, hierarchical tasks with logical subgoals).
• Measure sample efficiency, generalization to unseen logical structures, and interpretability of learned controllers.

References:

• 1. Kobayashi, S., et al. (2025). Emergent temporal abstractions in autoregressive models enable hierarchical reinforcement learning.
• 2. Li, D., Zhang, T., Huang, L., Wang, C., He, X., & Xue, H. (2024). KEHRL: Learning Knowledge-Enhanced Language Representations with Hierarchical Reinforcement Learning. International Conference on Language Resources and Evaluation.