Hierarchical Subgoal Discovery via LLM-Driven World Modeling for Self-Adaptive Agent Training

TL;DR: What if the agent could learn to invent its own roadmap of subgoals while reading or interacting—like plotting checkpoints on a treasure map—making long tasks less overwhelming? Here, we extend KLong by fusing generative world model scaffolding (Hill, 2025) with automated subgoal discovery, enabling the agent to dynamically construct and refine its own hierarchical curriculum during training.

Research Question: Does empowering LLM agents to autonomously discover, generate, and adapt hierarchical subgoals improve their ability to solve extremely long-horizon tasks, compared to fixed or externally defined subgoal decompositions?

Hypothesis: Agents using LLM-driven world models to dynamically propose and adjust subgoals during training will display higher sample efficiency, robustness to task variance, and better transfer to new domains.

Experiment Plan: - Train KLong-style agents with an integrated LLM-based world model that proposes subgoal hierarchies for each task (following Hill, 2025).

• Use reinforcement signals to refine which subgoals/paths are most effective over time.
• Evaluate on multi-stage, long-horizon benchmarks (PaperBench, InnovatorBench) and track improvements in task completion, subgoal diversity, and transfer to new, unseen tasks.
• Ablate the world modeling module to assess its contribution.

References:

• Hill, B.A. (2025). Generative World Models of Tasks: LLM-Driven Hierarchical Scaffolding for Embodied Agents.
• Zhang, N., Zhao, Y., Yang, M., & Dai, S. (2025). LLMs augmented hierarchical reinforcement learning with action primitives for long-horizon manipulation tasks. Scientific Reports.