Disentangled Skill Discovery in Internal RL for Foundation Models

TL;DR: Can we teach big language models to discover and reuse “skills” like humans do? By marrying ideas from unsupervised skill discovery (such as DUSDi) with internal RL, this project seeks to extract sets of disentangled, reusable latent controllers that can be composed to solve diverse, complex tasks. The experiment would focus on unsupervised pretraining, followed by downstream task transfer.

Research Question: Can internal RL frameworks be extended to autonomously discover and disentangle a repertoire of reusable latent skills within foundation models, enabling efficient transfer to novel tasks?

Hypothesis: Integrating mutual-information-based disentanglement objectives with internal RL will result in a library of composable skills, leading to faster task adaptation and improved sample efficiency during transfer learning.

Experiment Plan: - Augment the higher-order controller with a disentanglement loss (e.g., mutual information minimization between skill factors).

• Pretrain with unsupervised internal RL on a suite of diverse navigation or manipulation tasks.
• Evaluate transferability by composing discovered skills to solve new tasks with minimal finetuning.
• Compare with standard internal RL and vanilla skill discovery approaches.

References:

• 1. Kobayashi, S., et al. (2025). Emergent temporal abstractions in autoregressive models enable hierarchical reinforcement learning.
• 2. Hu, J., Wang, Z., Stone, P., & Mart'in-Mart'in, R. (2024). Disentangled Unsupervised Skill Discovery for Efficient Hierarchical Reinforcement Learning. Neural Information Processing Systems.