Step-Grained Reward Shaping for Fine-Tuned Multi-Step Tool Learning in Small Orchestrators

TL;DR: What if ToolOrchestra could get “mini-rewards” for each smart step, not just the end result? We propose integrating step-grained RL—rewarding each successful or efficient tool invocation—to see if this helps small orchestrators learn smarter, more robust multi-step strategies. An initial experiment would replicate StepTool’s approach within ToolOrchestra and compare learning speed and final performance on multi-stage tasks.

Research Question: Does step-level reward shaping accelerate and improve multi-step tool-use strategy acquisition in small orchestrators?

Hypothesis: Providing granular feedback for each effective tool interaction will yield faster convergence and better multi-step reasoning than coarse-grained, outcome-only rewards.

Experiment Plan: - Modify ToolOrchestra’s RL scheme to include step-level rewards for each tool invocation, following StepTool (Yu et al., 2024).

• Apply to tasks with multiple decision points (e.g., nested API calls, data pipeline construction).
• Measure learning curves, convergence speed, and ultimate task success vs. baseline RL reward schemes.
• Analyze tool-use patterns and error recovery behaviors.

References:

• Su, H. et al. (2025). ToolOrchestra: Elevating Intelligence via Efficient Model and Tool Orchestration.
• Yu, Y., Wang, Z., Ma, W., Guo, Z., Zhan, J., Wang, S., Wu, C., Guo, Z., & Zhang, M. (2024). StepTool: Enhancing Multi-Step Tool Usage in LLMs via Step-Grained Reinforcement Learning. Proceedings of the 34th ACM International Conference on Information and Knowledge Management.