What Does Multi-Turn Reinforcement Learning Really Improve in LLM Agents?
Abstract
While multi-turn agentic reinforcement learning (RL) excels on long-horizon tasks, its performance gains remain highly sensitive to interaction constraints. Evaluating across sampling breadth K and interaction depth T via Pass@(K,T), we show that trajectory-level RL advantages depend on sufficient turn budgets, diminishing significantly when budgets tighten. Besides, agentic RL does not uniformly elevate single-step decision quality; rather, its strength concentrates in error recovery and long-horizon resilience. To mitigate the deployment risks caused by this budget dependence, we provide a preliminary discussion on the benefits and limitations of process supervision signals and introduce BudgetMix RL as an alternative that varies interaction budgets during training. Experiments demonstrate that BudgetMix RL achieves robust performance across diverse execution settings. Project Page: https://rethinking-agentic-rl.github.io/
Community
While multi-turn agentic reinforcement learning (RL) excels on long-horizon tasks, its performance gains remain highly sensitive to interaction constraints. Evaluating across sampling breadth K and interaction depth T via Pass@(K,T), we show that trajectory-level RL advantages depend on sufficient turn budgets, diminishing significantly when budgets tighten. Besides, agentic RL does not uniformly elevate single-step decision quality; rather, its strength concentrates in error recovery and long-horizon resilience. To mitigate the deployment risks caused by this budget dependence, we provide a preliminary discussion on the benefits and limitations of process supervision signals and introduce BudgetMix RL as an alternative that varies interaction budgets during training. Experiments demonstrate that BudgetMix RL achieves robust performance across diverse execution settings.
Really useful framing. Pass@(K,T) makes visible something that plain trajectory-level success rates hide. The BFCL row in Table 1 stood out to me: at a 25% turn budget, GRPO-Full on Qwen3-4B lands at 2.5 versus 3.0 for the base model, while BudgetMix reaches 8.5. Since BudgetMix puts the total and remaining budget in the prompt, I'm curious how it behaves when the cap isn't announced, which is common in real deployments where the limit is a timeout or a cost ceiling. Does it learn a generally more careful early policy, or does it mostly condition on the stated budget?
Thank you for your interest in our work and for raising this insightful question! In our current evaluation protocol, all models (Base, RL, and BudgetMix) are explicitly informed of the interaction budget through the same prompt. Our results therefore mainly reflect how well models adapt their strategies to a given budget, such as making more careful decisions when interactions are limited. However, as you pointed out, an important question is whether BudgetMix naturally adopts a more cautious early-stage policy even without explicit budget information. We haven't tested this setting yet, but we will consider exploring it and analyzing the resulting behaviors in future work. Thanks again for your thoughtful question and suggestion!
This is an automated message from the Librarian Bot. I found the following papers similar to this paper.
The following papers were recommended by the Semantic Scholar API
- ReflectPO: Reflection-Guided Policy Optimization for Long-Horizon LLM Agents (2026)
- DeltaTTG: Training Multi-Turn Agents with a Token-to-Go Progress Meter (2026)
- TIDE: Teacher-Student Transition via Informative Distillation and Exploration for Agentic RL (2026)
- AutoCompact: Learning When to Compact Context in Long-Horizon Coding Agents (2026)
- Rollout-Time Intervention: Unlocking Agentic Reinforcement Learning Beyond Policy Limits (2026)
- Environments as Scaffold: Enriching Feedback to Bootstrap Self-Evolving Agents in Long-Horizon Tasks (2026)
- RSIPO: Recursive Self-Improving Policy Optimization for Agentic Reinforcement Learning via Skill-Policy Co-Evolution (2026)
Please give a thumbs up to this comment if you found it helpful!
If you want recommendations for any Paper on Hugging Face checkout this Space
You can directly ask Librarian Bot for paper recommendations by tagging it in a comment: @librarian-bot recommend
Get this paper in your agent:
hf papers read 2611.06483 Don't have the latest CLI?
curl -LsSf https://hf.co/cli/install.sh | bash Models citing this paper 0
No model linking this paper
Datasets citing this paper 0
No dataset linking this paper
Spaces citing this paper 0
No Space linking this paper