Authors: Zijian Zhang¹, Rizhen Hu¹, Jiangweizhi Peng¹, Shiyang Li¹, Sujay Bhatt², Sihan Zeng², Mingyi Hong¹

(¹ University of Minnesota, Twin Cities, ² JPMorgan AI Research)

📄 Paper (arXiv:2611.14515)

TL;DR. During RL post-training, a model produces a lot of experience (its reasoning, its actions, the feedback it gets), and the reward keeps one number per attempt. AutoScaffold puts a frozen teacher LLM next to the training run. Every few steps the teacher reads what the student did, works out why it is failing, and writes temporary guidance into the student's training prompts. The guidance is removed at evaluation. With the same procedure in four domains, students trained this way beat the corresponding RL baselines without any scaffold at test time, for example by 7.2 points of success rate on ALFWorld.

as_fig1_teaser.png

Figure: During RL training, the frozen teacher reads the student's training experience and writes scaffolds into its training prompts. The scaffolds are removed at evaluation.


RL keeps one number per attempt

Reinforcement learning with verifiable rewards (RLVR) is now a standard way to post-train language models for reasoning and agentic tasks. Every training step produces rich experience, but RL learns from it only through the outcome reward. A failed attempt may contain useful reasoning. A successful one may rely on an inefficient strategy. And on tasks that are too hard for the model, almost every attempt fails, so the reward carries no signal at all.

The last point can be made precise for group-relative methods such as GRPO. These methods sample G rollouts per task and compare them with each other. With binary rewards, a group in which every rollout fails (or every rollout succeeds) gives zero advantage and no gradient. If each rollout succeeds with probability p, the chance that a group contains both outcomes is

$$ u(p) = 1 - (1-p)^G - p^G $$

This is largest when p is near one half and is roughly Gp when p is small. A task the student almost never solves almost never contributes to training.

Scaffolds help, but people design them by hand

One way to recover what the reward loses is training-time scaffolding: add auxiliary text such as a strategy, a hint, or part of a solution to the student's prompt during training, and remove it at evaluation. On a task the student rarely solves, a scaffold lets some rollouts succeed, so the group has mixed outcomes and the task produces learning signal.

Existing scaffolding methods have three limits:

In practice a researcher decides what support a domain needs and how it should change, often over several training runs. We asked whether an LLM teacher can do that job during a single run, from the student's own training experience, in any domain.

AutoScaffold: a teacher that reads the training run