Authors: Zijian Zhang¹, Rizhen Hu², Athanasios Glentis¹, Dawei Li¹, Chung-Yiu Yau¹, Hongzhou Lin³, Mingyi Hong¹

(¹ University of Minnesota, ² Peking University, ³ Amazon. This work is independent of and outside of the work at Amazon.)

📄 Paper (arXiv:2607.01232)

TL;DR. RL post-training normally updates every layer of an LLM. We trained one transformer layer at a time, with everything else frozen, and found that the best single layer recovers 100% to 114% of the gain from full-parameter RL. In all seven models we tested, the layers that matter sit in the middle of the network, and training only those layers beats standard full-parameter RL.


Where do the gains from RL come from?

Reinforcement learning with verifiable rewards (RLVR) is now a standard step in post-training LLMs, and it is behind much of the recent progress on math, code, and agentic tasks. Most research effort has gone into objectives, rewards, and optimizers. Nearly all of these methods share one default: every layer is updated, with the same learning rate.

We wanted to know whether that default is doing anything useful. When RL improves a model, which layers does the improvement come from? Earlier work shows that the layers of a pretrained LLM play different roles at inference time and under supervised fine-tuning. Whether RL post-training has a similar structure had not been checked.

The experiment: one layer at a time

The setup is simple on purpose. Take a model with L transformer layers. Pick a layer k, freeze everything else (the other layers, the embeddings, and the LM head), and run RL as usual. Gradients still flow through the whole network, but only layer k is updated. Repeat for every k.

To compare layers we define the layer contribution of layer k:

$$ \mathcal{C}(k) = \frac{S_k - S_{\text{base}}}{S_{\text{full}} - S_{\text{base}}} $$

Here S_base is the benchmark score of the pretrained model, S_full is the score after standard full-parameter RL, and S_k is the score after training only layer k. A contribution of 1 means that one layer recovered the entire gain of full-parameter training. Above 1 means it did better.

To keep the comparison fair, we tune the learning rate for the full-parameter baseline and then reuse that learning rate, the same hyperparameters, and the same number of training steps for every single-layer run. The baseline runs at its best setting and the single-layer runs get no tuning of their own.

We ran this on seven models from 1.5B to 8B parameters: three Qwen3 base models (1.7B, 4B, 8B) trained with GRPO on math, Qwen2.5-Math-1.5B trained with Dr. GRPO, two Qwen2.5-Instruct models trained with GiGPO on the agentic benchmark ALFWorld, and DeepSeek-Distilled-Qwen-7B trained with GRPO.

Finding 1: a single layer can match full-parameter RL

fig1_layer_contribution.png

Figure: (a) Layer contribution of every single-layer run, plotted against relative depth (0 is the first layer, 1 is the last), for all seven models. Points in the green band beat full-parameter RL. Dashed lines are models where we trained a subset of layers. (b) Math average on the three Qwen3 models for the base model, full-parameter RL, the best single layer, and the best layer-guided strategy.

In all seven models, the best single layer matches or beats full-parameter training, while the worst layers recover very little.

Model RL algorithm Task Best layer C Worst layer C
Qwen3-1.7B-Base GRPO Math 1.14 0.28
Qwen3-4B-Base GRPO Math 1.06 0.66
Qwen3-8B-Base GRPO Math 1.07 -0.51
Qwen2.5-Math-1.5B Dr. GRPO Math 1.01 0.42
Qwen2.5-1.5B-Instruct* GiGPO Agentic (ALFWorld) 1.02 0.25
Qwen2.5-3B-Instruct* GiGPO Agentic (ALFWorld) 1.01 0.17
DeepSeek-Distilled-Qwen-7B* GRPO Math 1.05 0.33