Weak-to-Strong On-Policy Distillation

Overview

We introduce W2S-OPD, a simple yet effective OPD framework that improves the strong student by distilling from multiple weak models. W2S-OPD constructs a proxy teacher in logit space from a contrast pair of a stronger positive and a weaker negative model, both weaker than the student. Their logit difference isolates the capability direction separating them, which is then added onto the student's own base model. The resulting proxy teacher couples this direction with the student's own general ability, while staying distributionally adjacent to the student. The student then distills it by minimizing the per-token reverse KL on its own rollouts.

On four math reasoning and three code generation benchmarks, our main findings are threefold: (i) Surpassing the teacher it learns from. In the pre-RL/post-RL setting, W2S-OPD outperforms OPD by 11.4% and 12.0% relative on math reasoning under single- and multi-teacher distillation, and lifts the student above the domain expert itself. (ii) Improving from purely weaker models. In the smaller/larger and contrastive-hints settings, W2S-OPD still improves the student even though every supervision source is weaker than it. (iii) Different contrasts reinforce complementary reasoning patterns. The post-RL contrast places its largest offsets on the pivotal decision points that determine success (e.g., building the solution framework and verification), whereas the scale contrast emphasizes the basic solving procedure.

**Figure 1.** Overview of W2S-OPD. A positive and a negative model form a contrast pair whose logit difference isolates a capability direction, which is re-anchored onto the student's base model to form a proxy teacher. The student generates on-policy rollouts and minimizes the per-token reverse KL toward this teacher. The contrast pair can be instantiated as a post-RL expert against its pre-RL initialization, a larger base model against a smaller one, or a single model conditioned on correct against wrong hints.

Method

Given a positive model $m^{+}$ and a weaker negative model $m^{-}$ that share a weak generic foundation, subtracting their logits cancels the shared foundation and retains the capability direction that separates them. Adding this direction onto the strong base model yields a proxy teacher that combines the positive model's specialized skill with the student's general strength while staying close to the student:

$\pi_{T,\alpha}(\cdot \mid s_t) = \mathrm{softmax}\big(\, z_{\mathrm{base}}(s_t) + \alpha\,( z^{+}(s_t) - z^{-}(s_t) )\big)$

where $\alpha \geq 0$ amplifies the injected capability direction. The student then minimizes the per-token reverse $\mathrm{KL}(\pi_S \,\|\, \pi_{T,\alpha})$ on its own rollouts. The contrast pair can be instantiated in three ways:

(i) Pre-RL and post-RL. $m^{+}$ is a domain expert obtained by applying RL to a small base model, and $m^{-}$ is its pre-RL initialization. Their difference isolates the domain skill acquired through RL, transferring an expensive capability to the large student without ever running RL at the student's scale.

(ii) Smaller and larger. $m^{+}$ and $m^{-}$ are two off-the-shelf base models of different sizes (e.g., Qwen3-4B and Qwen3-0.6B). Their difference isolates the capability that emerges purely from scale. This signal comes for free from models that already exist, requiring no new data, reward design, or training.

(iii) Correct and wrong hints. $m^{+}$ and $m^{-}$ are a single base model conditioned on a correct and a wrong solution hint of the identical format. Their difference cancels the style shift the hint induces and isolates the instance-level direction toward the correct solution, turning any labeled dataset into token-level supervision with only one small model.

The construction naturally extends to multiple positive models: summing several capability directions on the shared anchor injects multiple domain skills into a single proxy teacher, merging them into one student in a single distillation run.

Distillation from Pre-RL / Post-RL

(a) A small post-RL expert substantially improves the large student. W2S-OPD consistently outperforms OPD across all benchmarks in the single-teacher setting, with 11.4% and 3.7% average relative improvements on math and code, respectively. Notably, on math reasoning W2S-OPD surpasses the domain teacher itself, whereas OPD remains below it. (b) W2S-OPD outperforms OPD when distilling from multiple domain teachers. In the multi-teacher setting, where the capabilities of the math and code experts are merged into a single student, W2S-OPD consistently leads to better performance than OPD on all benchmarks.

Method	Math Reasoning					Code Generation
Method	AIME24	AIME25	HMMT25 (Feb.)	HMMT25 (Nov.)	Avg.	HumanEval+	MBPP+	LCB	Avg.
Negative Model (4B)	21.6	19.9	10.0	8.2	14.9	79.9	63.6	20.9	54.8
Positive Model (4B-RL)	62.0	57.9	34.8	40.4	48.8	86.3	71.0	27.1	61.5
Student Base (8B)	26.1	21.1	11.6	9.0	17.0	81.2	70.0	21.7	57.6
Single-Teacher Distillation
SFT	60.2	54.4	30.7	37.6	45.7	82.5	71.8	24.8	59.7
OPD	62.1	54.8	30.6	38.3	46.5	83.1	71.2	22.0	58.7
W2S-OPD	68.9	60.1	35.1	43.1	51.8	84.8	72.4	26.8	60.9
Improv.	+6.8	+5.3	+4.5	+4.8	+5.3	+1.7	+1.2	+5.0	+2.2
Multi-Teacher Distillation
SFT	63.5	54.6	29.6	39.4	46.8	84.0	71.4	25.7	60.4
OPD	64.3	54.9	29.6	37.4	46.5	83.4	69.5	25.6	59.5
W2S-OPD	67.4	61.0	34.7	45.2	52.1	85.2	71.0	27.1	61.1
Improv.	+3.1	+6.1	+5.1	+7.8	+5.6	+1.8	+1.5	+1.5	+1.6

Table 1. Results for the Pre-RL/Post-RL contrast setting. W2S-OPD beats OPD and even surpasses the 4B expert on math. Improv. reports the absolute gain over OPD.

Distillation from Smaller / Larger Base Models

Two weak base models can improve the stronger student. When the proxy teacher is constructed from Qwen3-4B and Qwen3-0.6B, both weaker than the student, W2S-OPD still improves the student by an absolute 6.0% on math reasoning and 1.2% on code generation on average. This confirms that the inherent gap between two off-the-shelf weak models also encodes a transferable improving direction to distill from.

Method	Math Reasoning					Code Generation
Method	AIME24	AIME25	HMMT25 (Feb.)	HMMT25 (Nov.)	Avg.	HumanEval+	MBPP+	LCB	Avg.
Negative Model (0.6B)	1.6	2.3	0.1	2.7	1.7	25.6	26.3	4.6	18.8
Positive Model (4B)	21.6	19.9	10.0	8.2	14.9	79.9	63.6	20.9	54.8
Student Base (8B)	26.1	21.1	11.6	9.0	17.0	81.2	70.0	21.7	57.6
W2S-OPD	33.1	25.9	13.0	20.1	23.0	81.7	71.1	23.7	58.8
Improv.	+7.0	+4.8	+1.4	+11.1	+6.0	+0.5	+1.1	+2.0	+1.2

Table 2. Results for the Smaller/Larger contrast setting. W2S-OPD improves the 8B student above its own base even though both source models are weaker than it. Improv. reports the absolute gain over the student base model.

Distillation from Correct / Wrong Hints

A contrastive hint direction from a single weak model improves the student. W2S-OPD improves the student by an absolute 1.4% on math reasoning and 1.1% on code generation on average, despite the hint model being a 4B model weaker than the student. This shows that a meaningful learning signal can also be extracted from a difference in context, without requiring two different models.

Method	Math Reasoning					Code Generation
Method	AIME24	AIME25	HMMT25 (Feb.)	HMMT25 (Nov.)	Avg.	HumanEval+	MBPP+	LCB	Avg.
Pos./Neg. Model (4B)	21.6	19.9	10.0	8.2	14.9	79.9	63.6	20.9	54.8
Student Base (8B)	26.1	21.1	11.6	9.0	17.0	81.2	70.0	21.7	57.6
W2S-OPD	27.7	23.3	12.0	10.7	18.4	82.5	70.8	22.7	58.7
Improv.	+1.6	+2.2	+0.4	+1.7	+1.4	+1.3	+0.8	+1.0	+1.1

Table 3. Results for the contrastive hints setting. W2S-OPD improves the 8B student above its own base with only a single weaker and smaller model. Improv. reports the absolute gain over the student base model.

Analysis

1. Learning faster and more stably. W2S-OPD learns faster and consistently outperforms OPD throughout the training.

Training dynamics. — **Figure 2.** Training dynamics on the math and code benchmarks over training steps. W2S-OPD improves faster and outperforms OPD.

2. Amplification coefficient $\alpha$. A moderate $\alpha$ (e.g., 1.0) works best.

Impact of alpha. — **Figure 3.** Impact of $\alpha$ on the math and code benchmarks. OPD is included for reference (gray line). W2S-OPD outperforms OPD over a wide range of $\alpha$.

3. The three contrasts reinforce different kinds of tokens. The post-RL and hint contrasts place more weight on the reasoning framework, such as the Plan and Monitor episodes that structure and track the solution, whereas the scale contrast keeps more weight on the core solving steps, Analyze and Implement. The hint contrast additionally concentrates on the final Answer tokens. These emphases are complementary, resulting in a student learning to improve the reasoning by distilling different patterns.

Case study: the three contrasts strengthen different reasoning tokens. — **Figure 4.** The three contrasts reinforce different kinds of tokens. Tokens on a shared solution are shaded by how strongly each capability direction reinforces them (darker = stronger).

4. Out-of-domain generalization. Trained only on math, the distilled skill transfers to out-of-domain tasks without eroding general ability, as shown below.

Method	GPQA-Diamond	IFBench
Positive Model	51.8	26.0
Student Base	38.9	26.3
OPD	54.4	25.9
W2S-OPD	56.5	27.0
Improv.	+2.1	+1.1

Table 4. Out-of-domain generalization on GPQA-Diamond and IFBench (trained on math only). W2S-OPD transfers out of domain and improves general ability, whereas OPD can degrade it below the base. Improv. indicates the absolute gain over OPD.

Discussion

As models approach the frontier, the assumption of an ever-stronger teacher no longer holds, and where the supervision for OPD should come from becomes an open question. Weak supervision sources are abundant, yet how far they can push a stronger student before weak-to-strong supervision saturates, and how to elicit more informative signals from them, remain to be explored. Future work may develop weak-to-strong learning into a sustained paradigm for OPD and post-training more broadly, in which frontier models continue to improve from supervision sources cheaper and weaker than themselves.

BibTeX

@article{yu2026w2sopd,
  title={Weak-to-Strong On-Policy Distillation},
  author={Yu, Fangxu and Lin, Zinan and Liu, Xiaodong and Xu, Weijia and Xu, Michael and Zhou, Tianyi and Gao, Jianfeng},
  journal={arXiv preprint arXiv:2607.26246},
  year={2026}
}