AI guides

Ask-to-Distill optimizes training questions using teacher–student distribution signals

By 机器之心1 viewsSource: 机器之心
Ask-to-Distill question generation and teacher–student supervision

UIUC and collaborators propose A2D, training a question generator with frozen teacher–student disagreement and compatibility signals before on-policy distillation. The paper reports gains in math, code and agent tasks; its learning-value proxy has limits.

One good question may be worth more than a thousand ordinary ones.

In the training of large language models, it is generally believed that making a model smarter requires more high-quality data, more reliable reference answers, and more powerful teacher models. However, recent research on On-Policy Distillation (OPD) has revealed a phenomenon worth noting: even with only a very small number of training questions, a student model may be able to obtain a substantial performance improvement from a teacher model.

That raises a new question: if a few questions can enable substantial knowledge transfer, can models actively learn to generate the questions most useful for their own learning? And if those questions are self-generated, without reference answers or external verifiers, how do we judge which genuinely deserve to be used for training?

Researchers from the University of Illinois Urbana-Champaign, the Hong Kong University of Science and Technology, Texas A&M University and other institutions recently proposed Ask-to-Distill (A2D), a synthetic-question optimization framework for on-policy distillation. Instead of evaluating synthetic data through reference answers or task success, A2D directly measures, from teacher and student predictive distributions, how much usable teacher supervision a question elicits, then optimizes a question generator through reinforcement learning. Experiments in mathematical reasoning, code generation and agent tasks show that A2D can generate more effective distillation data without external training questions or correct answers, outperforming multiple real- and synthetic-data baselines on average in the evaluated settings.

More interestingly, the study found that A2D-generated questions shortened student responses by about 20% while increasing cumulative learning signals by around 28%. This means that genuinely valuable training questions may not need to make a model carry out longer and more complex reasoning; they may instead be the questions that can precisely expose its gaps in knowledge.

Article illustration

  • Paper title: Ask to Distill: Learning to Synthesize Data for On-Policy Distillation
  • Paper: https://arxiv.org/abs/2610.27450
  • Code: https://github.com/GasolSun36/Ask_to_Distill
  • Models and data: https://huggingface.co/collections/gasolsun/ask-to-distill-model-and-data-collection

From “how to answer questions” to “which questions to ask”

Knowledge distillation for large language models generally aims to transfer the capabilities of a strong teacher model to a relatively weaker student model. Traditional supervised fine-tuning often uses answers generated by the teacher as the training targets. However, trajectories generated by the teacher do not necessarily match the student model's own generation distribution, and during inference the student may also enter states that were not covered during training.

On-policy distillation offers another route. The student first generates its own response to a training question; the teacher then provides token-level supervision along that actual trajectory. Training directly on states the student visits can reduce the distribution mismatch of fixed teacher demonstrations.

An easily overlooked detail matters here: although the teacher supplies knowledge, the training question determines where supervision can be supplied. The same teacher and student may enter completely different generation trajectories when faced with different questions. Some questions are too simple: the predictions of the student and teacher are essentially the same, and the teacher has little additional knowledge to pass on. Some questions are too difficult: although the teacher and student disagree greatly, the answers preferred by the teacher may lie far beyond the range of candidates from which the student can currently learn effectively. Other questions fall precisely between these two extremes: the student has not fully mastered them, but the teacher can provide a clear direction for correction among the candidate answers the student already has. It may be this third type of question that truly offers high distillation value. The research team therefore proposes a new perspective:

Training questions are not merely tasks to solve; they are “distribution elicitors” that trigger teacher-supervision signals.

Their role is to guide students toward states where teachers can teach effectively, not just demand answers. Synthetic-data optimization therefore shifts from generating harder questions to generating questions with greater learning value.

The core method: without checking correctness,

how can we know a question is valuable?

The core idea of A2D is to have a question generator continually explore the following question: what kinds of questions can provide more effective supervision for a particular student–teacher model pair?

The framework has three stages.

Article illustration

Stage one: generate questions and observe teacher–student disagreement

A2D copies the initial student’s parameters to initialize a separate question generator, the Synthesizer.

This generator is responsible for proposing questions, rather than solving them. For each generated question, the system uses the frozen student model to generate an answer, then has the frozen teacher model make predictions along exactly the same trajectory generated by the student. As a result, at every token position, the system can obtain the respective probability distributions of the student and teacher. This enables A2D to observe the positions at which their knowledge differs without needing to know the correct answer to the question. Note that throughout the entire reinforcement-learning training process for the question generator, the teacher and the student used for scoring remain frozen; only the question generator is updated. Therefore, what A2D optimizes is not the student's problem-solving parameters, but the data distribution determining which questions the student will encounter next.

Stage two: disagreement must also be learnable

KL divergence can measure disagreement between teacher and student.

Intuitively, if the teacher and student make almost the same prediction for the next token, there is little disagreement at that position; if their predictions differ greatly, the disagreement is greater. However, A2D does not directly use KL divergence as its sole learning reward. The reason is that large disagreement does not necessarily mean effective learning. Suppose the student considers A, B and C the likeliest next tokens, while the teacher favors D, E and F. If distillation updates mainly act on the student’s top-K candidates but the teacher’s probability mass falls outside them, even substantial disagreement may not become effective supervision. A2D therefore introduces two complementary measures:

  • Disagreement: measures differences in predictive distributions, asking how much different knowledge the teacher can convey.
  • Compatibility: measures how much teacher probability mass falls on the student’s top-K candidate tokens, asking how much intended knowledge lies within the student’s current learning range.

A2D combines them into a learning-value signal for each token:

Article illustration

Article illustration

With this design, A2D is more inclined to reward positions that satisfy two conditions at once: the teacher and student disagree sufficiently clearly, and the teacher's supervision can cover the candidate tokens the student is currently considering. In other words, A2D is not simply looking for "questions the student cannot answer," but for “questions not yet mastered where the teacher can provide usable corrective signals.” This reward remains a proxy for learning value, however, and does not guarantee identical parameter-update gains from every high-scoring token.

Stage three: reinforcement learning improves question generation

A2D aggregates token signals into question-level rewards and trains the generator using Group Relative Policy Optimization (GRPO).

However, directly summing token signals can easily create new problems. For example, the generator may tend to propose questions that lead to extremely long answers, obtaining a higher reward simply by increasing the number of tokens; it may also repeatedly generate similar questions, increasing its score by repeatedly exploiting a small number of high-reward patterns. To avoid these phenomena, A2D further introduces length control and diversity control into the reward:

Article illustration

Here, the first term measures the cumulative learning signal induced by a question and reduces the inherent advantage of extremely long answers through sublinear length normalization; the second term penalizes excessively long answers; and the third imposes a diversity constraint based on how repetitive the generated questions are. Ultimately, the generator must not only learn to propose questions that elicit effective supervision from the teacher, but also avoid obtaining rewards merely by extending reasoning or repeating similar questions. Once training is complete, A2D uses the optimized question generator to synthesize training questions in batches, and then builds the final synthetic question bank through format filtering, lexical deduplication, and semantic deduplication.

Importantly, this bank stores questions only, not reference answers. Subsequent distillation restarts from the original student parameters: the student generates responses and the frozen teacher supplies token-level supervision through standard OPD. Generator optimization requires no external question data, human-labeled answers or additional reward model. A separate embedding model is used only for semantic deduplication after generation, not to calculate reinforcement-learning rewards.

Experimental results: from mathematical reasoning to code generation,

how much does optimizing questions help?

To verify the effectiveness of A2D, the research team conducted experiments on multiple student–teacher model pairs, covering mathematical reasoning, code generation, and agent tasks. The experiments not only compared A2D with conventional OPD using real data, but also considered directly using student-generated questions, selecting real questions with high KL disagreement, and other synthetic-data methods.

Mathematical reasoning: optimizing questions alone substantially narrows the teacher–student gap

The mathematical experiments used three student–teacher pairs of different sizes and types, including Qwen3 and DeepSeek-R1-Distill models, evaluated on six benchmarks including AIME, AMC, BRUMO and MATH500.

Article illustration

Across all three pairs, A2D achieved the highest six-benchmark average among the compared methods.

For Qwen3-1.7B, the original student averaged 25.7. OPD using real mathematics training data raised that to 43.0, and A2D reached 46.2, recovering approximately 90.2% of the teacher–student performance gap. For DeepSeek-R1-Distill-Qwen-1.5B, A2D averaged 51.6, close to the teacher’s 51.7, with reported gap recovery of 99.9%. For the larger DeepSeek-R1-Distill-Qwen-7B, A2D also achieved the highest average, improving the original student’s 58.1 to 64.5.

These results show that a specially optimized question generator can construct effective knowledge-distillation data for student models of different sizes.

Code generation: a 4B student can outperform its teacher on some tasks

Code experiments examined more complex program-generation scenarios on six benchmarks, including LiveCodeBench, MBPP+, APPS and CodeContests, using Qwen3 and Qwen2.5-Coder models.

Article illustration

One notable experiment used Qwen3-4B-Instruct-2507 as teacher for a Qwen3-4B student.

The original student averaged 28.7 across six benchmarks. Vanilla OPD with real training data reached 34.5, while A2D improved to 39.6, corresponding to 82.9% recovery of the teacher–student gap. On LiveCodeBench, A2D raised the student from 21.6 to 35.9, above the teacher’s reference score of 33.7. This does not mean the student surpassed the teacher overall: gaps remained on other coding benchmarks.

The result suggests that questions effectively eliciting supervision between teacher and student can enable a student to exceed the teacher’s reference performance on specific tasks, without changing the teacher. With a stronger Qwen3-30B-A3B teacher and Qwen2.5-Coder model combinations, A2D also obtained the highest average distillation scores, although gains varied markedly by pairing.

This also suggests that a stronger teacher does not necessarily mean better distillation. Which questions let the student receive effective supervision matters too.

Agent tasks: improving interactive decisions beyond mathematics and code

Article illustration

Article illustration

On these three tasks, A2D scored 46.3, 49.4 and 30.0, averaging 41.9. Compared with unoptimized Student-Generated OPD, the average rose from 37.3 to 41.9 and teacher–student gap recovery increased from 79.2% to 95.2%.

Question optimization using teacher–student distribution signals is therefore not limited to mathematics and code with clear reference answers; it can also apply to agent scenarios involving multi-turn interaction and decisions.

Why does A2D work?

The key is valuable supervision, not difficulty alone

Beyond the downstream training gains, a more fundamental question remains: why are these synthetic questions more effective than ordinary training questions? The team investigated through examples, token-level signal statistics and correlations with learning value.

Finding one: simpler questions may elicit denser learning signals

People often intuitively assume that the more complex a training question is, the more it will help improve a model's reasoning ability. But A2D's case analysis offers a different observation.

Article illustration

The paper compares a real mathematics question from DAPO-17k-Math with an A2D-generated question. The student answered both incorrectly, yet the synthetic question elicited denser teacher supervision along a shorter trajectory. The real question produced a 2,002-token response with an average token learning signal around 0.117. The synthetic question produced only 1,241 tokens but an average signal around 0.172, about 1.5 times higher. Surface complexity therefore does not directly indicate distillation value. What matters is whether the question repeatedly leads the student into states where the teacher can correct it effectively.

Finding two: 20% shorter answers, 28% more cumulative signal

To move beyond isolated examples, the team compared a thousand real mathematics questions with a thousand A2D synthetic questions.

Article illustration

Average student-response length fell from about 1,280 tokens to 1,021, approximately 20%. Yet cumulative support-aware learning signals rose from 141.7 to 181.7, an increase of around 28%. In addition, the researchers divided the answers into multiple intervals based on relative position to analyze token signals at different stages of generation. They found that questions generated by A2D maintained a higher average signal level throughout the entire answer, rather than producing peaks at only a few positions.

This means that A2D is not simply making the student generate more tokens and thereby accumulate more supervision; instead, it continually elicits denser learning signals within shorter trajectories. From the perspective of knowledge distillation, this offers a new direction for improving the efficiency with which training signals are used.

Finding three: can teacher–student disagreement really predict learning value?

One core assumption of A2D is that differences between the teacher's and student's distributions can be used to measure the potential value of training questions. But merely observing that questions with high disagreement produce higher rewards is not enough to prove that those questions genuinely have training value. The research team therefore conducted additional empirical analysis, directly examining the relationship between distributional signals and the gains obtained after updating the student's parameters.

Article illustration

On 1,500 real mathematics questions, researchers measured average teacher–student disagreement for each question and the learning gain after one distillation update using it. Gain meant not whether that question was answered correctly, but the reduction in teacher–student KL difference on fixed probe questions after updating.

Average disagreement and single-step learning gain had a Spearman correlation of 0.727. Controlling for response length, pass rate and entropy, the partial correlation remained 0.386. For the support-aware signal combining disagreement and compatibility, raw correlation reached 0.750, with partial correlation 0.399. Dividing questions into five groups by signal strength, the highest-signal group produced an average probe-KL reduction approximately 2.15 times that of the lowest group. Further experiments with a fixed token budget also found that, overall, question groups with higher distributional signals could produce better downstream training results, although this relationship was not strictly monotonic.

These experiments provide more direct evidence for A2D’s central idea: even without knowing the reference answer to a question, the probability distributions of the teacher and student during generation can provide effective signals for estimating that question's distillation value. Note that correlation does not mean KL divergence can fully predict the gains from parameter updates. More precisely, the research results show that distributional signals can serve as predictive proxy indicators of learning value.

Ablations: why not use KL divergence alone?

A2D's reward contains several components: disagreement between the teacher and student, compatibility, control of answer length, and the diversity of generated questions. Are all of these design choices necessary?

Article illustration

The team separately removed Compatibility, training-time diversity constraints and explicit length penalties, evaluating final distillation on AIME24, AIME25 and AIME26. The full reward achieved the highest scores on all three: 45.0%, 35.8% and 29.2%. Removing any component reduced performance.

On AIME25, removing compatibility, diversity or the length penalty reduced accuracy by 6.2, 6.0 and 6.6 percentage points respectively. Optimizing questions is therefore more than maximizing KL divergence. Without compatibility, the generator may favor large differences that cannot become useful supervision in the current distillation setup. Without length and diversity controls, it may earn rewards through verbose or repetitive output. Effective question-generation objectives must consider both how much knowledge differs and how much of that difference the student can use.

Is a stronger question-generating model enough?

Another natural question arises: since the teacher model is stronger than the student, why not directly have the teacher generate training questions? A2D further compares different combinations of initializing the question generator from the student or the teacher, and of whether targeted reinforcement-learning optimization is performed.

Article illustration

The experiments found that directly using a stronger teacher model to generate questions did not achieve the best distillation results on every benchmark. By contrast, reinforcement-learning optimization targeted at a particular student–teacher pair could improve the quality of the questions generated under both initialization approaches. Among these, A2D initialized from the student and optimized with RL showed strong competitiveness on multiple code benchmarks.

This supports the paper’s central argument: good questions depend not only on model capability but on how much effective supervision they elicit for a particular student–teacher pair. Different students have different knowledge boundaries, so the same question may have different distillation value. Synthetic-data generation may therefore need to move from universally high-quality data toward adaptive optimization for specific model capabilities and learning needs.

Research implications: data’s value may depend on what it lets a model learn

For a long time, one important direction in research on synthetic data for large models has been how to generate more correct, more complex, and more diverse training samples. A2D offers another perspective. For On-Policy Distillation, the training questions themselves do not need to carry reference answers. The actual supervision comes from the teacher's probability distributions along the trajectories generated by the student. This allows the question generator to move away from dependence on rewards for answer correctness and instead optimize the supervision signals that the questions can elicit.

From this perspective, the research significance of A2D lies not only in proposing a new method for generating synthetic questions, but also in re-examining how the value of training data for large models is defined: whether a piece of data is worth using for training may depend not only on what the data itself contains, but also on what kinds of learning opportunities it can bring to the current model.

Of course, this direction also has problems that remain to be resolved. For example, as the student's abilities continue to improve, does the distribution of the most valuable questions also need to be continually updated? Can a learning-value metric constructed for a fixed student–teacher pair transfer between different model pairs? When the teacher itself also has deficiencies in its knowledge, how can valuable disagreements for supervision be further distinguished from unreliable teacher preferences? All of these questions require further research.

Conclusion

Ask to Distill turns a key step in knowledge distillation—the selection and construction of training questions—into a directly optimizable learning process.

Unlike conventional synthetic-data methods, A2D does not depend on reference answers. It uses teacher and student distributions along generated trajectories to identify questions likely to elicit effective learning signals, then trains models to generate them through reinforcement learning.

From mathematical reasoning and code generation to agent tasks, the experiments show that optimized question distributions can produce better average distillation results. Shorter answers, denser supervision signals, and the positive correlation between distributional disagreement and actual learning gains also provide further support for this approach.

Perhaps improving models requires not only showing them more answers, but teaching them to ask questions truly worth taking to a teacher.

This article was originally published by Synced. Author: Synced.

Related topics

After reading

Join the AINRK community

Read how to join the community and connect with people working in AI.

Community information (opens in a new tab)