AI guides

D-DOIT guides frozen diffusion models toward biological-sequence rewards through inference-time sampling

By LRST0 viewsSource: 新智元
D-DOIT reward-guided discrete diffusion for biological sequences

D-DOIT targets a discrete Doob transform and approximates guidance through finite candidates and reward queries. It reports gains on predictive biological benchmarks, with wet-lab validation and inference-cost trade-offs still unresolved.

Editorial correction (November 16, 2026): The generative backbone is not updated, but practical sampling uses biased approximations and reward queries. Results are predictive metrics, not wet-lab success rates. Time-direction wording and formatting errors have also been corrected.

Discrete diffusion has become an important tool for biological-sequence generation over the past two years. Unlike autoregressive generation from left to right, masked discrete diffusion starts with a fully masked sequence and progressively reveals bases or amino acids at each position.

It naturally suits finite-alphabet data such as DNA and proteins and has shown promising generation quality for regulatory-element design and protein inverse folding.

But in real scientific research, simply sampling from the pretrained distribution is not enough. Researchers designing enhancers want sequences with higher activity in the target cells; researchers designing proteins want sequences that are more stable and can fold back into a specified structure.

What is needed is a trained generator that produces sequences toward a task reward.

This is harder than it sounds. Fine-tuning can improve rewards but adds training and often needs differentiable rewards. Training-free methods either also require gradients or use particle/search procedures whose cost grows with trajectory length or search width. Biological rewards such as Enformer predictions or structure-prediction scores are often black boxes that return values without usable gradients.

Can we adapt a frozen discrete diffusion model to any black-box reward solely by changing its inference-time sampling rules, without modifying the model parameters and without requiring reward gradients?

A NeurIPS 2026 paper approaches reward adaptation as distribution transport, using the classical stochastic-process tool the Doob h-transform to derive its form for masked discrete diffusion.

The researchers build D-DOIT—Discrete Doob-Oriented Inference-time Transformation—a training-free sampler requiring only reward values.

For HepG2 enhancer design, high-budget D-DOIT-Q reaches predicted activity of 6.19, above fine-tuned DRAKES at 5.61. For protein inverse folding, its joint structure/stability success is 68.1%, above training-free SMC at 63.6% and TDS at 62.9%.

Article illustration

Paper: https://arxiv.org/abs/2610.04938

Code: github.com/WeiminWu2000/D-DOIT

Project: https://weiminwu2000.github.io/blog/ddoit.html

Making frozen discrete diffusion follow rewards

Consider a sequence of length N. The forward process replaces each token with mask symbol m probabilistically; once masked, it stays masked. Reverse generation begins fully masked. At each step, a prediction head supplies a clean-token distribution for every masked position, and transition rules reveal some of them.

Article illustration

Figure 1: The generation process of standard masked discrete diffusion. Starting from a fully masked sequence, each step independently reveals a number of positions according to the transition rules. Positions already revealed are no longer changed, and this continues until a complete sequence is obtained.

Rows show the same sequence over time. The middle step samples only masked positions, carrying revealed positions unchanged into the next row.

α denotes the probability a clean token survives to forward time t and decreases with forward time. It starts near one; reverse generation starts from the noisy end, with alpha near zero.

Article illustration

Reverse generation begins fully masked. Prediction head μ gives clean-token posteriors at masked positions. Substitution into the analytical reverse posterior gives the one-step transition from t to t−1, independently for each coordinate j:

Article illustration

The three lines correspond to remaining masked, revealing a specific token, and preserving an already revealed position. The reverse chain factorizes across coordinates:

Article illustration

The equations show two properties. First, positions unmask independently (the product in the last expression): local and fast, but reward-unaware. Second, unmasking is irreversible (the third transition line): later steps cannot change a base once filled.

Training-based methods: effective but costly

Methods such as DRAKES update the backbone through reinforcement learning or reward fine-tuning. They can strongly improve rewards but add training costs and hyperparameter tuning and usually require differentiability. A new task or reward requires retraining. Classifier-Free Guidance (CFG) in the later tables also belongs to this category.

Training-free methods: gradients or trajectory-dependent cost

Other methods freeze the model and guide inference. Classifier Guidance (CG) needs reward gradients. Particle methods such as Sequential Monte Carlo (SMC) and Twisted Diffusion Sampler (TDS) reweight trajectory likelihood ratios, with history-dependent costs. Soft Value-Based Decoding (SVDD) selects candidate states using soft-value estimates.

Each has strengths, but a unified exact target-distribution perspective is missing: what distribution should guidance sample, and how should each transition probability change?

Reward-tilted distributions

The discrete Doob h-transform

The target distribution

The frozen model defines a distribution p describing natural-looking sequences. The desired result is neither that distribution alone nor simply the highest-reward sequence, which may exploit a predictor without resembling real sequences.

The target balances model-defined naturalness with high rewards. Formally, it is a reward-tilted distribution:

Article illustration

Here, β is temperature and the denominator normalizes the distribution. The pretrained model provides a sequence prior while high rewards are favored. Smaller β pursues rewards more aggressively; larger β stays closer to the original model, retaining naturalness.

This matters for biology: optimizing predictor scores should not produce sequences unlike real enhancers.

The continuous case: an additive drift correction

The Doob h-transform is a classical tool in stochastic processes: conditioning a Markov process on some event concerning the future produces a process that is still Markovian, but with an additional correction determined by h. Here, h is the conditional expectation of the future event from the current state—specifically, the terminal reward weight's conditional expectation.

In continuous diffusion, reverse sampling is a stochastic differential equation (SDE), and the Doob correction is added to its drift:

Article illustration

Equivalently, the sampling score changes from ∇log p to ∇log p+ ∇logh.

Discrete Doob transform: multiplicative reweighting, not additive drift

For masked discrete diffusion, the same Doob h-transform takes a different form. The discrete h-function is defined as the conditional expectation of the eventual terminal reward weight given the current partially denoised state:

Article illustration

It represents the current intermediate state's future value, or reward-to-go: the reward obtainable by continuing the original denoising process. Theorem 2.1 shows that reweighting each reverse transition kernel by the ratio of next-state to current-state h values,

Article illustration

makes the transformed chain's terminal marginal exactly the reward-tilted distribution, without changing model parameters.

In continuous diffusion, the correction is a drift term added to the score. Masked discrete diffusion instead has a categorical state space and irreversible reveal transitions, without a Euclidean score field.

The discrete transform therefore acts at the transition-kernel level, applying multiplicative reweighting to each candidate next state.

D-DOIT

Making the exact transform executable

The exact transform requires each candidate's future value h, whose exact computation needs many future denoising rollouts per candidate and is infeasible. To estimate h, D-DOIT uses two approximations and an optional module.

Article illustration

Figure 2: D-DOIT. Left, standard masked diffusion reveals positions independently, cheaply but without reward awareness. Right, one guided step proposes M candidates (M=3 shown), completes each in one prediction-head forward pass, scores them and samples by h. Below, optional late best-of-K refinement (K=4 shown).

Approximation one | Prediction-head h estimation in one forward pass

A masked diffusion model's prediction head already provides a distribution over clean tokens for every masked position. The researchers take advantage of this property: for each candidate next state, they fix revealed positions and complete all remaining masks in one prediction-head pass, then query the reward function to estimate its h value, using temperature γ to rescale the head during completion:

Article illustration

This has two benefits:

First, it avoids nested rollouts—running denoising from the current state all the way to a full sequence—and needs only one forward pass per candidate.

Second, rewards are queried only on completed sequences, without differentiability or backpropagated gradients, so any black-box scorer can be connected.

Approximation two | Doob-weighted selection among finite candidates

At each guided step, D-DOIT samples from the model's reverse kernel at temperature τ to propose M next states. These M proposals share one forward pass; each is then completed and scored to estimate its h value. The next state is selected with probability proportional to those estimates raised to ω, by sampling among candidates:

Article illustration

There are two temperatures here that are easy to confuse. They operate on the same prediction head but have different responsibilities, and both rescale that head by taking powers. τ applies to proposals and controls the diversity of the M candidates:

Article illustration

γ applies to completion, controlling sharpness of the full sequence used for scoring:

Article illustration

When τ=1, this is a finite-candidate self-normalized approximation of the Doob kernel. Parameter ω controls value-following sharpness and, together with β, corresponds to effective inverse temperature ω/β. Each guided step costs 1+M model forward passes and M reward queries.

Optional module | Late best-of-K: hedging irreversible reveal decisions

Revealed tokens cannot be changed, so later denoising cannot directly correct early choices. Optional late best-of-K adds candidates only in later stages and cannot guarantee repairs to fixed positions.

There are two main reasons to place the split point near the end. The first is diversity: early in denoising, many positions have not yet been revealed, and sampling itself sends trajectories in markedly different directions. At that stage, making K copies then adds limited diversity.

Later, most tokens have already been fixed and little randomness remains, so the trajectory is almost bound to converge to a single result. Branching at this point supplies the variation that is otherwise missing. The second consideration is compute: the later the split point, the fewer steps each copy must recompute, so the K-fold overhead applies only to the tail, not the whole trajectory.

Two configurations: LBOK-S and Q

Both use the same sampler with different budgets. Per sequence in the HepG2 setting:

  • D-DOIT-LBOK-S: small candidate budget (M=9) plus late best-of-K (K=4), about 1,970 model forwards, near TDS runtime (about 1.16×), for practical low-budget use;
  • D-DOIT-Q: more inference budget for candidate evaluation and refinement (M=52, K=8), about 54,272 model forwards, exploring the frozen backbone's quality ceiling.

Results

D-DOIT is evaluated on three biological-sequence tasks, keeping all masked discrete diffusion backbones frozen.

HepG2 enhancer design: training-free competition with trained methods

The DRAKES HepG2 benchmark generates 200-base-pair enhancers with high predicted HepG2 activity and natural statistics. Metrics include Pred-Activity, chromatin accessibility (ATAC-Acc), 3-mer correlation and JASPAR motif correlation.

Article illustration

Figure 3: HepG2 results. Left, median predicted activity; right, mean chromatin accessibility.

Article illustration

Figure 4: Statistical naturalness of generated HepG2 sequences. Left, 3-mer correlation, which can be negative; right, JASPAR motif correlation.

Notable results:

  • D-DOIT-Q beats every training-free baseline on all four metrics, with predicted activity 6.19. It also exceeds fine-tuned DRAKES on Pred-Activity, 3-mer Corr and JASPAR Corr, trailing slightly only on ATAC-Acc (91.7% versus 92.5%).
  • D-DOIT-LBOK-S, with runtime near TDS, gives comparable predicted activity (4.56 versus 4.58) while raising ATAC-Acc from 40.0% to 65.0% and JASPAR Corr from 0.798 to 0.948. Gains thus extend beyond predictor-score optimization toward sequences resembling real high-activity enhancers.
  • At similar wall-clock time, D-DOIT-Q's means exceed SVDD-PM on all four metrics. Against time-matched Frozen Best-of-N, it improves activity, accessibility and 3-mer correlation, while the latter has higher JASPAR correlation.

Cell-type-specific enhancers: stricter controllable generation

Ctrl-DNA requires sequences active in a target cell line and silent in others, covering HepG2, K562 and SK-N-SH. Specificity ΔR is target score minus the mean nontarget score.

Article illustration

Figure 5: Ctrl-DNA results. Left, specificity ΔR; right, motif correlation. Frozen Best-of-708 is an endpoint-search control matched to D-DOIT-Q's runtime.

Among diffusion-guidance baselines, CG, SMC and TDS barely improve specificity. LBOK-S raises ΔR from 0.002 to 0.033 at near-TDS runtime, with Motif Corr increasing from 0.573 to 0.648. D-DOIT-Q reaches ΔR=0.119 at some cost to motif preservation.

Time-matched Frozen Best-of-N also achieves most gains on this task. D-DOIT-Q and Frozen Best-of-708 have very similar means.

Protein inverse folding: transfer across modalities

To test generality across sequence spaces and rewards, the team uses the DRAKES protein inverse-folding benchmark:

generate amino-acid sequences for a target backbone, assessing predicted stability (Pred-ddG, predicted folding free-energy change) and structural consistency (scRMSD, self-consistency root-mean-square deviation). Success means meeting both thresholds.

Article illustration

Figure 6: DRAKES inverse-folding results. Left, joint stability/structure success; right, predicted stability.

D-DOIT achieves 68.1%±3.5% joint success, above SMC (63.6%) and TDS (62.9%), with the highest predicted stability; TDS has slightly better structural fidelity. The configuration (M=52, K=8, one completion per candidate) was fixed before test evaluation.

Robustness

  • Completion quality: one completion per candidate is practical. Two and three raise final guided rewards from 5.272 to 5.764 and 5.978, but add 41% and 81% runtime. Single-step and multistep completions yield very similar h distributions.
  • Reward-predictor robustness: using Enformer, CNN or MLP guidance improves all four metrics over unguided sampling. Pred-Activity reaches 4.491, 2.203 and 1.544 respectively, versus 0.371 for the frozen model, with improvements retained under independent held-out evaluators.

Limitations and outlook

The work has clear boundaries:

  • Approximation bias: temperature scaling, weight sharpening and best-of-K selection move practical sampling away from the exact target. Late branching cannot correct revealed tokens.
  • Wet-lab validation: current evaluation uses predictive biological benchmarks and ultimately requires experimental verification.
  • Inference cost: the highest-quality configuration requires substantial inference expenditure, and that cost must be paid again for every generation. How to balance rewards against compute therefore remains a practical question.

Doob-style transition reweighting offers a general framework for sequence-level reward guidance of discrete generators. It is not restricted to DNA or proteins: the approach applies to masked discrete diffusion when complete sequences can be scored.

Further questions worth exploring include how to learn more accurate h-estimators, how to perform better backtracking and correction under the constraint that reveals are irreversible, and how to combine the approach with training-based methods to amortize costs.

Conclusion

Returning to the opening question: can we adapt a frozen discrete diffusion model to any black-box reward simply by changing inference-time sampling rules? D-DOIT's answer is yes, with an exact theoretical object to guide how this should be done: the discrete Doob h-transform.

It tells us that guidance does not mean adding a direction in discrete space. Instead, it means reweighting transition probabilities according to the future value of each candidate.

References:

https://arxiv.org/abs/2610.04938

This article comes from AI Era. Author: LRST.

After reading

Join the AINRK community

Read how to join the community and connect with people working in AI.

Community information (opens in a new tab)