Aalborg study examines how diffusion-language-model sampling affects training-data extraction

The paper compares masked completion with autoregressive models, finding that finer sampling steps can increase reproduction probability in matched tasks. The results do not establish that diffusion models are safer in every setting.

Figure 1: Personal-information extraction from LLaDA-8B. Token-by-token restoration reconstructs the target email address, while one-step restoration makes errors. Blue indicates matches, red mismatches, and 〈M〉 a mask. (Original paper Figure 1)
Large language models sometimes recite content they encountered during training verbatim, and this may include personal information such as email addresses and phone numbers. As diffusion language models move to larger scales, a question also arises: if a model no longer generates left to right, token by token, does it leak training data in the same way?
Autoregressive models continue a sequence, predicting the next token from preceding text. Masked diffusion language models instead fill gaps, recovering missing text across multiple rounds. This enables potential parallel generation and changes the routes by which training data can be extracted.
Researchers at Aalborg University in Denmark examine this systematically in Characterizing Memorization in Diffusion Language Models: Generalized Extraction and Sampling Effects.

- Paper: https://arxiv.org/abs/2603.02333
One notable finding: changing only sampling steps during generation can substantially change extraction risk for the same diffusion language model. In the paper's personal-information completion tasks, diffusion models leak less than similarly sized autoregressive models, but finer generation steps often increase exact reproduction of training content.
From completing a suffix to filling arbitrary gaps:
How should diffusion-model memory be measured?
A common approach to language-model memory supplies the first part of training text and checks whether the model reproduces the second part exactly.
Diffusion models can also use context on both sides of gaps. Mask positions, restoration order and tokens restored per round can all affect results. One generation's success or failure cannot fully characterize this behavior.
The researchers propose a generalized probabilistic extraction framework: given partly visible text and repeated attempts, measure the probability of recovering all masked original content. It supports arbitrary masking and restoration orders, with traditional prefix continuation as a special case.
They also test the feasibility of probability estimation. Across 200 samples, estimates from 1,024 random-mask samples are close to empirical results from 100,000 random generations, supporting lower-cost evaluation.

Figure 2: Estimated extraction probabilities versus those measured through many actual generations. (Original paper Figure 2)
Why can more generation steps
Make leakage more likely?
You can think of this process as filling in a familiar passage of text. If all the blanks are filled at once, the clues available at each position are limited. If a few positions are filled in correctly first and those new clues are then used to continue filling in the passage, the remaining content may be easier to recover.
The paper formalizes this mechanism. Assuming additional correct context does not reduce recovery probability for the remaining content, keeping restoration order fixed while splitting batches into finer steps cannot decrease the probability of reconstructing the entire original.
These conditions matter: the result concerns correct context and a specific refinement relationship. It does not mean more steps necessarily worsen leakage under every sampling strategy.
Experiments show a corresponding trend. The team selects 200 qualifying mask configurations and performs many generations with one, two, five and ten steps, as well as token-by-token restoration. Exact reconstruction generally becomes more successful as steps become finer.
The parameters do not change; the opportunity to extract existing memories does. Sampling steps therefore affect not only speed and quality but also training-data extractability.

Figure 3: Finer sampling increases exact reconstruction success relative to one-step generation. Shading indicates 95% confidence intervals. (Original paper Figure 3)
Completing the same email addresses:
How do diffusion and autoregressive models compare?
The researchers train diffusion and autoregressive models with 1.1B parameters each, aligning datasets, tokenizers, compute budgets and other conditions. Compared models then follow a common adaptation process on Enron emails.
Models receive the 100 tokens preceding a target email address or phone number and try to complete it. Email prompts containing other email addresses are filtered to reduce extra clues.
With 3,000 email-completion samples and a budget of up to 10,000 queries, the results are:

Table 1: Adapted from the paper's Table 1. Values count samples reaching specified success-probability thresholds within the query budget, rather than leakage rates for a single generation.
Two results stand out.
First, among matched 1.1B models, even token-by-token diffusion reaches the 50% extraction-probability threshold for only 16 email samples, versus 213 for the autoregressive model.
Second, switching LLaDA-8B from one-step to token-by-token restoration raises the corresponding count from nine to 179, nearly twentyfold. Sampling settings alone create substantial risk differences within one model.
These are matched prefix-completion results, however. They do not establish that diffusion models are safer in all settings, particularly their native bidirectional gap-filling setting.
Recovered content:
Memory or a correct guess?
Completing an email does not necessarily demonstrate memorization. Common formats and language patterns may also let models guess some content correctly.
The team therefore compares training-seen Enron email with same-domain TREC 2007 Spam email excluded from training, sampling 10,000 text windows from each and evaluating reconstruction probability.
The training-data distribution has higher reconstruction probabilities overall, indicating that the evaluation detects differences associated with training exposure and supporting a memorization component in extraction.
Lower extraction does not mean no memory: under current prompts and sampling conditions, existing memories may simply be harder to retrieve.

Figure 4: Reconstruction probabilities for training and unseen email. Blue is Enron training data; orange is unseen TREC data. (Original paper Figure 4)
From architecture to sampling:
What should privacy evaluation consider?
The work offers three lessons.
Evaluate generation settings alongside models. The same parameters can show different extraction risks with different sampling steps.
A failed attempt does not rule out extraction through repeated queries. Reports should include query budgets, success probabilities and masking patterns.
Diffusion models' lower leakage merits study, but its boundaries need verification. Whether it persists with bidirectional gap filling, instruction tuning or preference optimization requires further experiments.
The study brings an often-overlooked variable into safety evaluation: training determines what models encounter, while generation influences how readily it reappears.
Authors
The work is by Aalborg University researchers in Denmark. Xiaoyu Luo and Wenrui Yu are co-first authors, Qiongxiu Li the corresponding author, and Johannes Bjerva a co-author. The team studies language-model memorization, privacy and safety.
This article comes from Synced. Authors: Xiaoyu Luo and Wenrui Yu.
After reading
Join the AINRK community
Read how to join the community and connect with people working in AI.




