AI news

OpenAI Details Anti-Distillation Safeguards and Internal Reasoning-Extraction Tests

By 新智元0 viewsSource: 新智元
OpenAI describes defenses against adversarial distillation and protected-reasoning extraction.

OpenAI reports weekly enforcement against thousands to tens of thousands of accounts. In an internal adaptive black-box evaluation using six attacker models, the average success rate for extracting earlier protected reasoning fell from 54.0% to 0.2%.

Editor's fact-check (November 22, 2026): This article concerns unauthorized adversarial distillation. The 0.2% figure is the average success rate for extracting protected reasoning from an earlier request in an internal adaptive black-box evaluation involving six attacker models; it does not mean that all attacks are ineffective. The official account describes investigating and enforcing against networks of accounts that violate policies. It does not say that exposing one node inevitably causes all data in every associated account to be lost, nor does it establish that safeguards can never be bypassed. Discarding reasoning after conversation history is rewritten is an API protection mechanism, not a ban on normal prompt editing. The report does not establish the scope of the claim that these protections are 'fully rolled out on Sol.' OpenAI's original report (opens in a new tab)

OpenAI has declared war!

In the early hours of this morning, OpenAI released a major statement: 'Combatting adversarial distillation.'

Article illustration

On its face, this report shares technical details. In practice, it reads like a declaration of war.

OpenAI states it bluntly: Since this summer, to crack down on accounts taking its capabilities for free, it has banned as many as tens of thousands of accounts a week, even taking down entire networks in one sweep!

Now OpenAI has called on the US Congress to pass legislation strengthening export controls.

The report also lays out comprehensively how OpenAI's systems identify and dismantle coordinated distillation attacks.

Get ready: Another major wave of account bans is coming from OpenAI.

Tens of Thousands of Accounts Wiped Out Each Week

This time, OpenAI means business.

OpenAI says ChatGPT has carried out multiple large-scale account-banning campaigns this year.

It found that those trying to copy its expertise often use vast networks of accounts and distributed requests to piece together the models' underlying logic.

OpenAI shows no mercy: Once it catches them, it takes down the whole network of accounts!

OpenAI can ban thousands, or even tens of thousands, of accounts in these coordinated groups every week!

The rate of bans rose sharply this summer in particular.

Its aim is to make distillers obtain less material, pay more and work more slowly.

Note two key phrases: 'the entire network of accounts' and 'tens of thousands of accounts'.

In other words, once one node in your account network is exposed, the other associated accounts will also be punished by association, and all their data will disappear.

Article illustration

OpenAI says distillation, a gray-area practice everyone quietly understands, is far more than simple infringement: It can cause a serious safety crisis!

That is because distillation copies a model's powerful capabilities while leaving behind the safeguards OpenAI spent enormous sums building.

A model with those safeguards would never help write Trojan malware or provide recipes for biological and chemical weapons.

But once it is distilled into an open-source model without such boundaries, it becomes a weapon for hackers and terrorists.

That is why OpenAI launched this ruthless purge.

In internal tests, OpenAI deployed six AI attacker models.

They took turns using all kinds of tricky black-box attacks to coax information out of the defense system.

The defender, such as the GPT-6.2 family, must judge for itself: Is this a legitimate question, or a malicious attempt to obtain its reasoning for free?

The Core Secret, Chain of Thought, Must Be Defended

If you call OpenAI's API now, you can see only what it chooses to show you: a summary.

What distillers covet most is CoT. These unvarnished, raw chains of thought contain higher-level logic and generalization abilities absent from the answer itself. They are arguably the most valuable asset.

But OpenAI has now hidden the actual CoT.

To demonstrate this, OpenAI conducted a controlled experiment.

Starting from the same base student model, it trained separate versions on two types of data.

Group A learned from the original, complete, exact reasoning traces.

Group B learned from the reasoning summaries and final answers users normally see in the interface.

The results were, indeed, as expected.

After supervised fine-tuning, or SFT, on data including trajectories from Astra-level teacher models, Group A, trained on raw chains of thought, generally trounced Group B on cybersecurity benchmarks!

Article illustration

Article illustration

After both groups underwent another lightweight round of reinforcement learning, Group A still led at the largest inference compute budget. Only at smaller compute budgets did Group B narrowly pull ahead.

The experiment demonstrates that exact reasoning traces can teach a student model to generalize from what it learns!

Even when the extracted data concern only permitted defensive tasks, the student model can learn general skills applicable to offensive tasks.

You could say that frontier models' trump card is that fleeting 'thought in their mind' as they solve a problem.

Whatever it takes, OpenAI intends to lock that away.

OpenAI Builds Four Layers of Anti-Distillation Defense

So what exactly is OpenAI doing?

In the report, it reveals its anti-distillation techniques for the first time.

This is a four-layer network combining adversarial testing, model training, classifiers and API-level protections. OpenAI has made deep architectural changes.

Article illustration

Layer One: Preventing Capability Theft Through Model Training

Through adversarial training, OpenAI teaches models to recognize requests attempting to extract reasoning.

Once a malicious request is detected, the model firmly refuses to disclose its CoT.

At the same time, however, it can precisely distinguish legitimate user requests.

Why can it make that distinction so precisely?

Because its training data come from internal red-team/blue-team exercises and attack logs supplied by external security researchers.

Layer Two: Watertight Classifiers

OpenAI has also deployed exceptionally powerful output classifiers.

Most strikingly, it has not overlooked tool calls either.

That is because a model may quietly write its reasoning into tool-call arguments without showing it in the final answer.

Output classifiers can block those leaks directly!

Layer Three: API-Level Restrictions That Make Developers' Hearts Race

This is the most devastating move for the entire developer ecosystem.

In the newly released GPT-6.2 Astra and GPT-6.2 Sol, OpenAI has tightly bound a model's reasoning to the conversation history that produced it.

If you try rewriting an earlier prompt to see how the model changes its thinking, sorry: The API will invalidate all the reasoning that followed it!

That means the tactic of repeatedly tweaking context to squeeze out a little more reasoning has been rendered completely ineffective.

Layer Four: Isolation Across Models and Conversations

Specifically, older models with weaker defenses are strictly prevented from reading newer models' reasoning records.

OpenAI also closed a major loophole just this September:

Previously, hackers could forcibly move encrypted chain-of-thought content into another conversation and trick the model into reading the encrypted content aloud, even exposing another user's encrypted reasoning.

That route has now been blocked as well.

Under successive attacks from six AI models, OpenAI produced a formidable set of results.

In tests simulating real production conditions, when attacker models tried to extract protected reasoning from a previous conversational turn, their average success rate against GPT-6 Astra's old defenses was 54.0%.

Against GPT-6.2 Astra with its updated defenses, that figure fell to an astonishing 0.2%!

The success rate for extracting protected reasoning from the current conversational turn also plunged, from 22.9% to 0.9%.

GPT-6.2 Sol has now fully rolled out these protections, and the older GPT-6 Astra is also being urgently patched.

Article illustration

OpenAI also turns detected attack techniques into regression tests and feeds them back into model training.

It even plans to share these techniques with other companies for coordinated action across the industry.

And that is not all. OpenAI says: 'We plan to share more soon about recent campaigns we have disrupted.'

Perhaps more companies will be named.

Finally, OpenAI publicly addressed the US government, expressing full support for three bills:

The Chip Security Act

The AI OVERWATCH Act

The MATCH Act

On one front, OpenAI wants to use hardware export controls to restrict adversaries' access to the chips needed for frontier models.

On another, it wants anti-distillation defenses to prevent adversaries from extracting expertise from existing models.

It seems the time has come to forge a way forward through fundamental algorithms and innovation.

References:

https://openai.com/index/combatting-adversarial-distillation/

This article was originally published on the WeChat public account 'AI Era,' by AI Era.

Related topics

After reading

Join the AINRK community

Read how to join the community and connect with people working in AI.

Community information (opens in a new tab)