AI news

OpenAI Explains Its Performance Factory: Codex Investigates, Engineers Approve Fixes

By 新智元0 viewsSource: 新智元
OpenAI uses Codex to investigate ChatGPT performance issues while engineers approve fixes.

OpenAI describes a pipeline linking alerts, Codex investigations and proposed fixes, adversarial review and post-deployment checks. It reports over 1,700 investigations and 600 root-cause cases in just over three months; engineers still approve deployment.

Editor's fact-check (November 22, 2026): OpenAI's original report (opens in a new tab) describes more than 1,700 investigations and over 600 cases with identified root causes in just over three months, not 600 distinct bugs that have all been fixed. In a sample of 100 root-cause suggestions, 70% were accurate; the other 30% included analyses needing refinement or lacking sufficient causal evidence. They cannot all be called confirmed errors, nor is this a universal current error rate. AI can investigate, propose and review, but engineers currently still decide whether fixes are deployed. More autonomous deployment remains a future area of exploration.

Codex helping OpenAI build ChatGPT faster is already a familiar story.

On November 19, OpenAI went a step further in an official post: Codex is now also helping it keep ChatGPT fast.

AI is not just helping build AI; it is beginning to take over routine maintenance too.

Article illustration

When a ChatGPT conversation suddenly breaks or an image fails to load, AI may already be investigating the cause behind the scenes.

Behind that is an internal system OpenAI has just disclosed called the Performance Factory. When production slows or errors appear, Codex automatically takes over: investigating causes, proposing fixes and watching the data after deployment to check the results.

Article illustration

From alerts to AI investigation and fix proposals, then human review and deployment, followed by AI verification of recovery. If recovery is incomplete, the process returns to investigation.

In just over three months, the Performance Factory followed more than 1,700 incident investigations, averaging 17 responses a day. In more than 600 cases, it identified the underlying root cause.

When ChatGPT Stalls,

AI Is First on the Scene

The most direct reason ChatGPT keeps slowing down turns out to be that it changes too quickly.

Martin Spier, who leads ChatGPT's performance team, described two simultaneous pressures in a talk: More people are using ChatGPT, and engineers are updating code at an ever-faster pace.

By February 2026, more than 900 million people were using ChatGPT weekly. Back in October 2025, OpenAI had already disclosed a 70% increase in weekly merged pull requests, or PRs, among engineers using Codex.

Every new feature takes a small slice from the same speed budget.

Each individual effect may be tiny, but repeated updates add up to visible sluggishness on your screen.

An even more sobering observation from Spier was that software releases used to contain changes someone fully understood, but that default assumption no longer holds at OpenAI.

With more changes than human eyes can review, the hard work of investigating problems has to be handed to AI.

How does it investigate?

When ChatGPT slows or errors occur, monitoring raises an alert.

The system receives the signal, immediately creates a ticket and dispatches a Codex agent to the scene.

It first establishes two things: which metrics changed and when the changes began.

After establishing the timeline, it examines code and logs, follows request traces and reviews CPU and memory behavior in detail.

If the same problem tends to trigger multiple alerts, it also groups, consolidates and prioritizes them.

Codex also drafts a fix and submits it to engineers for review.

AI gradually pushed the team toward this pipeline. In the beginning, everything was investigated manually, and the performance team accumulated more than 30 specialized tools.

When Codex arrived, engineers brought it in as a partner, having AI sift through data while they interpreted the answers.

Later, the team turned experienced investigators' methods into reusable skills, allowing Codex to follow the process and produce a diagnosis when it received an alert.

As that continued, engineers realized they had become human message relays: Their most common daily task was copying alerts into Codex one by one.

So the team connected alerts directly to Codex, which automatically posted reports to Slack after investigating.

As investigation volume grew, every problem ultimately became a ticket moving through a workflow, with agents advancing it from station to station. The Performance Factory gradually took shape.

Step by step, AI took over the difficult, tedious work engineers disliked.

Article illustration

From manual investigation to system-dispatched tickets, the performance team gradually built an agent factory.

Three Wrong Diagnoses in Ten:

Why Trust AI to Fix Sluggishness?

When AI tells you it has found the cause, would you trust it blindly?

OpenAI itself has stumbled over that.

Early agents rushed to wrap up after finding the first plausible explanation. Later reviews showed that many of their conclusions did not hold up.

OpenAI manually sampled 100 AI root-cause judgments and found only 70 were accurate.

Of the other 30, some analyses needed more refinement, while others lacked sufficient evidence that they had identified the actual root cause.

If AI sends engineers down the wrong path, spending hours chasing a false cause wastes more time than not finding a cause at all.

OpenAI's solution was to add adversarial review to the diagnosis.

An independent Codex agent acts as reviewer, with one task: overturn the previous Codex agent's diagnosis.

In its official blog, OpenAI shared a real example of a diagnosis being contradicted.

A series of ChatGPT conversations were failing.

The investigating Codex agent looked into it, found an image-download error and immediately declared that the cause had been found.

But the Codex reviewer discovered that the internal service responsible for locating files had recovered long before, while image requests were still producing a steady stream of errors.

That hasty diagnosis was therefore sent back for further investigation.

Even if a diagnosis passes, the proposed fix must undergo the same review.

The Codex reviewer keeps pressing: Does the change really address the root cause? Will fixing the old problem introduce a new bug?

Expecting one AI never to make mistakes is unrealistic. Having two AI agents challenge each other can catch more conclusions that do not stand up.

For Some Fixes,

Humans Still Make the Final Call

Even with the safeguard of agents challenging each other, OpenAI acknowledges that prescribing the right remedy is much harder than finding the root cause.

Agent-proposed fixes sometimes overcomplicate simple problems or patch one issue by creating another.

Adding a cache is a classic example.

When responses are slow, another caching layer can indeed make them faster, but the cost is that users may see stale content.

Speed or freshness? A few code changes cannot settle that; it is an architectural tradeoff.

AI's limitation immediately becomes clear: It can calculate millisecond-level latency, but cannot make the final decision about human product experience.

OpenAI therefore maintains a final line of defense internally: Every fix must receive an engineer's approval before deployment.

Article illustration

A Codex-generated task report lists performance changes, root causes and proposed fixes for engineers to review.

After code goes live, agents take back the tedious monitoring work, and the task closes once the metrics have fully recovered.

In the Future, the Ones Fixing ChatGPT for You

May All Be AI

With so many agents examining code and logs in core backend systems, how are security boundaries maintained?

OpenAI keeps them on a tight leash: Each agent runs under a separate service identity with bounded permissions. Every action, including what it inspected and what it changed, can be traced precisely to the specific AI agent.

That is still just reactive firefighting.

Spier says the team also maintains specialized agents that act proactively.

They monitor CPU, memory, network requests and database queries around the clock, looking specifically for opportunities to extract better performance.

These agents are named after colleagues who excel in the corresponding specialties, such as the Ben agent and Brendan agent.

Leading engineers' expertise has been copied into tireless AI counterparts available 24 hours a day.

People imagining AI improving AI have often pictured a technological leap arriving suddenly one day.

Reality may be less dramatic.

It may be hidden behind a moment of lag, in repeated rounds of investigation, debate and revision between AI agents.

References:

https://openai.com/index/how-openai-performance-factory-keeps-chatgpt-fast/

This article was originally published on the WeChat public account 'AI Era,' by AI Era.

Related topics

After reading

Join the AINRK community

Read how to join the community and connect with people working in AI.

Community information (opens in a new tab)