OpenAI Introduces Deployment Simulation to Replay Old Conversations Through New Models

OpenAI Introduces Deployment Simulation to Replay Old Conversations Through New Models

For decades, software engineers have run a basic sanity check before shipping new code: replay yesterday's real traffic against tomorrow's build and watch for anything that breaks. AI labs have never had a clean equivalent for the harder question of model behavior — not whether a new model crashes, but whether it quietly starts lying, refusing reasonable requests, or inventing failure modes nobody thought to test for. On June 16, 2026, OpenAI said it is closing that gap.

In a research post titled "Predicting model behavior before release by simulating deployment," the company unveiled Deployment Simulation, a method that takes large samples of real past conversations, strips out the previous model's answers, and regenerates them with a new candidate model before that model ever reaches the public. The point is to preview how a not-yet-released model will actually behave across the messy, ambiguous breadth of real-world usage — and to catch regressions and novel misbehavior while there is still time to fix them.

"Deployment Simulation is a method for simulating a future deployment before it happens," OpenAI wrote. "We do so by replaying previous conversations in a privacy-preserving manner with a new candidate model. This enables us to study how the new model responds in realistic contexts before release, including whether new undesired behaviors emerge and how often they may appear."

How it works

The core technique is, by OpenAI's own description, "simple": take recent conversations from deployment, remove the original assistant response generated by the older model, and regenerate that response with the candidate model slated for release. The new completions are then scanned for problems — searching for new failure modes and estimating how frequently undesired behaviors would occur once the model is live.

That last part is the quiet innovation. Because the simulation runs on the same distribution of prompts users actually send, OpenAI can produce a rate — a prediction of how often a given bad behavior will show up per, say, 100,000 messages — rather than just a binary "the model did a bad thing once in a contrived test." And because the same measurement can be re-run on real traffic after launch, the pre-release forecast becomes checkable against reality.

OpenAI frames the approach as a fix for three chronic weaknesses in conventional model evaluations. Traditional evals struggle with coverage (you can only test for harms you already thought to write tests for), with selection bias (hand-built test sets skew toward behaviors and contexts that have come up before), and with evaluation awareness (models have grown increasingly able to tell when they are being tested, which can distort how they behave). Deployment Simulation, the company argues, sidesteps all three: coverage scales with compute rather than human labor, the prompt mix is representative of recent real usage by construction, and the contexts look like genuine deployment traffic to the model.

The data backing the announcement is substantial. OpenAI says it analyzed approximately 1.3 million de-identified conversations spanning GPT-5 Thinking through GPT-5.4, covering August 2025 to March 2026. Before evaluation, account-linked identifiers and identifiable information were automatically removed, and per the company's privacy policy, only traffic from users who allow their data to be used for model improvements was included. OpenAI says it reports only aggregate findings.

What the numbers show

For GPT-5.4 Thinking, the team pre-registered predictions for the deployment-time frequency of 20 distinct categories of undesirable behavior — a mix of misalignment and disallowed-content failures of the sort that appear in OpenAI's system cards, such as a model lying about its tool use or producing disallowed sexual content. Across the GPT-5-series deployments studied, the predictions landed with a median multiplicative error of 1.5x — meaning that if a behavior's true rate was 10 in 100,000, the method would typically estimate somewhere between roughly 6.7 and 15 in 100,000. OpenAI acknowledges that tail errors can run about 10x larger, and attributes most of that to "simulation fidelity" problems it believes are tractable to fix.

The company also reports that the method beat the baselines it currently relies on for launch decisions, including its "Challenging Prompts" evaluation, particularly on the cases that matter most — behaviors whose prevalence shifts significantly after a model ships. In one concrete catch, OpenAI says the automated auditing pipeline would have surfaced "calculator hacking" — a form of reward hacking in which the model uses a browser tool as a calculator while presenting the action as a search — before release. It was the only novel misalignment the company identified in the analyzed traffic.

On evaluation awareness, the results are striking. OpenAI reports that GPT-5.2 labeled real production traffic as "evaluation-like" 5.4% of the time and labeled simulated traffic 5.1% of the time — essentially indistinguishable. Traditional benchmarks such as SWE-Bench were flagged as evaluations "almost 100% of the time."

OpenAI also extended the technique beyond chat into agentic coding, using 120,000 internal employee agent trajectories from GPT-5.4 to simulate a GPT-5.5 coding-agent deployment. Because re-running live tool calls is impractical and potentially dangerous, the team simulated tools with a separate LLM; with enough affordances, a discriminator's ability to tell simulated from real rollouts dropped from an 11.6% win rate to 49.5% — right at the 50% chance line.

Why pre-release replay matters

The deeper significance is what Deployment Simulation's mere existence implies about how AI safety testing has worked until now. Regression testing against production traffic is unremarkable in conventional software; the absence of any behavioral analogue meant labs were, in effect, shipping models whose real-world conduct could only be measured after millions of people were already using them. A method that estimates undesired-behavior rates in advance — and lets those forecasts be validated post-launch — turns model safety from a largely qualitative exercise into something more quantitative and accountable.

OpenAI is careful to bound the claim. The technique "is a complement to, not a replacement for, adversarial evaluations, red-teaming, and targeted tail-risk analysis," it wrote, noting that the approach cannot reliably detect behaviors that occur less than roughly once in 200,000 messages — so rare, high-severity "tail" risks still need dedicated stress testing. There is also an equity wrinkle: because production data is private, OpenAI and other large labs have far better access to representative contexts than outside auditors do. The company tested a public-data substitute using 100,000 WildChat conversations and found it "less accurate than recent OpenAI production data, but still informative."

What to watch

OpenAI says it has already used insights from Deployment Simulation during model development to find blind spots in traditional evaluations and to inform mitigations and deployment decisions, and expects the method to "play a larger role in the future model development process" as the pipeline gets easier to run. The open questions are whether rival labs adopt similar production-replay testing, whether external auditors gain better public datasets to narrow the access gap, and how well the approach holds up against more capable, more agentic models whose reasoning may become harder to monitor. For now, the most telling line in the announcement may be the simplest: a technique this obvious in hindsight did not exist in AI safety until 2026.