How OpenAI Reviews Codex With Codex

OpenAI’s own engineering team ships code reviewed primarily by AI agents, and the mechanics are public. Here is what they run internally, what you can install today, and how to copy the review loop into your repository this week.

What harness engineering actually measured

OpenAI’s harness engineering beta produced roughly one million lines of code from an empty git repository over about five months, with approximately 1,500 merged PRs and zero manually written lines — even the initial AGENTS.md was authored by Codex. The team grew from 3 engineers to 7, and throughput held at about 3.5 PRs per engineer per day, rising as the team grew (Harness engineering). The point of the experiment was not raw volume. It was to find out whether an agent-first workflow — where agents write, review, verify, and merge — can hold up under a real product’s quality bar. The answer they published is yes, with a specific review architecture doing the load-bearing work.

The review loop: local first, specialist second, human last

OpenAI instructs Codex to review its own changes locally, request additional specific agent reviews both locally and in the cloud, respond to any human or agent feedback, and iterate until all agent reviewers are satisfied — the loop the team calls the “Ralph Wiggum Loop.” Single Codex runs were observed working a single task for upwards of six hours (Harness engineering). The published quote is blunt: “Over time, we’ve pushed almost all review effort towards being handled agent-to-agent.” Human review still exists, but it enters late and selectively, gated by risk classification rather than applied uniformly. The Pragmatic Engineer walkthrough of OpenAI’s agentic software factory describes this as the core structural difference from conventional review: review is not a human bottleneck with AI assistance bolted on, it is an agent-to-agent process with humans as the escalation path.

Shipped vs internal: what you can actually use today

The shipped parts are Codex Code Review, which reads rules from AGENTS.md; the review-agent skill sample in the open-source openai/codex repository; the public Codex skills documentation; Codex GitHub PR auto review; and a Codex SDK cookbook recipe for running structured review in CI. The internal-only parts are the specialist reviewer agents, the Perf Harness, the internal context graph, the agentic-deploy agent, and the harness-engineering beta itself (Harness engineering, Pragmatic Engineer). To be plain: you cannot install the internal harness. But the shipped components cover most of the loop’s mechanics, and the rest of this post shows how to reassemble them. For background on the pieces, see our guide to building an AI code review agent and a multi-agent Codex delivery pipeline.

AGENTS.md is the review-rules layer

Codex Code Review reads review rules from AGENTS.md and cites them directly in a finding, which turns repository conventions into machine-checkable review criteria. Repository-wide rules live at the root; service-specific rules live in a nested AGENTS.md (Custom Code Review rules for Codex). The worked example comes from the Codex repository itself. The breaking-change rule protects rawResponseItem/*. The app-server emits rawResponseItem/completed, it is marked experimental, and Codex Cloud already consumes it — so a cleanup diff renaming it to rawResponseItem/done compiles cleanly and still breaks clients. The finding text shape is explicit: keep the existing notification, keep it marked experimental, or add a backward-compatible event. In evaluation, rule-guided variants recovered 98% of required custom findings versus 58.3% for the baseline control (Custom Code Review rules for Codex), measured across coverage, restraint, retention, and actionability. OpenAI’s weekly PR volume has more than doubled since Q4 (Custom Code Review rules for Codex), which is the throughput context for why the rule layer matters. The guidance is specific: start with a consequential, non-obvious invariant; scope rules to the code they govern; state the invariant and the safe path; keep rules durable; keep formatting and mechanical checks in CI; and expect broad instructions to create noise.

Repo-local skills: metadata, SKILL.md, references

Codex discovers skills in .agents/skills in every directory from the working directory up to the repository root, and progressive disclosure means only name and description metadata load at startup (Using skills to accelerate OSS maintenance, Codex skills docs). SKILL.md loads only when the agent chooses the skill, and references/ or scripts/ load only when needed — so a repository can carry a large skill library without bloating every prompt. Repository policy, by contrast, lives in AGENTS.md. The Python Agents SDK repo ships 8 repo-local skills: code-change-verification, docs-sync, examples-auto-run, final-release-review, implementation-strategy, openai-knowledge, pr-draft-summary, and test-coverage-improver (Using skills to accelerate OSS maintenance). The JS repo adds 3 more: changeset-validation, integration-tests, and pnpm-upgrade (Using skills to accelerate OSS maintenance). The split between what goes in AGENTS.md versus a skill is functional: rules are constraints the reviewer must obey; skills are procedures the agent invokes on demand.

The shipped review-agent skill contract

The shipped review-agent skill in openai/codex performs a read-only, defect-first review of a specified code change and returns every actionable finding, with a strict flagging bar and a P0–P3 severity output contract (the review-agent SKILL.md). Its description is the whole design: “Perform a read-only, defect-first review of a specified code change and return every actionable finding.” The body requires reading the applicable AGENTS.md, inspecting the complete diff plus enough surrounding code to understand each changed path, identifying concrete regressions, continuing through the whole diff after finding the first issue, and checking relevant tests and call sites to confirm each finding is real. It is explicitly read-only: no file modification, no commits, no pushes, no posting review comments, no delegating. The flagging bar is five-part: the issue must affect correctness, security, performance or maintainability meaningfully; be discrete and actionable; have been introduced by the reviewed change; be demonstrable from the code; and be something the author would probably fix. Speculative concerns, pre-existing problems, intentional behaviour changes, and style nits are excluded. Output is findings first, ordered by severity, one entry per finding as [P1] Imperative finding title — path/to/file.rs:line plus one short paragraph. Priorities: P0 is a universal release blocker or critical failure; P1 an urgent defect to fix next; P2 an ordinary defect; P3 low-impact but worth fixing. If nothing qualifies, the skill says “No findings.” — and forbids inventing one. It closes with an overall assessment naming material test gaps and residual risks. For base-branch reviews, it resolves the comparison ref to the branch’s upstream when that upstream exists and is ahead of the local branch, then runs git merge-base HEAD <comparison-ref> and inspects git diff <merge-base-sha>. This contract is the best starting template if you are building an AI code review agent yourself.

Specialist reviewer agents and risk classification

OpenAI’s internal review spins off multiple agents, each with a domain specialist configuration — the walkthrough names data, infrastructure, cloud, and security, plus risk classification — rather than relying on one generic AI reviewer (Pragmatic Engineer). The framing from the source: equivalent to having a human domain expert from each relevant infrastructure team review every change. “In the past, it would have been impractical for a cloud infrastructure engineer and a security engineer to review every single code change. With agents, that becomes possible.” Risk classification then decides what happens next: low-risk areas can opt an agent into auto-approving its own PRs, while higher-risk changes get an additional human engineer review. The Perf Harness routes problematic PRs to a Synthetics A/B framework. Two caveats are worth carrying over. Gergely Orosz himself expresses scepticism about whether a specialist-framed agent really reviews differently from a generic one, and all Codex agents have full access to OpenAI’s code and docs — the specialisation is prompt-level framing, not access-level isolation. We cover the isolation question separately in prompt-injection defenses for agents.

Verification stack and CI enforcement

The mechanical layer behind agent review is a set of short if/then triggers in AGENTS.md paired with exact verification commands the agent must run before handoff (Using skills to accelerate OSS maintenance). The triggers: before editing runtime or API changes, call $implementation-strategy; if the change touches SDK code, tests, examples, or build behaviour, call $code-change-verification; if a JavaScript package change touches release metadata, call $changeset-validation; for OpenAI API or platform integrations, call $openai-knowledge; at handoff, call $pr-draft-summary. The Python verification stack is make format, make lint, make typecheck, make tests. The JavaScript order: pnpm i, pnpm build, pnpm -r build-check, pnpm -r -F "@openai/*" dist:check, pnpm lint, pnpm test. Measured throughput on the Agents SDK repos: 457 merged PRs from 1 Dec 2025 to 28 Feb 2026 versus 316 in the prior quarter — Python 182→226, TypeScript 134→231 (Using skills to accelerate OSS maintenance). OpenAI’s own verdict on agent review as a required path: “For straightforward program bugs, regressions, and missing tests, relying on Codex as the required review path is now safe enough in practice.” Per the September 2026 coverage of the Codex lead’s interview, AI reviewers can block a merge, and the security check is mandatory without a human enforcing it (The New Stack). We cover the enforcement mechanics in OpenAI Agents SDK in production and multi-agent production patterns.

The cookbook recipe for structured code review

The Codex SDK cookbook recipe runs Codex CLI headless (exec) inside a CI runner with the shipped Code Review prompt, constrains the response with a structured-output JSON schema, parses the JSON, and posts inline comments through the SCM API (the cookbook recipe). The reviewer prompt, verbatim:

“You are acting as a reviewer for a proposed code change made by another engineer. Focus on issues that impact correctness, performance, security, maintainability, or developer experience. Flag only actionable issues introduced by the pull request… Prioritize severe issues and avoid nit-level comments unless they block understanding of the diff. After listing findings, produce an overall correctness verdict (‘patch is correct’ or ‘patch is incorrect’) with a concise justification and a confidence score between 0 and 1. Ensure that file citations and line numbers are exactly correct using the tools available; if they are incorrect your comments will be rejected.”

The example recommends gpt-5.5. One caveat: the older archived GitHub Actions example exposed tokens to the whole job and combined review with PR write permissions — current guidance separates review from posting feedback and enforces human approval for commands. The core invocation is a single headless call:

codex exec \
  --output-schema codex-output-schema.json \
  "$(cat review-prompt.txt)"

The JSON schema pins the output shape, so your CI script parses findings, severity, verdict, and confidence without free-text parsing.

Comparison: shipped Codex review vs OpenAI’s internal specialist-agent review

The table below separates what ships publicly from what runs inside OpenAI, so you know exactly which parts of the loop you can adopt versus which parts you would have to rebuild yourself.

Dimension	Shipped Codex Code Review / review-agent skill	OpenAI internal specialist-agent harness
Availability	Public via Codex CLI/Cloud and the open-source skill sample	Internal-only beta; not installable externally
Reviewer configuration	One reviewer following AGENTS.md rules and the shipped P0–P3 contract	Multiple domain-specialist agents for data, infrastructure, cloud and security, plus risk classification
Merge gating	Can be a required review path for straightforward bugs, regressions and missing tests; comments posted via the SCM API	Low-risk changes can opt into agent self-approval; higher-risk changes get an extra human review; the security check is mandatory
What it can cite	Repository AGENTS.md rules, file paths and line ranges	The full internal codebase and docs, plus logs/metrics/traces via LogQL and PromQL
Output contract	Findings ordered by severity P0–P3, “No findings.” when empty, verdict with a 0–1 confidence score	An iterate-until-satisfied loop across agent reviewers; a deploy agent can babysit the rollout

What a solo developer can copy this week

Everything below uses only shipped components — AGENTS.md, repo-local skills, Codex Code Review, and the SDK recipe — and each step maps directly to a mechanism OpenAI documents.

1. Write one consequential, non-obvious invariant into AGENTS.md — a compatibility boundary or data boundary — and include the safe path alongside the prohibition. Rule-guided review recovered 98% of required custom findings versus 58.3% baseline (Custom Code Review rules for Codex).
2. Add a repo-local review skill under .agents/skills/, modelled on the shipped review-agent contract: read-only, defect-first, return every actionable finding, P0–P3 severity, “No findings.” when clean (the review-agent SKILL.md). This gives you a consistent reviewer without building custom infrastructure.
3. Encode the flagging bar into that skill so the reviewer cannot report pre-existing problems or style nits. Restraint is a measured dimension, and unbounded reviewers drown authors in noise.
4. Run Codex Code Review as a required review path for straightforward bugs, regressions, and missing tests. OpenAI states this is “safe enough in practice” for exactly that class of defect (Using skills to accelerate OSS maintenance).
5. Add a structured-output review step in CI using the cookbook’s JSON schema, and separate the token that reviews from the token that posts (the cookbook recipe). Separation prevents a compromised review step from writing to your repository.
6. Keep formatting and mechanical checks in CI — do not encode them as review rules. Mechanical checks are deterministic; spending reviewer attention on them wastes the agent’s judgement budget.
7. Keep the rule set small: if deleting a rule would not change a review, delete it. Broad instructions create noise, and noise erodes the 98% figure’s premise.
8. Adopt the two golden principles worth stealing: prefer shared utility packages over hand-rolled helpers, and do not probe data YOLO-style — validate at the boundary or rely on typed SDKs (Harness engineering). The team replaced its weekly “AI slop” cleanup Friday, about 20% of the week (Harness engineering), with these encoded rules.

For a deeper treatment of the orchestration around these steps, see our guide to custom coding subagents.

Limits, and what we did not verify

The harness numbers come from OpenAI’s own engineering write-up, the specialist decomposition from a published interview walkthrough, and the skill and rule mechanics from public repository files — no independent benchmark confirms any of them. Perf Harness, the internal context graph, the agentic-deploy agent, and the specialist reviewer agents are internal-only and cannot be inspected. Skill counts must be split by repo — Python has 8, JS adds 3 (Using skills to accelerate OSS maintenance) — not summed into a single figure. The “superhuman” framing is attributed to the Codex engineering lead via The New Stack interview coverage, not presented as an independent benchmark. If you want to stress-test these patterns against competing models, the NiteAgent model arena is where we run that comparison, and our guides index tracks every follow-up.

FAQ

Is OpenAI’s internal harness available for me to install?

No. The harness-engineering beta, the specialist reviewer agents, the Perf Harness, the internal context graph, and the agentic-deploy agent are internal-only (Harness engineering). What ships publicly is Codex Code Review, the review-agent skill sample in openai/codex, the skills system, GitHub PR auto review, and the SDK cookbook recipe — which together cover most of the loop’s mechanics.

Can one Codex agent really review another agent’s code?

Yes, and it is the core of OpenAI’s loop: Codex reviews its own changes locally, requests additional specific agent reviews locally and in the cloud, and iterates until all agent reviewers are satisfied (Harness engineering). OpenAI reports that almost all review effort is now handled agent-to-agent, with humans entering selectively for higher-risk changes.

What should my first AGENTS.md review rule be?

Pick one consequential, non-obvious invariant — typically a compatibility boundary or a data boundary — and state both the prohibition and the safe path (Custom Code Review rules for Codex). Scope it to the code it governs, keep it durable, and leave formatting and mechanical checks in CI. One good rule beats a page of broad instructions.

How do I run the shipped review-agent skill without building custom infrastructure?

Copy the skill sample from the openai/codex repository into .agents/skills/ in your repo (the review-agent SKILL.md). Codex discovers it automatically via progressive disclosure. It is read-only and self-contained, so no extra infrastructure is needed — the agent runs the review against your diff and returns P0–P3 findings.

Is AI code review safe enough to block merges on a small team?

For straightforward program bugs, regressions, and missing tests, OpenAI states that relying on Codex as the required review path is “safe enough in practice” (Using skills to accelerate OSS maintenance). AI reviewers can block merges and the security check is mandatory without human enforcement (The New Stack). Keep a human in the loop for architectural and risk-classified changes.

The Bottom Line

Copy two things immediately: one consequential invariant written into AGENTS.md with its safe path, and the shipped review-agent contract as your repo-local review skill. Leave behind the fantasy of installing OpenAI’s internal harness — the specialist agents, Perf Harness, and context graph are not public. The evidence justifying confidence is OpenAI’s own: 98% rule-guided finding recovery versus 58.3% baseline (Custom Code Review rules for Codex), 457 merged PRs in one quarter (Using skills to accelerate OSS maintenance), and agent review as a stated required path for a class of defects.

How This Guide Was Built

Sources used for this analysis: Harness engineering, Using skills to accelerate OSS maintenance, Custom Code Review rules for Codex, Codex skills documentation, the review-agent SKILL.md, the Codex SDK code-review recipe, the Pragmatic Engineer walkthrough, and The New Stack interview coverage.

This analysis is based on OpenAI’s official documentation, engineering blog posts, and published interviews — we did not run Codex hands-on.

📖 Related Reads

• ToolBrain — tool reviews, LLM comparisons, and AI workflow guides
• CodeIntel Log — code quality, debugging, and software engineering benchmarks
• NoCode Insider — AI workflow automation with no-code tools, agents, and APIs

Cross-links automatically generated from NiteAgent.

← Back to all posts