10 Million Products, 20 Codex Agents: From Repair Plans to a Verified Merge

A catalog of ten million products. Twenty Codex agents proposing repairs on separate data branches. One measured merge of 4,552,565 approved rows. This is the database run behind the 60-second film on our homepage.

The completed run held 33,248 proposed rows for review and excluded 14 deliberately injected price violations. Here is how we built the fixture, assigned the work, checked the proposals, assembled an approval branch, and turned the final evidence into a video.

1. Define the change contract

The scenario began with: “You have a 10-million-row product catalog.” To make it executable, we specified which fields an agent could repair and what had to remain unchanged.

The allowed repairs covered brands, categories, attributes, descriptions and SKUs. Price was protected. Every product had a stable id primary key and supplier-source columns such as source_brand and source_description. These gave each repair a factual reference within the fixture.

The supplier values and confidence scores were synthetic too. Most rows had source_confidence = 0.99; rows where id % 97 = 0 had 0.60. The coordinator held proposed changes below 0.80 for review, giving us a deterministic way to check the selection policy.

2. Build a deliberately imperfect catalog

The issue predicates deliberately selected disjoint primary-key sets. That made each task's effect easy to inspect. The SKU task restored canonical identifiers for distinct product IDs, preserving every product row.

The seed script inserted 200,000 rows per batch, for 50 batches. Their combined client-observed insertion time was 36.865 seconds, excluding table creation, initial validation and snapshot creation. We then recorded the baseline issue counts and created a named snapshot.

CREATE SNAPSHOT run_base FOR TABLE catalog main;

DATA BRANCH CREATE TABLE catalog.agent_brand_0
  FROM catalog.main{snapshot='run_base'};

SQL in this article uses shortened database and snapshot names for readability. The logs retain the exact run identifiers and executed statements. In the snapshot statement, the database and table names are separated by a space.

3. Give each Codex task a bounded repair

We split each of the five repair types into four ID shards of 2,500,000 IDs. This produced 5 × 4 = 20 independent Codex tasks, each assigned to a task branch. Each branch had the full logical ten-million-row baseline; the task's writes were limited to its assigned ID range.

main · 10,000,000 rows
│
├── attributes-0 … attributes-3
├── brand-0      … brand-3
├── category-0   … category-3
├── description-0 … description-3
└── dedup-0      … dedup-3

20 task branches → approved branch → main

Every task received the schema, issue predicates, allowed field, ID range and an output contract. It returned one JSON file containing its reasoning, a repair UPDATE, a validation query and a review note. Database credentials remained with the coordinator, which inspected the plans before executing them.

The first brand task produced this repair:

UPDATE {{branch}}
SET brand = source_brand
WHERE id BETWEEN 1 AND 2500000
  AND MOD(id, 10) = 2
  AND source_confidence >= 0.80
  AND NOT (brand <=> source_brand);

The <=> comparison handles NULL values. The coordinator replaces {{branch}} with the task's fully qualified table name. Each statement processes its eligible rows in bulk, so the model's role is to turn a bounded repair task into an executable plan.

Codex tasks ran in waves of at most three concurrent tasks. Database execution also used at most three worker connections. The film's twenty branches reflect actual created branches; execution concurrency was separately limited.

4. Execute on branches and inspect the full change set

The executor checked task types, shard bounds and basic SQL shape, then ran each plan and its validation query on the assigned branch. These checks handled reviewed experiment inputs. A production deployment needs database permissions or a constrained execution service to enforce access.

For every branch, we collected SQL counts alongside native MatrixOne diff results:

DATA BRANCH DIFF catalog.agent_brand_0
  AGAINST catalog.main{snapshot='run_base'}
  OUTPUT SUMMARY;

DATA BRANCH DIFF catalog.agent_brand_0
  AGAINST catalog.main{snapshot='run_base'}
  OUTPUT LIMIT 3;

The summary provided complete scope counts, while the small sample illustrated individual changes. Native diff counts reconciled with the corresponding SQL counts. We also checked main again: its original issue counts were unchanged, and native DIFF against the baseline reported zero inserted, deleted and updated rows.

At this point, the task branches contained millions of proposed changes while main still matched its baseline.

5. Inject fourteen changes that should be rejected

After the brand repair ran, the coordinator deliberately set the price to 99 on fourteen rows in that branch. The product shown in the video originally cost 100.20; its proposed branch value became 99.00.

This fault injection exercised a concrete rejection path: a row could contain a valid brand repair and a forbidden price edit at the same time.

PICK selects row changes, so every changed field on a selected row matters. We excluded the entire violating row. Approving a field name in a report cannot remove a different, forbidden field change from the source row being picked.

6. Build a traceable data change report

Combining the twenty branch results produced these measured counts:

The final three categories are exclusive and sum to the proposed-row count. The price violations occurred on rows that already had brand proposals, so per-field change counts overlap.

The initial fixture contained 4,600,000 problem rows. Some Codex plans proactively skipped low-confidence rows, leaving 14,173 problem rows without a proposal. Those rows stayed unchanged and are additional to the 33,248 proposed rows held for review. Keeping those categories distinct makes the report auditable.

7. Assemble approved rows, then merge once

We created an approved branch from the same baseline. For each task branch, the coordinator picked keys within that task's ID range and issue predicate, with a changed target field, source confidence of at least 0.80, and a price equal to main.

DATA BRANCH PICK catalog.agent_brand_0
INTO catalog.approved
KEYS (
  SELECT b.id
  FROM catalog.agent_brand_0 b
  JOIN catalog.main m ON b.id = m.id
  WHERE b.id BETWEEN 1 AND 2500000
    AND MOD(b.id, 10) = 2
    AND NOT (b.brand <=> m.brand)
    AND b.source_confidence >= 0.80
    AND b.price = m.price
)
WHEN CONFLICT FAIL;

This price comparison fits the fixture's known non-NULL prices. An adaptation needs NULL handling and checks for every protected field, inserts, deletes, key scope and cross-row business constraints.

Approval in this experiment meant the coordinator applying the predefined selection policy within the user-authorized run. The video's Approve button visualizes that stage. A team deployment would add its own independent approval record and interface.

After assembling the approval branch, we executed the final operation:

DATA BRANCH MERGE catalog.approved
INTO catalog.main
WHEN CONFLICT FAIL;

The twenty sequential PICK statements took 643.391 seconds in total. The final MERGE took 840.808 seconds, approximately fourteen minutes. The execution script took about 28 minutes and 31 seconds from branch creation through final verification, excluding earlier data seeding and Codex plan generation.

After the statement completed, we checked the actual target:

• Main still contained 10,000,000 rows.
• Native final DIFF reported 4,552,565 updated rows, with zero inserted or deleted rows.
• Price changes in main: zero.
• Low-confidence, unreviewed repairs reaching main under the experiment's checks: zero.

These timings are client-observed measurements from one run on a shared instance. They describe this experiment's execution.

8. Three problems we encountered

An aggregate needed different syntax

The first baseline query attempted to sum Boolean expressions, which this instance rejected. All ten million rows had already been inserted. We preserved the data, changed the aggregates to SUM(CASE WHEN ... THEN 1 ELSE 0 END), resumed the baseline checks and created the snapshot.

Re-merging an original source after PICK can conflict

A separate small probe verified safe row selection and conflict rejection. It also observed a conflict when merging an original source after a previous PICK on this instance. The measured catalog workflow therefore assembled one approval branch and merged it once. After a timeout or lost connection, reconcile operation state and target diff before deciding whether to retry.

Data isolation still needs a resource plan

A later Playground check received a busy-compute-node error, mentioning possibly excessive active transactions; a direct connection received the same response. That observation did not establish the error's root cause. It highlighted a deployment concern: branches on a shared instance can still compete for resources. Production use needs workload budgets, concurrency limits and, when appropriate, a separate workspace instance.

9. Render the film from the completed evidence

The video renderer reads the actual execution.json report. Before rendering the final film, it requires the recorded state to be merged. The proposed, review, blocked and merged counts on screen come from that report.

We drew the frames with Pillow and encoded them with FFmpeg into a 1280 × 720, 20 fps H.264 video, with English captions. Long database operations became short sequences. The labels “Synthetic catalog” and “Edited replay” remain visible to explain how the film relates to the measured run.

Bring the workflow to your own project

The experiment demonstrates a concrete path: agents prepare bulk repairs on branches; a review process identifies the exact acceptable changes; native database operations apply those approved rows.

Production use also needs restricted proposal credentials, independently held merge credentials, approvals tied to frozen source versions, complete business validation, resource limits and a recovery policy. This synthetic run exercised the database workflow; each deployment must establish its own production authorization boundary.

We packaged the workflow and those boundaries into an open-source Agent Skill. Use it with Codex and an existing MySQL client, start with a small dataset, and adapt the validation rules to your own business.