Flaky Test Management in Automated Software Factories

An asynchronous coding agent finishes a task, opens a pull request, and hands it off for review. Nobody is watching when the build comes back red. On a human-paced team, a developer glances at a failed CI run, recognizes the usual suspect, clicks rerun, and moves on with the day. In a software factory, where agents like GitHub's Copilot cloud agent, Cursor's Cloud Agents, OpenAI's Codex cloud, Google Jules, and Devin open branches and submit PRs continuously, that red build lands with no one present to make the call. The question of whether the failure is real or flaky falls to the pipeline itself, and most pipelines were never built to answer it.

In an agent-driven CI pipeline, a flaky test becomes a structural blocker because no human is present to recognize the failure, dismiss it as a known offender, and rerun the job. The agent either retries blindly or stalls, and inside a merge queue, a single flaky failure invalidates the speculative merge commit every queued PR behind it was relying on, forcing each of those PRs to restart validation from scratch. At the volume a software factory produces, that cascade recurs at the flakiness rate rather than the code-quality rate, compounding rerun costs across the entire queue simultaneously rather than one PR at a time. Pipeline-level quarantine addresses this by splitting required merge-gate checks from an informational suite that runs in parallel but cannot block a merge: the flaky test gets pulled from the blocking suite the same day it is identified, assigned an owner, measured against a flakiness threshold over a rolling two-week window, and returned to the gate only after it has demonstrated stability, while the distinction between required and informational checks is surfaced in a machine-readable form the agent can act on directly rather than written down somewhere for humans to consult.

A flaky test is one that passes and fails on the same code with nothing changed between runs. None of these five causes is new, and none is mysterious. What changes in a software factory is not the taxonomy of flakiness but the volume and speed at which it gets triggered.

Coding agents made writing code the cheapest part of shipping software. They did not make landing that code any cheaper. Sustained execution across long, multi-step workflows sets agentic AI apart from earlier developer tooling; inside one, a flaky failure that fires mid-workflow doesn't pause a human waiting on a notification, so there's no one to catch it. It derails the entire run, and the agent, with no judgment to apply to a red build, either retries blindly or stalls.

A common reply to all of this is that retries already solve the problem, that teams have handled flaky tests for years by rerunning the job and moving on. Retries hide the failure signal instead of resolving it. That was a tolerable inefficiency when a handful of engineers merged a handful of times a day. At agent frequency, with dozens of PRs queuing continuously, the same workaround behaves like a denial-of-service attack on the pipeline.

How a Flaky Test Cascades Through a Merge Queue

In the merge queue, a flaky test turns from a local nuisance into shared infrastructure damage. When a check fails on a queued entry, the queue removes that PR, but every PR lined up behind it was tested against a speculative merge commit that assumed the failed entry would land. Once that assumption breaks, every one of those downstream PRs needs a new speculative commit and a fresh CI run. This cascading restart is widely cited as the single biggest source of frustration with merge queues, and it has nothing to do with whether the code behind any of those PRs is good.

Some queues handle the failure naively, ejecting the entire batch and retesting everything from scratch. Either approach spends real compute and real time recovering from a single red result, and either way, the cost is identical in kind: one flaky test invalidates the tested state that every other PR in the batch was relying on.

The rate at which this cascade recurs tracks the flaky-test rate rather than the quality of the code moving through the queue. A flakiness rate low enough to shrug off at ten merges a day turns into a near-continuous cascade once a factory is pushing through ten times that volume. Uber's experience before it built dedicated flake management illustrates the scale of the exposure: its iOS mainline was green only 52% of the time. A fraction of a percent of unreliable tests was enough to clog the queue for every engineer behind them.

Base Power Company's engineering team made what it calls the uncommon choice to pull end-to-end suites that traditionally run post-merge, nightly, or before a release into the pre-merge CI gate. Even with diff-aware selective testing and caching in place to keep that gate fast, a single flaky failure still triggers a queue rejection, and that rejection forces every subsequent PR to restart validation from scratch. The mechanics don't change just because the testing strategy got smarter upstream.

Agent behavior turns this into something closer to a structural hazard than an occasional bad day. A single flaky test can already add a substantial block of rerun time to one pull request. A factory of agents produces PR volume high enough that the rerun cost compounds across the entire queue at once, not one PR at a time.

Why agent harnesses inherit the flakiness problem

Agent harnesses, the scaffolding that tells an agent how to invoke tools and interpret results, are themselves largely untested. A large share of these harnesses have neither a test nor an eval behind them. Monitoring catches drift that evals were never built to catch, and transcript evidence only means something once someone has actually read the transcripts behind it.

When a skill invocation inside an agent fails intermittently, because a tool call times out, an environment variable is missing on a particular runner, or a third-party dependency starts rate-limiting, the failure looks exactly like a flaky test from the outside. A flaky agent step and a flaky test require different fixes even though they present identically in a build log.

That resemblance creates a detection problem specific to agent-driven pipelines. The standard flake-detection heuristic depends on running the same test repeatedly under the same conditions and comparing results, which requires a history on that specific commit. A 2025 analysis of GitHub Actions Java projects found that 67.73% of rerun builds already exhibit flaky behavior, a number that assumes the retries are happening against the same commit. Asynchronous agents often don't retry that way. Each attempt from an agent is frequently a new PR rather than a rerun of the old one, so the flakiness pattern never has a chance to accumulate into a detectable history attached to a single test.

One emerging response is a Flaky Test Triage Agent, a system whose job is narrowly to answer whether a given failure is flaky or a genuine bug. It operates at what's described as trust level A2: it can recommend but cannot disable, quarantine, or skip a test on its own, a constraint that is deliberate. It lets the triage agent unblock pipelines by supplying a judgment humans would otherwise have to make manually, without giving it the authority to mask a real regression by quietly suppressing it.

Cypress Cloud MCP, released in 2026, points at the same need from a different angle. The underlying requirement in both cases is the same: when agents are generating the PRs and could also be the ones consuming flake data, that data has to live in a form a machine can read and act on inside the pipeline itself. A dashboard a human checks once a day doesn't reach an agent that merges at 3 a.m.

Detecting flakiness systematically rather than discovering it reactively

Reliable detection depends on designed observability at the pipeline level, not on engineers noticing a pattern after a build has already broken a few times. The basic heuristic hasn't changed: run the same test repeatedly under the same conditions, and if it fails on one run and passes on the next with no code change in between, it's flaky by definition. What has to change is how systematically that heuristic gets applied, and what happens to the result once it's confirmed.

Root-cause categorization matters as much as detection itself. TestDino assigns each flaky test a stability percentage along with a root-cause category: timing, environment, network, assertion, or other. That category is what determines the eventual fix. Without it, a quarantined test just sits there, unexamined, and the quarantine list turns into a graveyard rather than a working backlog.

TigerBeetle's approach offers one of the more actionable patterns in this space. Atlassian found that 21% of master build failures in its Jira Frontend repository were attributable to flaky tests. Collecting those failures in one visible place compresses the time it takes to notice a pattern, makes prioritization possible, and reveals correlations between failures that would otherwise look unrelated.

Agent-driven pipelines need an additional layer on top of all this. Because asynchronous agents tend to produce a new PR per attempt rather than retrying the same commit, history-based flake tagging may never build up on any single test. Detection in this environment also has to watch for failure patterns across different agent-generated PRs that touch the same code paths, since that's often the only place the signal appears.

Detection also needs an owner. Flakiness should be tracked the way technical debt is tracked: a specific person responsible for their module's flakiness budget, with a defined threshold over a rolling window that triggers an alert when it's breached. Detection without ownership just produces a longer list that nobody acts on.

What does pipeline-level quarantine look like in a software factory?

The single most consequential structural decision in a software factory is splitting the required merge gate from an informational test suite that runs but doesn't block. Required checks, things like unit tests, type checking, linting, and security scans, stay lean and reliable inside branch protection. End-to-end tests, visual regression, and performance benchmarks run in parallel but have no power to stop a merge. Only the required checks gate the queue. A flaky test can be pulled out of the release gate the same day it's identified without losing the signal it produces.

OpenAI's own operating philosophy makes the strongest case for this split. A bad merge costs one extra PR to go back and fix. A blocked queue costs every engineer and every agent behind it, every single time it happens.

The honest risk in all of this is that quarantine, left unmanaged, becomes exactly the graveyard that critics of this approach warn about. A test gets pulled from the gate, nobody revisits it, and the same failure that used to mask a real regression in the required suite now masks it in a different, less-watched bin. The fix is discipline around quarantine: a flakiness alert threshold measured over a rolling two-week window for each test, an assigned owner responsible for that test's status, and a clear return-to-gate criterion, so a test only goes back into the blocking suite once it's proven stable again.

Blind retries remain the anti-pattern to avoid inside this structure, whether or not a quarantine split exists. They hide the failure signal, quietly inflate pipeline time, and let root causes sit unaddressed. One flaky checkout test was silently retried and auto-passed 14 times in a single month before a real payment bug made it through to production behind the same cover. Framework-level retry mechanisms, Playwright's retry option, Jest's jest-circus retry, pytest-rerunfailures, are only useful if every retried pass gets logged as a flake signal rather than disappearing into a silent green checkmark.

For agent-operated merge queues specifically, this split has to be visible to the machine rather than written down somewhere for humans to read. An agent that can't tell a required check apart from an informational one will treat a failure on either as a reason to block or retry indefinitely. The pipeline has to surface that distinction in a form the agent can act on directly.

Fixing flaky tests at the root rather than managing them indefinitely

Quarantine buys time. It doesn't fix anything on its own, and each of the five root causes behind flaky tests has a specific, known solution that removes the failure rather than just hiding it behind a gate.

Timing and async issues get solved by replacing fixed waits with web-first auto-waiting. Of the fixes available for UI test suites, this is the one with the biggest single impact.

Test interdependence and shared state get solved by isolating every test so execution order stops mattering. Each test should set up its own data and tear it down when it finishes. Shared state between tests is the main driver behind flakiness that only appears under parallel execution or when test order changes between runs.

Unstable test data gets solved by making that data deterministic. Deterministic seed data paired with explicit teardown removes this failure mode at the source.

Network and third-party variability gets solved by controlling external calls directly: mocking third-party APIs and external services inside CI rather than hitting them live. Tests built around real responses from payment gateways or authentication providers will flake every time one of those dependencies rate-limits, slows down, or changes its response shape, none of which has anything to do with the code under test.

Environment drift is often the infrastructure cause behind failures that look like code problems. Testkube addresses this by running tests as Kubernetes Jobs inside clusters using the same configurations, secrets, and network policies as production workloads, removing environment drift as a source of flakiness. Tests that used to spend far longer sitting in a CI/CD queue complete in under 10 minutes by running in parallel across available cluster capacity instead.

UI brittleness deserves particular attention given how AI coding agents work today, since they produce UI refactors constantly, and tests bound to specific selectors couple themselves to DOM details that shift with every one of those refactors. Self-healing platforms that resolve elements by intent rather than by static selector reduce this inflow at the source rather than reacting to it after each refactor breaks another batch of tests.

Fix priority should follow impact rather than convenience. Set a flakiness threshold per test over a rolling window, identify which tests breach that threshold most often, and fix those first. TigerBeetle's practice of collecting every main-branch failure in one place does exactly this kind of prioritization automatically, surfacing the highest-frequency offenders before anyone has to go looking for them.