Flaky Integration Tests Root Cause Analysis

To integration-test stateful APIs without hitting production, replace fixed responses with a stateful simulator checked against the real API so that create, read, and cancel operations use the results of earlier calls. Keep live calls for end-to-end tests that run on a separate schedule. Flaky integration test root cause analysis determines where this isolation belongs, because timing, shared state, environment variance, and external dependencies require different fixes.

Why integration tests fail intermittently even when the code is correct

A consistently failing test points to a real bug or a stale test. A test that passes and fails on the same code, configuration, and data can instead expose non-determinism in the test or its environment. An intermittent failure can also reveal a genuine product race condition, and dismissing that case as noise is how concurrency bugs reach production.

Integration tests sit at particularly exposed boundaries. They flake more than unit tests and less than end-to-end browser tests because they cross networks, shared databases, and external services. Bitrise Mobile Insights 2025 found a large rise in the number of teams with flakiness over three years as suites grew, more tests ran in parallel, and each test had more third-party dependencies. The lasting damage is not the wasted rerun. Once engineers learn that red often means noise, they stop looking into failures, and the quality gate can no longer catch regressions.

The four root cause categories that account for most integration test flakiness

The four useful diagnostic categories are timing dependencies, shared state and test order coupling, environment variance, and external service dependencies. Luo et al.’s taxonomy places async waits at nearly half of flaky-test causes, with concurrency and race conditions and test order dependency among the next largest contributors. Resource leaks and network dependencies make up the next largest groups. These proportions indicate priorities whose distribution varies by integration suite. External services form a small but disproportionately damaging slice because their behavior changes outside the team’s control.

A retry policy does not remove shared-state coupling, and an explicit wait does not restore a third-party API that is down. To decide where to spend their effort, teams should take a random sample of recently flaky tests, group them by category, and measure each category’s share. That local distribution identifies where one timing or infrastructure repair could resolve a cluster of failures and save test-by-test patching.

Timing and async dependencies: the largest single source of integration flakiness

A fixed sleep is a bet on completion time. Calls such as await page.waitForTimeout(2000) and Thread.sleep(5000) assume that the application reaches the required state before the interval ends. Integration tests make that assumption across process boundaries, including API calls, queue consumers, and database writes followed by reads. A slower runner or higher query latency can invalidate it without any application change.

The diagnostic sequence is to confirm intermittent failure, measure its rate, and identify the assumed wait. Repeated runs against the same commit establish a baseline against which a repair can be assessed. Fixed sleeps and hard-coded timeouts are the next things to check, especially when most failures happen in CI. Local success can indicate that runner load exposes an assumption hidden by a faster development machine.

The corresponding fix is an explicit readiness condition. Depending on the assertion, that condition may be a completed network request, an acknowledged queue message, or a status field reaching its target value.

Shared state and test order coupling: the flakiness that only appears in a full suite run

Timing failures can occur when one test runs over and over, while order coupling often stays hidden until the tests run together as a suite. A test may pass alone but encounter a “record already exists” error on a second run because earlier execution left state behind. The dependency can also work in reverse. A test that expects another test to create its preconditions will fail in isolation or when execution order changes.

A suite stable at 50 sequential tests can become flaky at 500 parallel tests as collisions against shared resources become more frequent. Randomizing execution order in scheduled CI runs exposes dependencies concealed by a fixed sequence. When a test passes in fixed order but fails after shuffling, the diagnosis points toward inherited state. The repair replaces reliance on another test’s side effects with per-test ownership of preconditions, with setup in a before hook and teardown in an after hook.

Environment variance: the flakiness that no test code change can fully fix

CI hardware, OS versions, container cold starts, and network conditions can alter behavior or latency between otherwise equivalent runs. The useful clue is where failures cluster. Concentration on particular runners, at particular times, or after a deployment points toward an environmental cause beyond a single wait assumption. Research presented at FSE 2025 found that nearly half of flaky tests across Java, JavaScript, and Python projects were directly shaped by CPU, memory, or computational resource availability.

There is no clean test-code fix for this category. Explicit waits can remove brittle timing assumptions, but they cannot standardize the resources available to a process. The infrastructure remedy is version-pinned containers, consistent resource limits, and replacement of accumulated shared environments with fresh ephemeral ones. Where variation is intentional, as in cross-browser or cross-OS testing, tagging and separate pipelines keep infrastructure-sensitive tests from blocking the main pipeline. Without that separation or standardization, environment variance sets a floor on achievable reliability.

External service dependencies: why this category demands its own remediation strategy

An external dependency introduces an independently operated system into the test’s result. A real HTTP call can fail because the service responds slowly, returns an error, enforces a rate limit, or undergoes maintenance. The team cannot repair that system’s availability by editing its own runner or test. Retries can turn a pipeline green while concealing both regressions that happen to pass on retry and a rising dependency-related failure rate.

Availability is only one source of uncertainty. Stripe states that general sandbox settings and capabilities are not synchronized with live settings and can diverge, although its legacy test-mode sandbox shares many settings with live mode, and its event delivery can be duplicated or out of order in both environments, while IC+ pricing cannot be exercised in a sandbox. Kalshi’s token-bucket model, updated 2026-04-23, uses seven tiers with two independent buckets per tier and provides no Retry-After header. A rate-limited test therefore receives no machine-readable instruction for how long to wait.

Service isolation is the direction for integration tests. But an isolated test validates meaningful integration behavior only if the chosen test double supports that behavior.

Why stateless mocks solve the availability problem but create a correctness problem

Consider an order flow that creates an order through POST, receives a 201 with an ID, polls that ID until completion, and then receives a 409 when attempting cancellation. A stateless mock returns the same fixture for every poll, so the loop either never ends or ends at once, whatever progress the real order has made. Request-to-response mappings do not represent a world whose state changes after each operation. The test can pass solely because its fixtures agree with its assertions.

That silent success is harder to detect than an outage. Each change to the real API needs a matching stub update, but an outdated mock keeps returning its old response correctly. Slack’s automated drift check on 2026-10-05 found nine new agents.conversations.* methods in the Java SDK main branch that were missing from client TypeScript files and had no matching request and response types. Those gaps were absent from the 2026-09-28 check.

Tests also run into a mismatch when they need objects that earlier calls created but their mocking layer is still stateless. Mock adoption removes live-service availability from the result without necessarily removing that structural dependency.

How can integration tests use a stateful, continuously verified API simulator

A stateful simulator remembers the cumulative result of prior calls. In the order flow, creation establishes an object that later reads and cancellation requests can act upon. The test then checks the order of operations as part of the behavior. Statefulness alone, however, does not establish API fidelity. Continuous verification against the live API before each release exposes discrepancies in the behavior exercised by verification probes.

Published fidelity scores report match rates of Slack 96.7%, GitHub 91.5%, Kalshi 91.7%, PayPal 97.8%, and Lob 100.0%, each representing the proportion of probes that matched live behavior exactly. The variation distinguishes measured agreement from a claim of universal equivalence and leaves the scope of untested behavior separate from the score.

Realistic pre-seeded records give flows a history of users, events, transactions, and the links between them. Running locally or in containers also keeps live-service availability, rate limits, and live credentials out of the integration run, even in offline or air-gapped environments. Controlled latency, 429s, and 503s let the application’s error handling and retry behavior be exercised reproducibly, independently of live-service failures.

How to detect which root cause is responsible before writing any fix

The practical workflow starts by running the same test many times against the same commit and recording its failure rate. That baseline supports both prioritization and later proof that a repair helped. Classification then proceeds by eliminating competing explanations. CI-only failures that increase under load suggest timing or environment variance, while suite-only or randomized-order failures suggest shared state. Concentration on specific runner types or OS versions strengthens the environment diagnosis, and correlation with service status or rate-limit windows points toward an external dependency.

Logs and traces must capture the moment of failure. A connection timeout or non-200 response from an external host directs investigation toward the dependency, while “element not found” after a slow response suggests a timing assumption. A “record already exists” error instead points toward shared-state contamination. Evidence of a genuine concurrency bug belongs in product debugging.

Quarantine temporarily stops a test from blocking the pipeline but removes its coverage from that gate while repair remains pending. A time-bounded resolution policy limits the risk of a quarantined-test graveyard in which coverage gaps accumulate quietly.

Remediation by category: the specific fix for each root cause

For timing failures, replace fixed sleeps with conditions that poll for the state being asserted. Timeouts for cross-process operations should reflect realistic end-to-end latency under CI load. For shared state, give each test its own setup and teardown through before and after hooks. Randomized execution then serves as a regression check. For environment variance, use version-pinned containers and explicit runner resource limits, with environment-sensitive tests separated into their own pipeline stage.

For external dependencies, use a stateful, verified simulator for the majority of integration tests. Real-service end-to-end tests should have clear tags and their own schedule instead of running on every commit, with different tolerances for external-service failures and application failures. For resilience tests, simulator fault injection provides controlled, reproducible latency, 429s, and 503s without depending on live conditions.

Root cause analysis identifies which uncontrolled variable entered the result, and that diagnosis determines the intervention.