Eliminating State Pollution Between Integration Test Runs

Integration testing against stateful APIs requires an isolated, known-good starting state and an API substitute that remembers the effects of earlier calls. A stateful simulator, reset before each run and verified against the real API, provides that foundation without sending test workflows to production. Eliminating state pollution is therefore an initialization problem.

Why integration tests become order-dependent over time

Shared mutable environments make tests inherit one another’s history. A test may be correct when run alone yet fail after another test creates a resource or changes a configuration value. This differs from an infrastructure-dependent failure, where a missing database, queue, or external service causes the test to fail even in isolation. Missing dependencies have a relatively local diagnosis. Cross-test contamination separates the cause from the failure, making execution order part of the test’s undeclared input.

Consider a test that creates a user and then logs in, assuming the user does not already exist. In a QA environment reset weekly, that assumption holds. In a staging environment that is never reset, the same suite fails on its third run with a conflict error. The application can be behaving correctly throughout. The starting state changed.

Configuration mutations create the same hidden dependency. GitHub issue #75 in RepairYourTech/schem-sync-portal describes a test that sets MOCK_FAIL_PROBABILITY and resets it only on its normal completion path, allowing a failed assertion to contaminate later tests. Hardcoded environment values, unenforced data assumptions, and deployment timing differences further obscure the cause across environments. At four or more environments, diagnosis can take longer than the fix. These failures look like product defects or flaky infrastructure, which hides where they came from.

Why teardown routines cannot be made reliable enough to solve this

Teardown tries to fix a shared environment after it has changed. Its success depends on cleanup running, retaining the information needed to undo changes, and avoiding races with other work. The same issue describes an afterEach that deletes process.env.RCLONE_CONFIG_PATH before checking or removing the file it identifies. With that path gone, the cleanup branch cannot remove it. A teardown routine can appear to execute while leaving its intended work undone.

Database transaction rollbacks address a narrower problem. They work for tests confined to reading and writing rows, but many databases cannot roll back schema changes such as ALTER TABLE. Nor can a rollback undo a side effect already sent to an external service. Pre/post-test hooks also need a reachable cleanup path, while cleanup placed only after assertions can be skipped when those assertions fail.

Dedicated truncation scripts, admin API calls, and shell scripts operate after contamination has occurred and require tuning to avoid unintended deletions in shared environments. Ephemeral containers come closer to isolation, but a container provisioned per run still permits parallel tests within that run to interfere. The boundary is still too broad for those tests. Keeping repair logic up to date adds work whenever the system changes, while cleanup on failure paths rarely gets used. When cleanup fails, teams pay again to rerun the pipeline or have less trust in the result.

What hermetic tests actually require — isolated, known-good state at the start of each run

A hermetic test includes the whole context in which it runs. It inherits no state from an earlier run, shares no mutable state with a concurrent run, and leaves no residue that affects a later run. The decisive guarantee exists before execution begins. Cleanup can still reclaim resources, but a later test’s correctness cannot depend on an earlier test repairing the environment successfully.

The isolation boundary differs by test type. Unit tests use pure functions and deterministic fakes without shared state. Integration tests run production orchestration against a fake process or API adapter, and a single helper takes a snapshot of environment variables and restores them so changes cannot leak across files. E2E/system tests use real dependencies with isolated, ephemeral fixtures. These boundaries distinguish what each test owns from what other tests may change.

Initialization must also establish that the environment is usable. That means waiting for a real dependency health check, running the production migration path rather than a separately maintained test schema, and loading only the records needed for the scenario. Environment-specific values belong outside test logic and are injected by the pipeline. Parallel jobs need their own isolated data even when they have different execution slots. A post-test assertion that the working tree and fixture root contain no unowned artifacts then makes residue a machine-checkable failure and automates enforcement of the convention.

Why stateless mocks fail the hermetic standard even when teardown succeeds

An isolated database does not make an API workflow trustworthy if its substitute cannot represent changing state. A stateless mock maps a request to a fixture, such as returning a fixed response for a GET on a particular path. It does not remember what a previous POST created. Isolation prevents interference, but workflow fidelity requires more. The substitute must preserve the causal relationship between operations within the test.

An order workflow exposes the distinction clearly. The application creates an order, receives a 201 with an ID, polls that ID until completion, and then receives a 409 when it attempts to cancel the completed order. With a fixed polling response, the loop either ends at once or never ends. The returned ID may not match the created order, and the conflict path remains unreachable. Successful cleanup cannot repair that missing model of the workflow.

The same gap appears when a billing API interprets an idempotency key differently from the application. Thousands of passing unit tests can leave that interface failure undiscovered until an integration job reaches it. Stateless mocks remain faster and simpler for stateless checks. But for workflow tests, responses kept up to date by hand bring costs in the work of maintaining fixtures and in failures the suite misses. A clean test environment is necessary, but it cannot compensate for a substitute that discards the consequences of each call.

How can you do integration testing against stateful APIs with resettable simulators?

A stateful simulator combines an isolated starting point with a model of sequential operations. It stores a resource on POST, returns it on GET, includes it in list results, applies PATCH changes, and fires a webhook on a state change. Those operations belong to one evolving world. A reset endpoint brings that world back to a known-good baseline before the next run, so it does not depend on the earlier run to undo every change.

The baseline should be a sandbox with realistic seed data in it. An empty simulator forces each test to construct every precondition, reintroducing coupling through setup. Representative seed data lets a test concentrate on its own workflow while adding only the records it needs. Parallel jobs still require separate state ownership. Each can use its own simulator instance, with resets applied within that boundary rather than against an instance another job is actively using.

The open-source one-payment-six-failures project has a control API with POST /simulation/reset. Its simulations cover authorization failures, lost responses from idempotency gaps, retry storms, rate-limit bypass through X-Forwarded-For spoofing, conflicting refund updates involving ETag/If-Match, and latency from broken trace-context propagation. The reset operation wipes state and restores the seed. Fidelity requires a separate safeguard, however. Continuous verification against the live API before each release addresses drift, while published fidelity scores provide a machine-readable signal of how closely the substitute matches the API.

Applying the isolation pattern to specific APIs — Stripe, Slack, GitHub, and payment workflows

Persistent resources are only part of API state. Idempotency keys affect repeat requests, webhooks follow state changes, and rate limits accumulate across calls. An application’s sequence of operations must respect these behaviors to be valid. A useful integration test therefore needs more than individually plausible responses. It needs each response to reflect the operations that preceded it.

Stripe’s documentation states that its APIs are rate limited and its frontend interfaces include security measures that prevent automated testing. Its guidance is to simulate interface outputs and API requests with mock data to test application behavior and error handling. The localstripe project has a local, stateful Stripe server that runs without touching real Stripe servers. Actions affect subsequent queries.

Slack and GitHub introduce a repeat-action problem because they have no native idempotency for the workflows described here. To make repeat actions safe, the workflow must look up a marker in message metadata or an issue body before acting again. The second lookup must reflect the first action. Similarly, a GitHub PR status change that triggers a Slack notification requires the simulator to update state and fire the outbound event. A stateless response mapping cannot establish that causal chain.

Payment workflows also need controlled failures during execution. Tests of retries, backoff, and graceful degradation require specific conditions such as 429s, 503s, and timeouts. Resetting between fault scenarios prevents a deliberately injected failure from becoming unexplained contamination in the next test.

Hermetic isolation for AI agents that call external APIs

An agent can produce coherent text while sending malformed JSON to a payment API. That makes a plausible answer insufficient evidence of correct tool use. Contaminated external state creates another ambiguity because the agent’s behavior depends on what it observes. A resource left by an earlier run may change the current response even though that response still looks right.

Reinforcement learning makes this requirement part of the execution model. Training episodes need deterministic, controllable state that can return to a known baseline, with independent environments scaled across thousands of containers. Adversarial conditions include rate limits, mid-workflow tool failures, and customers changing their minds. These are behavioral stress tests. Without isolated resets, the episode’s conditions include accidental history as well as the intended challenge.

AWS’s ToolSimulator within Strands Evals provides an LLM-powered framework for testing agents that depend on external tools at scale without risking live calls that expose PII or trigger unintended actions. The isolation standard remains independent of the framework. Resettable state gives each episode a clean start, and realistic seed data gives it a world like the one it will work in. Fault injection tests behavior under controlled adversity. Continuous verification then checks that this world still matches the API the agent will call in production, beyond simulated interactions that look convincing.

What a CI pipeline looks like when isolation is built in from the start

Initialization must be an explicit pipeline responsibility. Environment-specific configuration is injected at the pipeline level, with no environment-specific values embedded in test files. Each job boots its own simulator container and waits for a real health check. Process startup alone is not readiness. Where the application requires a database, the job runs the same migration path used by production to create its test schema.

Once dependencies are ready, the job resets its simulator through the control API to the pre-seeded baseline. It loads only the additional records needed for the scenario and runs the integration tests against that isolated instance. Parallel jobs perform this sequence independently. If tests within a job share mutable state, they need an isolation boundary that prevents concurrent mutations too. A per-job container does not by itself make every test within it hermetic.

A separate fault-injection stage starts by resetting to the baseline again. It injects a specific rate limit, timeout, or 5xx condition and checks the application’s retry and backoff behavior. Each fault scenario gets a reset before the next begins. After execution, containers and temporary volumes are destroyed even on failure, and the pipeline asserts that the working tree and fixture root contain no unowned artifacts.

Blocking tests protect critical contracts synchronously before merge, while quarantined tests remain visible and produce evidence without blocking delivery during investigation. Asynchronous tests exercise broader workflows after the fast gate clears. An integration suite’s results become a reliable gate when the pipeline controls the starting state, preserves isolation during execution, and makes leaks visible without training developers to disregard failures.