Deciding What to Mock vs. Hit Live in CI

A team mocks POST /v1/payment_intents, asserts a 200, and merges the pull request. The first real checkout in staging fails anyway, because the handler never stored the PaymentIntent ID and the second step of the webhook path reads undefined. The mock proved that the response had the right shape. It said nothing about whether the code could carry that response through a full lifecycle. That gap between shape and lifecycle is the entire problem with how most teams decide what to mock and what to hit live in CI: the decision gets made by convention, not by what the test actually needs to prove. The opposite failure is just as common and just as costly. A team that hits a live third-party API on every pull request push is one rate-limit spike or vendor outage away from a blocked release for reasons that have nothing to do with their own code. Neither habit is a policy. State dependency, external side effects, rate-limit exposure, and fidelity requirements are knowable before a single line of test code gets written, and a framework built on those four signals replaces guesswork with a routing decision that holds up under scrutiny.

The four signals that determine where a test belongs in the pipeline

Every test carries four properties, and reading them together tells you which layer of the pipeline that test belongs in. A test that creates a resource with POST and then retrieves it with GET depends on state. A test that checks for a 400 response on malformed input does not. Stateless tests can be answered by a stub. State-dependent ones cannot, because a stateless mock has no memory of what happened in an earlier step and can't tell you whether the customer created in step one still exists by step three.

The second signal is external side effects. If an agent has write access to live tools, it will leave synthetic garbage data behind in staging or production, and that mess multiplies with every retry, every branch, and every run.

The third signal is rate-limit exposure: how many parallel jobs touch this test, and what does the cumulative call volume look like across a sprint? Rate limits don't degrade politely. They throw 429s that fail a test for reasons that have nothing to do with whether the code works, and tracking that down can eat hours that should have gone to real debugging.

The fourth signal is the fidelity requirement: does the test need to prove that the real API behaves the way you expect, or does it only need to prove that your code handles a known response shape correctly? Schema and contract tests need real fidelity. Error-handling tests usually don't. Mixing up these two needs is what sends teams chasing live APIs they don't need, or keeps them parked on mocks past the point where that's safe.

Mapping signals to pipeline stages

A three-tier pipeline turns those four signals into an actual routing mechanism: fast mocked runs on every pull request, sandbox or contract runs on merge, and live or near-production runs at release gates.

Tier 1 runs on every push, and it relies on stateless stubs and in-process mock servers. It belongs to tests with low state dependency, no external side effects, no rate-limit exposure, and a fidelity requirement that stops at "correct response shape under known conditions." This tier proves that code handles specific response shapes, including error states like 4xx and 5xx that are hard to trigger reliably against a real service on command. It does not prove that the real API still returns that shape today, or that a sequence of calls produces consistent state across steps.

Tier 2 runs on merge or on a schedule, and this is where stateful simulators and schema contract runs live. It belongs to multi-step flows with real state dependency, and to tests that check the real API's current response schema against your mock definitions. Validating mock specifications against the real API, or its sandbox, catches drift before it reaches production. If you schedule a check of the response schema against your mock definitions, then when a vendor ships a breaking change, the contract test fails before a customer notices anything.

Tier 3 is the pre-release gate, and it runs against a real sandbox or a near-production environment. It covers the full end-to-end lifecycle and tolerates a wider side-effect surface, provided that surface stays an isolated sandbox. A nightly, manually triggerable flow covering checkout, webhooks, and subscriptions on a cron schedule, never on every pull request push, is a practical example of the cadence this tier calls for.

Side by side, the four signals and three tiers make the routing close to mechanical: low state dependency, no side effects, no rate-limit exposure, and shape-only fidelity point to Tier 1; state dependency or schema fidelity points to Tier 2; full lifecycle fidelity with a controlled side-effect surface points to Tier 3.

Stateful simulation as the missing layer between stubbing and live testing

Stateless mocks and live APIs leave a gap between them, and that gap is exactly the multi-step workflows that need memory across calls. That's fine for checking a single endpoint in isolation, but it misrepresents anything that depends on sequence. A stateful simulator behaves differently: post a resource and it gets stored, the next GET returns it, the next list call includes it, a state machine enforces which transitions are even valid, and webhook events fire when state actually changes.

The checkout failure described earlier, where the PaymentIntent ID was never stored and the webhook handler read undefined on the next step, is exactly the class of bug a stateful simulator catches and a stateless mock lets through. Drift compounds the problem: hand-maintained stateless mocks drift fastest, because every change to the real API means a manual update to the stub, and that gap usually surfaces in production.

State dependency is the clearest of the four signals for a reason: a test that creates a resource and then retrieves it simply cannot be answered by a stateless stub, because the stub has no memory of what it was asked to create. Stateful API simulators like Rystic hold that cumulative state across calls, so they are the right tool for proving a sequential workflow actually works end to end without needing to touch a live API. When a vendor changes its API, the simulator's published fidelity score moves, giving a team visibility into drift inside CI ahead of a surprise from a customer.

How AI agents break the standard routing logic

AI agents add a second source of non-determinism on top of whatever variability already exists in a live API, and that stacking makes the case for mocking the API layer stronger while it demands a different way of checking results. Traditional deterministic testing assumes a fixed input produces a fixed output. An API that's also flaky or rate-limited tangles two sources of failure instead of one, turning root-cause analysis into an hours-long dig.

For MCP-based agent pipelines, three interception points are available, and each tests something different. Mocking the LLM itself to emit predefined tool-call requests checks only the MCP server's JSON handling and says nothing about the agent's reasoning. Mocking the MCP transport validates the JSON-RPC messages: tool names, schema correctness, argument routing. Mocking the underlying third-party API at the network layer leaves the agent talking to a real LLM and the LLM talking to a real MCP server, while the MCP server itself talks to a mocked version of the third-party SaaS API, which lets a team check pagination, rate limits, error envelopes, and auth refresh logic directly.

The network-layer approach holds up best, and it maps cleanly onto the Tier 1 and Tier 2 structure already laid out: the simulator carries state and fidelity, and the agent's reasoning gets tested against it rather than against a live, moving target. A practical CI cadence follows the same three-tier shape applied to evaluation instead of function: a small prompt smoke set on every pull request, a full prompt regression set on merge to main, and a larger benchmark set at release.

RL and agent training loops need deterministic, controllable state that can scale across many containers at once, and a live API can't offer that. A verified stateful simulator, including the kind Rystic builds, can meet that need directly. Stripe's own "Minions" pattern reflects the same principle from a different angle: Stripe runs unattended coding agents in disposable sandboxes with no internet access and no production data, and the real test suite and CI pipeline decide whether the work passes, not the agent's own claim that it succeeded.

Fault injection as a required column in the routing decision, not an optional extra

Fault injection belongs at the mock layer because a live API can't produce a controlled failure on demand, so you can't do resilience testing without a simulator built to support it. The failure classes to inject in CI start with latency, using delay headers that trigger a client's timeout handling, and extend to protocol faults such as malformed JSON or the wrong content type, semantic faults where the JSON is valid but the values inside it are impossible, and partial success, where a request gets accepted but its response is lost, which is the scenario that actually tests idempotency. That idempotency case matters most for retry testing: confirming that an agent passes idempotency keys to the mock server shows that a network timeout doesn't quietly create a duplicate record on retry.

A live API cannot return a controlled 503 on command, cannot delay a response by a specific number of milliseconds, and cannot hand back semantically invalid JSON, so a team that puts off fault testing until it reaches the live layer isn't doing fault testing. Checked against the four signals from earlier, fault injection tests have zero rate-limit exposure, zero external side effects, and a fidelity requirement aimed squarely at the failure mode rather than the happy path, which places them in Tier 1 by the framework's own logic. Rystic treats fault injection as a core simulator feature, with latency, 429s, and 503s configurable at the simulator level, so resilience tests run in CI with the same ease as any happy-path test.

API-specific routing decisions for Stripe, GitHub, and PayPal

If you apply the framework to named APIs, you see the right mock-vs-live split isn't the same from one service to the next, because each one has a different state model, a different side-effect surface, and sandbox tooling of varying depth.

Stripe sandboxes don't move money through real banks or card networks, and Stripe provides test primitives that simulate payout success and failure, disputes, inquiries, and early fraud warnings. The PaymentIntent lifecycle is a textbook case of the multi-step stateful workflow that breaks a stateless mock: create the intent, confirm it, capture it, then receive the webhook, with each step depending on the state the previous step left behind.

GitHub webhooks require HMAC-SHA256 verification through the X-Hub-Signature-256 header, and a mock that doesn't reproduce that header correctly will cause a verification handler to reject the event. You have to trigger redelivery manually through the delivery view or through the REST API, so automated retry testing against the live service is cumbersome at best. That puts webhook handler tests entirely in the mock layer: a simulator that generates correctly signed payloads lets a team test signature verification, event routing, and retry idempotency without depending on live GitHub infrastructure.

PayPal offers a REST sandbox with webhook configuration support, and the standard development pattern uses a sandbox REST app and its webhook configuration, submitting PayPal's transmission headers and the webhook ID to PayPal's verification endpoint. Every PayPal webhook delivery carries the possibility of duplication, so processing has to be idempotent by design. That places idempotency testing in the mock layer, with explicit duplicate-delivery fault injection as part of the suite, while the PayPal sandbox handles Tier 3 contract validation.

Keeping mocks from drifting into false confidence over time

A routing framework only holds up as long as the mocks it depends on stay accurate, and without active drift management, a mock quietly stops telling the truth while still returning green test results. Mocks drift faster than live APIs because people maintain them by hand: every change to the real API requires someone to update the stub, and most teams discover the gap in production.

Three practices keep that drift from compounding. A quarterly cadence of running the suite against a real sandbox in record mode, then committing those recordings as fixtures, pins the exact wire format the third party actually returned, rather than relying on what the mock assumes it returns.

At larger scale, across many squads and microservices, silent mock drift, duplicated configurations, and pipeline failures that cascade from one service into the next become the real operational risk, and the fix involves squad-level ownership of mocks, a centralized registry, and automated drift detection. Rystic's approach to this problem is structural: every simulator gets checked against the real API before each release, and the resulting fidelity score is published. The verification work sits with the simulator's maintainer instead of falling on each engineering team separately, and drift appears as a change in that published score rather than as a production incident discovered after the fact. That's the real distinction between a mock that's merely maintained and one that's actually governed: the fidelity guarantee comes from outside and can be checked, instead of being asserted by the team that wrote the mock.