A fake model and an empty environment

Problem: Agent tests that call real models are slow, cost money, need keys, and can read your real credentials.
Idea: Test against a scripted fake model, in an environment with nothing of yours in it, and name regression tests after their issue.

Testing an agent is awkward. The interesting behavior depends on a model that's slow, costs money, needs a key and never answers the same way twice. The easy way out is to let tests call a real model "just for the important ones." This repo closes that door at two levels.

Level 1: a model that doesn't exist

The suite runs against a faux provider: a fake LLM that streams exactly the messages a test scripts for it, tool calls included.

fauxAssistantMessage([
  fauxText("Let me look at the file."),
  fauxToolCall("read", { path: "src/app.ts" }),
])

What's fake

Only the model. It streams in small token-sized chunks like a real provider, and can return errors and stop reasons on cue.

What's real

The agent loop, tool execution, retries, queues, sessions and rendering, all running for real against the scripted stream.

Do not use real provider APIs, real API keys, network calls, or paid tokens.

Level 2: an environment with nothing of yours in it

Faking the model isn't enough if a test can still read your machine. The test script starts from an empty environment and adds back an allowlist:

env -i "${test_env[@]}" npm test

# test_env includes:
HOME=$test_root/home     TZ=UTC      LANG=C
GIT_CONFIG_GLOBAL=/dev/null          GIT_ASKPASS=false
NPM_CONFIG_USERCONFIG=$test_root/npm-userconfig

No credentials reachableA throwaway HOME means no real keys, settings or sessions, even if a test forgets to mock something.
Nothing can promptNo global git config, and an askpass that always fails, so a test can't hang waiting for a password.
Same result everywhereFixed time zone and locale remove a whole class of "works on my machine".
Cleanup that can't hurt youThe script refuses to delete any directory that doesn't carry the marker file it created itself.

Tests that remember why

Every bug fix gets a regression test named after its issue: 81 files like 5208-late-bash-output.test.ts and 3317-network-connection-lost-retry.test.ts. Months later, the filename still leads to the conversation that explains the edge case.

For MCP, CI runs the official conformance suite against the client and fails on any regression from a committed baseline, so support can only ratchet up.

Steal this

Fake the model, not your code, and run tests in an environment that contains nothing of yours: no home directory, no credentials, no global config. Name regression tests after the issue so the "why" survives.

Use it in your code

A prompt for your coding agent

Run it from the root of your repo. It checks whether the idea fits before it changes anything.

Make this project's tests independent of real external services and of my machine's environment.

The idea: replace the external dependency (an LLM, a payment API, any third-party service) with a scripted fake that behaves like the real one, and run tests in a clean environment: a temporary HOME, a fixed time zone and locale, no global git or package-manager config, and no credentials.

1. Find tests that call real external services, need API keys, or read from the real home directory, global config or environment variables.
2. For the most important external service, propose a scripted fake that returns exactly what each test asks for, including streaming and errors if the real one has them.
3. Add or update the test script so it runs with an allowlisted environment, for example env -i with PATH, a temporary HOME, TZ=UTC and LANG=C.
4. Move one or two tests onto the fake and the clean environment, and show that they pass without any keys.

Name new regression tests after the issue they fix. Keep the change small.

Read the source

Where this comes from

Pi (earendil-works/pi), the minimal, extensible agent harness. The files behind this lesson: