Quarantining a valid test is how a bug ships
Tests that sometimes fail are placed in quarantine, and genuine defects are found during that quarantine. A test that fails once in every twenty runs might be poorly written, or it might be exposing a race condition that will eventually affect a customer.
A flaky test usually fails independently of the code it covers, at certain times, on specific commits, on particular runners, and in a different way each time. A genuine intermittent defect is tied to something: a change, a load pattern, or an ordering.
Each mistake has a cost. If you quarantine the test that is catching a real defect, the defect ships. If you chase a test that is truly flaky, the engineer loses a week.
The agent investigates and states a position based on its evidence. It neither disables nor quarantines anything. The team that owns the test makes the final decision.
How the agent is built in Agent Studio
The main input for the agent is historical run data, stored in PostgreSQL and retrieved as full failure records rather than rates, because rates would hide the patterns that answer the question.
Prompt skills list the infrastructure's known instabilities: common components, a process that runs more slowly, and an integration that has a rate limit, so those environmental causes do not need a fresh investigation every quarter.
The system prompt requires every stated position to include evidence that could contradict it. The alternative interpretation holds only under stated conditions. The agent must say whether the data rule that alternative out.
If the evidence is thin, the agent does not decide. If the history is not enough, it says so, and it names what would settle the question.
A Start Scheduled flow gathers the candidates. The Agent Chat node receives the failure record, plus the commits and runner metadata for each occurrence.
A GitHub node supplies the test's source code and its most recent changes, because a test failure after the most recent edit would otherwise be a completely different issue.
Results go to Linear as a single entry for each test, with the relevant argument. A Human in the Loop node keeps the owning team between the finding and any quarantine.
What this agent is built from
- PostgreSQL (Data): the full failure record, not a flakiness rate.
- Prompt skills: your infrastructure's known instabilities.
- System prompt: a stated position with its counter-evidence.
- System prompt: thin evidence reported as thin, with what would settle it.
- Agent Chat (Util): failure record plus commits and runner metadata per occurrence.
- GitHub (Integration): the test's own source and its recent changes.
- When an ActionFlow is enough: If you only need failures grouped by signature and split into new versus recurring, build the workflow. That workflow produces the candidates in the first place.
Frequently asked questions
Start building AI workflows
Create a free account, open a template or a blank canvas, and run your first ActionFlow.
Newsletter
Get product updates
New nodes, agents, and product notes. We send mail only when we have something worth opening.