Fifty failures are usually four problems
Fifty failing tests in one run is a number that tells you nothing except that someone must read fifty stack traces. Most of them share a cause: one fixture, one changed shape, one timeout in a shared dependency.
Grouping by the failure rather than by the test name is what reduces fifty to four. Two tests with unrelated names that fail on the same assertion in the same frame are one problem.
New and recurring need separating too. A failure that appears in nine of the last ten runs is a known condition and belongs in a different list from the one that appeared for the first time in this commit.
The report that gets read says four problems, one of them new, with the commit range attached. The report that gets ignored says fifty.
How the ActionFlow runs on the canvas
Test results arrive through a Webhook trigger from your runner, or an HTTP Request node reads them after a Start Scheduled run.
An Extract Data node reduces each failure to its signature: the assertion, the top frames of the trace, and the test file. It discards the rest, because the rest is what makes the output unreadable.
A Uniq node collapses identical signatures. A Generate Object node groups the remainder by the failure they describe, with member tests listed under each.
A PostgreSQL or MongoDB node stores the history of signatures, which is what allows new ones to be distinguished from recurring ones at all.
An If node splits the groups against that history. A Math node counts how many of the last runs each recurring group appeared in.
A Sort node puts new failures first, then recurring ones in order of how long they have been failing, which is the order an engineer would choose anyway.
A Generate Text node writes the summary, including the commit range and group counts. A Slack node posts it to the engineering channel.
No build is passed or failed here, and no test is disabled. The flow reports.
Nodes this ActionFlow uses
- Webhook (Triggers): results from the runner as soon as the run ends.
- Extract Data (Data): reduces a failure to assertion, frames, and file.
- Uniq (Control): collapses identical signatures before grouping.
- Generate Object (AI Core): groups by the failure described, members listed.
- PostgreSQL (Data): signature history, so new and recurring can be told apart.
- If (Control) with Math (Util): splits against history and counts recurrence.
- Sort (Data): new first, then recurring by age.
- Slack (Communication): posts the summary. The build decision stays with the pipeline.
Frequently asked questions
Start building AI workflows
Create a free account, open a template or a blank canvas, and run your first ActionFlow.
Newsletter
Get product updates
New nodes, agents, and product notes. We send mail only when we have something worth opening.