Finding flaky pipelines

You want the pipelines you already retried that failed anyway, and you want to see which of those failures came from a test and which came from a runner. A table that says only "failed" renders both the same way, and they have different owners and different fixes.

flaky-tests
One tab holds every pipeline that was retried and failed again. Reading the reason column down the table splits it in two: the script failures belong to whoever owns the suite, and the runner failures belong to whoever owns the runners.

Steps

StepKeyWhat it does
1b then Enter on platform, then Tab to pipelinesThe group tree, CI view
2f then status:failed retried:true updated < 7dRetried, and failed anyway
3P, named Failed againKeeps the list as a tab
4read the reason columnSplits the runner failures from the test failures

Read the tab column-wise rather than row-wise. A project that appears three times in a week, twice on main, all script_failure, has one broken suite that keeps being retried until it passes by luck.

Reading the reason column

"Flaky" names two problems that look identical in a list.

A retried failure that fails again with script_failure is a test problem: either a genuinely non-deterministic test, or a broken one that occasionally passes. It belongs to whoever owns that suite.

A failure with runner_system_failure or stuck_or_timeout_failure was never a test. The runners died, or none was free. A retry hides it, which is how a capacity problem gets reported as flaky tests.

The queued column confirms the second case. Jobs that wait nine to fourteen minutes for a runner and then time out are a capacity problem with a specific fix, and nothing in the word "failed" says so.

Row actions

KeyActionKeyAction
Enterthe failing assertion, without opening a logLthe full log, when the assertion is not enough
ycopy the failure block into an issue/narrow to one project to see the pattern

Variations

One project over time

When you already suspect a project, press b down to platform/gateway and filter to ref:main updated < 30d with no status filter at all. Every run of one branch for a month, passes and failures together, is where a test that fails one run in five becomes obvious. The preview also carries a sparkline of the last runs for that ref, so a pipeline that has crept from four minutes to eleven shows up here.

Scheduled pipelines only

Add source:schedule to the Failed again filter and pin that as a second tab. Nightly builds fail differently from merge request pipelines, and usually nobody is watching them.