# CI - Failure Triage and Silent Non-Runs Most CI reds are not caused by the PR: regressions cluster, corruption scatters, and red baselines or infrastructure windows explain whole batches, so classify the batch before you triage a single test. Green checks hide a second trap: conflicting state, fork approval, or trigger semantics can stop CI from running at all. ## Key points - Real regressions cluster in one suite. Corruption scatters one failure per suite across dozens of unrelated modules, and that scatter needs no per-test triage. - Cluster red checks by timestamp before you blame code. Failures inside one external-service error window are infrastructure, and rerunning the affected runs turned whole PRs green. - Group default-branch failures by commit SHA across 24h, 48h, and 7d windows to separate recovered from pre-fix failures. - Enumerate the full failure set before you infer flakes. A partial count makes a minority-failing test look non-deterministic, and paginated APIs hide the real total. - Check whether the default branch is green first. A red baseline blocks every PR, and fixing the baseline unblocked all such PRs with no PR-side change. - A `tail` window of a failed job log can fabricate a root cause. Grep the full log. - Trend failure buckets over 48h instead of re-triaging each new red. Commits built before a fix still show the old signature. - Remove the root cause instead of retrying around it. Pinning the corrupting toolchain to a known-good version went uniformly green after four retry-hardening attempts failed. - One transient upstream error can mask many failures through step ordering. A registry blip plus an `if: always()` job produced dozens of errors that looked independent. - A threshold-check failure is signal, not noise, when the ratio holds steady across runs while absolute times move. - Cross-branch A/B is impossible in one CI run. Self-contained control-versus-treatment tests within the same branch give reviewers re-runnable evidence. - Canary PRs per change shape (docs-only, leaf package, single module, CI config) catch what local tests miss. Four canaries found bugs the local suite passed. - When a CLI returns empty logs for failed jobs, step-level data settles causality: which steps ran, what each downloaded, and a rerun to confirm. - Gates that re-measure identical binaries whenever only dependency files change produce runner noise. An empty-commit retrigger gives a clean pipeline. - A conflicting PR gets zero `pull_request` runs: GitHub cannot build `refs/pull/N/merge`, so only `pull_request_target` (which runs on the head) fires. Merging the base branch in and regenerating the conflicting artifact restored all runs. - Fork PRs land in `action_required` until a base-repo maintainer approves the run. Pushing more commits from the fork side cannot clear it. - A low check count can mean CI never ran: for first-time contributors, GitHub holds runs awaiting maintainer approval, so two passing checks can mean the matrix never executed. - A green CI badge can mean CI never linted the code. Confirm the workflow ran on HEAD, not only on older commits. - Path-filtered classifiers make green CI on a docs-only PR carry almost no signal. Only the always-on checks ran. - `pull_request_target` always runs the workflow file from the default branch because the job consumes secrets. A PR that fixes such a workflow can never go green on its own PR, so merge the fix with a gate override. - When a workflow never triggers on `pull_request`, prove a fix with a `workflow_dispatch` run on the branch: the PR's own checks exercise nothing. - Close and reopen a PR to retrigger bare `pull_request:` workflows. They default to `opened`, `synchronize`, and `reopened` events only. - A PR diff view stays pinned to the base snapshot from PR creation. Stale hunks render although the merge is clean, and a content-identical rebase with force-push triggers the recompute. ## How to apply 1. Check the default branch and the check count before you read PR checks as a verdict. Two passing checks on a first-time contribution mean the matrix has not run. 2. Confirm the workflows ran on HEAD, not only on older commits. 3. Cluster red checks by timestamp. Rerun only the runs inside one external-service error window. 4. Group default-branch failures by commit SHA across 24h, 48h, and 7d windows. 5. Grep the full job log before you trust a `tail` window. 6. Page through the checks API before you infer flakes. 7. Prove a workflow fix with `workflow_dispatch` on the branch when the workflow never triggers on `pull_request`. 8. Retrigger bare `pull_request:` workflows by closing and reopening the PR. 9. Silence a gate that re-measures identical binaries with an empty commit. 10. Restore missing runs on a conflicting PR: merge the base branch in and regenerate the colliding artifact. ## Observations - [ci] Corruption scatters one failure per suite across unrelated modules, while real regressions cluster. #ci (source: agent sessions, 2026-07) - [github] Fork PRs wait for maintainer approval before the test matrix runs. #github-actions (source: agent sessions, 2026-07) - [ci] Fixing a red default-branch baseline unblocked every open PR with no PR-side change. #triage (source: agent sessions, 2026-11) - [github-actions] A conflicting PR gets zero `pull_request` runs while `pull_request_target` still fires. #github-actions (source: agent sessions, 2026-11) ## Relations - relates_to [[resources/documentation/GitHub - Pull Request Workflow]] (PR state and retrigger events decide whether CI runs) - relates_to [[resources/documentation/GitHub Actions - Semantics and Caching]] (pull_request_target and if: always() semantics) - relates_to [[resources/documentation/Renovate and Dependabot - Automation Semantics]] (dependency-only PRs inherit a red baseline) - relates_to [[resources/documentation/GitHub - gh CLI and API Usage]] (paginated checks API and empty job logs) ## Sources - Personal agent-session notes, February 2026 to November 2026