Back to writing

Four faults behind one red check: moving CI to a self-hosted runner

8 min read

One evening I moved this site's CI off GitHub's hosted Ubuntu runners and onto a Mac mini, two days after doing the same for my app, because the monthly Actions minutes kept running out. The diff was small: five runs-on lines, and the gitleaks download switched from the Linux archive to the macOS one.

From that commit on, the E2E job and the Lighthouse job were red on every run, main included. Neither had run a test or measured a page. Quality, Secret Scan and Link Check passed. The last green run on main was the last one on ubuntu-latest.

Getting both jobs green took four fixes between 03:12 and 04:04 the next morning. There were four faults, and each one hid the next: I could only see one at a time. Times are local, UTC+3.

What the runner is

The runner is the machine I work on. It runs under a dedicated macOS user that is not me. The service starts at boot with --startuptype service, so it has never had a desktop session. There is one runner for this repository, which means one job at a time, and the workflow cancels a superseded run so a stale one cannot sit in the queue.

The part that matters for this story is that the machine persists. A hosted runner is a fresh VM that is thrown away when the job ends. This one keeps its processes, its ports and its installed applications from one job to the next. Three of the four faults come from that.

One small sign of it sits in every log. The service inherited a PATH from my login, with directories the runner's user cannot read, so the setup actions print an EACCES warning for each one. They are noise, and they made the real errors harder to find.

Fault 1: the system Chrome cannot start

The E2E job prints my resume to PDF before the suite runs. The script that does it has preferred the system Google Chrome whenever one exists, and has for months:

const launchOptions =
  existsSync(SYSTEM_CHROME)
    ? { executablePath: SYSTEM_CHROME }
    : {};

Hosted Ubuntu has no Chrome at that path, so in CI the script always fell through to Playwright's bundled Chromium. The Mac mini has one. Under the runner's user it dies at launch:

FATAL:base/path_service.cc:264] Failed to get the path for 1001

Lighthouse CI found the same browser. Its healthcheck printed "Chrome installation found", and then its first run failed with "Unable to connect to Chrome". The Playwright browsers the E2E job had installed one step earlier were never used.

The fix: when CI is set the PDF script ignores the system Chrome, and the Lighthouse script hands Lighthouse the headless shell that playwright install chromium downloads. A local run of either is unchanged. Two of my other repositories on the same machine already installed only Playwright's Chromium in their workflows, so they never launched the system one.

I did not find out why Chrome cannot start under that user. The change routes around it.

Fault 2: a server that outlived its job

With the browser fixed, Lighthouse passed on the runner for the first time and the PDF step failed in a new way. The step was inline in the workflow and had this shape:

pnpm exec next start -p 3000 &
SERVER_PID=$!
until curl -sf http://localhost:3000/ > /dev/null 2>&1; do sleep 1; done
BASE_URL=http://localhost:3000 node scripts/generate-resume-pdf.mjs
kill $SERVER_PID

When the PDF script crashes, the step exits before its last line. On a hosted runner that does not matter, because the VM goes away with the server inside it. Here the server kept running under the runner's user, holding port 3000 and serving the build of whichever commit started it.

The logs show this had already happened twice:

  • 20:12. The first E2E job this runner ever ran, on a pull request. The server came up in 476 ms, Chrome crashed about a second later, the step died. Five minutes later the first run on main hit EADDRINUSE: address already in use :::3000. A restart of the machine at 22:53 cleared that leftover.
  • 02:53 the next morning. The port was free again, so the same thing happened again. The next two runs both logged EADDRINUSE.

Nobody saw it either time, me included, because on every one of those runs the browser crash was the error at the bottom of the log.

At 03:23, with the browser working, the step did this: next start failed with EADDRINUSE, the curl loop was answered by the leftover, and the PDF script printed my resume from it. The log says Wrote ... shai-sinai-resume.pdf. That file came from a server started three runs earlier, from another commit's build. Then the last line ran:

kill: (19306) - No such process

That is the only thing that failed the step. Everything before it, printing from the wrong server included, had exited 0.

The PDF script has a guard of its own: it checks for status 200 and for my name in the h1. A stale copy of my own site passes both.

The fix is scripts/run-resume-pdf.sh. It uses a port of its own, 3333. It refuses to start if anything already answers there. It checks that its own server process is still alive before it trusts a response. It stops the server in an EXIT trap, however the script ends:

# scripts/run-resume-pdf.sh, the
# lines that matter. ci-free-port.sh
# is the port sweep from Fault 4.
bash scripts/ci-free-port.sh "$RESUME_PDF_PORT"
 
pnpm exec next start -p "$RESUME_PDF_PORT" \
  >"$RESUME_PDF_SERVER_LOG" 2>&1 &
RESUME_PDF_SERVER_PID=$!
trap cleanup_resume_pdf_server EXIT
 
# Still ours, and answering.
kill -0 "$RESUME_PDF_SERVER_PID" 2>/dev/null || {
  cat "$RESUME_PDF_SERVER_LOG"
  exit 1
}

The Link Check step had the same shape on port 3111, so it got the same trap.

Fault 3: WebKit never connects

With the PDF step fixed, the suite started on the runner for the first time. Both Chromium projects ran all 208 of their tests with no failure and no retry. Then webkit-desktop began, and every attempt ended the same way:

TimeoutError: browserType.launch: Timeout 180000ms exceeded.

A process is spawned and never connects. In 19 minutes the run reached 15 WebKit tests and logged 30 launch timeouts. CI retries a failed test twice, the four projects hold 416 tests, and the workflow had no timeout-minutes, so the job was on its way to GitHub's six-hour limit on the only runner this repository has. I cancelled it.

I did not find out why WebKit cannot connect under that user either.

The decision: CI installs Chromium only and runs chromium-desktop and chromium-mobile. The two WebKit projects stay in the config and in a local pnpm test:e2e, where they pass. The cost is that a regression only Safari shows is caught on my machine before a commit, or it is not caught. Every job now carries timeout-minutes, so a stall cannot hold the runner for hours.

Fault 4: cancelling a run leaves a server too

Cancelling that run ended Playwright and left the server Playwright had started, on port 3111. Playwright's config reuses a server it finds on the port only outside CI (reuseExistingServer is !isCI there), so the next E2E job refused it and failed in 67 seconds, which was the right call. A local run would have quietly tested the leftover:

Error: http://localhost:3111 is already used, make sure that nothing is running on the port/url

Link Check ran after it and did something worse than fail fast. Its own next start hit EADDRINUSE, its curl loop was answered by the leftover, and it crawled the leftover. It reported 17 broken links, one per page in the sitemap, all the same file: a webpack chunk returning 400. I did not confirm why. My reading is that the leftover's HTML named a chunk from its own build, and the build directory under it had been replaced by then. The site had no broken links. The report described a server that should not have existed.

So a crashed step and a cancelled run both leave a server behind on this machine, and the workflow cancels superseded runs by design. That makes it routine.

The fix is scripts/ci-free-port.sh, which runs before every step that starts a server: the PDF on 3333, the E2E suite and Link Check on 3111, Lighthouse on 3222. In CI, where a leftover belongs to the same dedicated user, it sends TERM and then KILL to whatever of that user's still listens on the port. Then it asks the port itself with curl, because lsof cannot see another user's process, and refuses to go on if anything answers. Outside CI it kills nothing and only reports.

What guards it, and what does not

The next run was green, and so was the one on main after the merge: 208 tests, 199 passed, 9 skipped, no retries, in under 35 seconds, and Lighthouse on all eight URLs.

What that does and does not show:

  • The port sweep has never met a real leftover on the runner. Link Check's exit trap, added two commits earlier, stops whatever of the runner's user listens on 3111, and the next run found the port free. So ci-free-port.sh has only ever seen free ports there. I ran it locally in three situations: a free port passes silently, a busy port outside CI is reported and left alone, a busy port with CI set is cleared.
  • Its last check could be fooled. It used curl -f on /, so a leftover that answered with an error status read as a free port. A later change counts any answer at all, or a connection that hangs, as a busy port.
  • A trap covers a crash. The sweep covers a cancel. The cancelled run ended Playwright without ending Playwright's server, and no trap of mine was in that process.
  • Two causes are still unknown. The system Chrome and WebKit both fail under the runner's user, and both fixes go around the failure.
  • Safari has no gate in CI. WebKit runs when I run the suite locally.

What I take from the night: on a runner that persists, every step that starts a server needs its own port, a sweep before it and a trap after it, because a crash and a cancel both leave one behind. And on a new runner, fixing one fault is how you find the next.