Synthetic production checks

Two failures used to go unnoticed until a person stumbled on them:

1. main is green but the app is not deployed. A deploy job is skipped, cancelled or scoped wrong, and production keeps serving an old commit. scripts/check-production-drift.mjs can tell, but only when someone runs it.
2. A page is broken and nobody noticed. A Worker answers 5xx, or answers 200 with an empty shell.

The checks run inside production, on a Cloudflare Cron Trigger, so they cost no GitHub Actions minutes and do not depend on anyone opening a page.

Where it runs

• Worker: nemo12-api (workers/api). It already has a */5 * * * * trigger, KV (CONFIG) and the email path, so no new Worker and no new trigger. The checks run on every third tick (UTC minute 0, 15, 30, 45), inside ctx.waitUntil so they never delay the other 5-minute jobs.
• Code: workers/api/src/modules/synthetic/
  • targets.ts: the URLs, expected statuses and markers; the version targets are derived from .github/app-paths.json (the same file deploy uses).
  • checks.ts: probes and the state machine (pure, fetch injected).
  • run.ts: cron entry, KV state, email, logs.
  • checks.test.ts: unit tests with a mocked fetch; run.test.ts: the cron entry on a test D1.
• The api Worker sets the compatibility flag global_fetch_strictly_public: without it a fetch to one of our own *.nemo12.com sites goes straight to the origin, skipping Cloudflare Access, and the Access checks would see 200. api:health is asked of the Hono app in-process, because a Worker fetching its own custom domain gets a Cloudflare 522.

What is checked (no login, no test account)

Pages. Each URL is fetched without following redirects, with a 10 s timeout.

Check id	URL	Expect
page:web	https://nemo12.com/	200 + <title>Nemo12
page:learn-login	https://learn.nemo12.com/login	200 + portal title
page:marlins, page:partners, page:tutors, page:docs, page:b21, page:compass, page:reef, page:pearl, page:playbooks, page:mm, page:gull, page:ui	site root	200 + a title marker
page:ielts-redirect	https://ielts.nemo12.com/	301 to learn.nemo12.com/ielts
page:mentors-access, page:admin-access, page:coral-access	Dolphin, admin, coral	302 to Cloudflare Access login (a 200 would mean Access is off)
api:health	https://api.nemo12.com/v1/health	200 + "database":"ok"

A page must fail on two consecutive runs before it counts, so one network blip sends no mail, and must pass two consecutive runs before it counts as recovered. A check that fails again within 6 hours of recovering is flapping and then needs 8 passing runs (2 hours) to recover, so a page that fails half the time sends one alert pair, not twenty mails a day. Pages are probed 5 at a time: a Worker opens 6 connections at once and a queued probe's timeout already runs.

Deployed version vs main. For every Worker in .github/app-paths.json:

1. Read the live version's nemo12-commit:<sha> stamp from the Cloudflare API (written by .github/scripts/deploy.sh). A secret change (wrangler secret put, the dashboard) deploys a new unstamped version with triggered_by: secret and unchanged code: the check walks back over such versions to the last stamp. Any other unstamped live version (a manual deploy) is unknown (skipped), as in scripts/check-production-drift.mjs.
2. Read main's head SHA from the GitHub API, and the files changed between the two.
3. If none of those files matches the app's paths/shared patterns, the app is current (being older than main is normal). If some do, the app is behind. GitHub caps that file list at 300, and rarely deployed apps are routinely 300+ files behind main with none of them theirs, so a capped list is never read as "behind": the check then asks, for each literal path prefix of the app, for the newest commit on main touching it, and the app is behind only if live does not contain that commit.
4. Behind for more than 2 hours (counted from the first run that saw it, stored in KV) is a failure. The 2 hours cover the nightly merge plus the deploy pipeline.

If the tokens are not set, the version checks are skipped (logged as synthetic_version_checks_skipped), never reported green or red. Once they are set, being unable to run is a check of its own, monitor:version-checks: a 401/403 from GitHub or Cloudflare (a token expired, was revoked or lost its scope; fine-grained GitHub tokens always expire) fails it after two runs, and any other error fails it once the version checks have been blind for 2 hours. Without it an expired token silently turned the "main green but not deployed" detector off.

Alerts

• State lives in KV CONFIG, key synthetic:state:v1 (one JSON object, per check: status, since, detail, consecutive fails, behind-since).
• An email is sent only on a state change: healthy to failing, or failing to healthy. One email per run lists what broke, what recovered, and what is still failing from before.
• The state is saved after the email. If the mail did not go out (send error, address on the suppression list, D1 down, which is exactly when api:health fails), the transition is put back and the next run detects it and mails again (synthetic_alert_not_delivered is logged).
• State of a check that no longer exists (target removed, app renamed in app-paths.json) is dropped, so it is not listed as still failing forever.
• Sent through dispatchOnce with template ops-synthetic-alert (system alert, operations stream, respectCap: false, so the 3 per day cap never holds it back). Recipient: SYNTHETIC_ALERT_TO, else IELTS_REPORT_TO, else EMAIL_COPY_TO (the owner's inbox).
• Every transition is also logged: synthetic_check_failed (level error) and synthetic_check_recovered; each run ends with synthetic_checks_done. These land in Workers observability, and error-level lines also reach Sentry when SENTRY_DSN is set (the api wires logEvent("error") to it, see Sentry).

Secrets to set (owner, once)

Name	Where	Scope
SYNTHETIC_CF_TOKEN	nemo12-api secret	Cloudflare API token, Account: Workers Scripts Read
SYNTHETIC_GITHUB_TOKEN	nemo12-api secret	Fine-grained GitHub token, repo dac2205/nemo12.com, Contents: Read
SYNTHETIC_ALERT_TO	optional nemo12-api secret (not a dashboard var: wrangler deploy wipes those)	Overrides the recipient

CF_ACCOUNT_ID already exists as a var. Without the two tokens the page checks still run.

Changing the checks

• New public site: add a row to PAGE_TARGETS with a marker that only the real page has (its <title> is usually best; check with curl -s <url> | grep -o '<title>[^<]*').
• New Worker: nothing to do; it is picked up from .github/app-paths.json. The API Worker must be redeployed to see the change, since the file is bundled at build time.
• Reset all state: delete the KV key synthetic:state:v1. The next run starts from healthy and alerts only what is failing then (after the two-run rule). Removing a target needs no reset: its state is pruned on the next run after the api deploy.