--- url: https://docs.nemo12.com/ops/synthetic-checks.md description: >- Every 15 minutes the API Worker probes every public site and compares each deployed Worker with main; it emails the owner only when a check changes state. --- # Synthetic production checks Two failures used to go unnoticed until a person stumbled on them: 1. **main is green but the app is not deployed.** A deploy job is skipped, cancelled or scoped wrong, and production keeps serving an old commit. `scripts/check-production-drift.mjs` can tell, but only when someone runs it. 2. **A page is broken and nobody noticed.** A Worker answers 5xx, or answers 200 with an empty shell. The checks run inside production, on a Cloudflare Cron Trigger, so they cost no GitHub Actions minutes and do not depend on anyone opening a page. ## Where it runs * Worker: `nemo12-api` (`workers/api`). It already has a `*/5 * * * *` trigger, KV (`CONFIG`) and the email path, so no new Worker and no new trigger. The checks run on every third tick (UTC minute 0, 15, 30, 45), inside `ctx.waitUntil` so they never delay the other 5-minute jobs. * Code: `workers/api/src/modules/synthetic/` * `targets.ts`: the URLs, expected statuses and markers; the version targets are derived from `.github/app-paths.json` (the same file deploy uses). * `checks.ts`: probes and the state machine (pure, fetch injected). * `run.ts`: cron entry, KV state, email, logs. * `checks.test.ts`: unit tests with a mocked fetch; `run.test.ts`: the cron entry on a test D1. * The api Worker sets the compatibility flag `global_fetch_strictly_public`: without it a fetch to one of our own `*.nemo12.com` sites goes straight to the origin, skipping Cloudflare Access, and the Access checks would see 200. `api:health` is asked of the Hono app in-process, because a Worker fetching its own custom domain gets a Cloudflare 522. ## What is checked (no login, no test account) **Pages.** Each URL is fetched without following redirects, with a 10 s timeout. | Check id | URL | Expect | | --- | --- | --- | | `page:web` | `https://nemo12.com/` | 200 + `Nemo12` | | `page:learn-login` | `https://learn.nemo12.com/login` | 200 + portal title | | `page:marlins`, `page:partners`, `page:tutors`, `page:docs`, `page:b21`, `page:compass`, `page:reef`, `page:pearl`, `page:playbooks`, `page:mm`, `page:gull`, `page:ui` | site root | 200 + a title marker | | `page:ielts-redirect` | `https://ielts.nemo12.com/` | 301 to `learn.nemo12.com/ielts` | | `page:mentors-access`, `page:admin-access`, `page:coral-access` | Dolphin, admin, coral | 302 to Cloudflare Access login (a 200 would mean Access is off) | | `api:health` | `https://api.nemo12.com/v1/health` | 200 + `"database":"ok"` | A page must fail on **two consecutive runs** before it counts, so one network blip sends no mail, and must pass **two consecutive runs** before it counts as recovered. A check that fails again within 6 hours of recovering is flapping and then needs 8 passing runs (2 hours) to recover, so a page that fails half the time sends one alert pair, not twenty mails a day. Pages are probed 5 at a time: a Worker opens 6 connections at once and a queued probe's timeout already runs. **Deployed version vs main.** For every Worker in `.github/app-paths.json`: 1. Read the live version's `nemo12-commit:<sha>` stamp from the Cloudflare API (written by `.github/scripts/deploy.sh`). A secret change (`wrangler secret put`, the dashboard) deploys a new unstamped version with `triggered_by: secret` and unchanged code: the check walks back over such versions to the last stamp. Any other unstamped live version (a manual deploy) is **unknown** (skipped), as in `scripts/check-production-drift.mjs`. 2. Read main's head SHA from the GitHub API, and the files changed between the two. 3. If none of those files matches the app's `paths`/`shared` patterns, the app is current (being older than main is normal). If some do, the app is **behind**. GitHub caps that file list at 300, and rarely deployed apps are routinely 300+ files behind main with none of them theirs, so a capped list is never read as "behind": the check then asks, for each literal path prefix of the app, for the newest commit on main touching it, and the app is behind only if live does not contain that commit. 4. Behind for more than **2 hours** (counted from the first run that saw it, stored in KV) is a failure. The 2 hours cover the nightly merge plus the deploy pipeline. If the tokens are not set, the version checks are **skipped** (logged as `synthetic_version_checks_skipped`), never reported green or red. Once they are set, being unable to run is a check of its own, `monitor:version-checks`: a 401/403 from GitHub or Cloudflare (a token expired, was revoked or lost its scope; fine-grained GitHub tokens always expire) fails it after two runs, and any other error fails it once the version checks have been blind for 2 hours. Without it an expired token silently turned the "main green but not deployed" detector off. ## Alerts * State lives in KV `CONFIG`, key `synthetic:state:v1` (one JSON object, per check: status, since, detail, consecutive fails, behind-since). * An email is sent **only on a state change**: healthy to failing, or failing to healthy. One email per run lists what broke, what recovered, and what is still failing from before. * The state is saved **after** the email. If the mail did not go out (send error, address on the suppression list, D1 down, which is exactly when `api:health` fails), the transition is put back and the next run detects it and mails again (`synthetic_alert_not_delivered` is logged). * State of a check that no longer exists (target removed, app renamed in `app-paths.json`) is dropped, so it is not listed as still failing forever. * Sent through `dispatchOnce` with template `ops-synthetic-alert` (system alert, `operations` stream, `respectCap: false`, so the 3 per day cap never holds it back). Recipient: `SYNTHETIC_ALERT_TO`, else `IELTS_REPORT_TO`, else `EMAIL_COPY_TO` (the owner's inbox). * Every transition is also logged: `synthetic_check_failed` (level error) and `synthetic_check_recovered`; each run ends with `synthetic_checks_done`. These land in Workers observability, and error-level lines also reach Sentry when `SENTRY_DSN` is set (the api wires `logEvent("error")` to it, see [Sentry](sentry.md)). ## Secrets to set (owner, once) | Name | Where | Scope | | --- | --- | --- | | `SYNTHETIC_CF_TOKEN` | `nemo12-api` secret | Cloudflare API token, Account: Workers Scripts Read | | `SYNTHETIC_GITHUB_TOKEN` | `nemo12-api` secret | Fine-grained GitHub token, repo `dac2205/nemo12.com`, Contents: Read | | `SYNTHETIC_ALERT_TO` | optional `nemo12-api` secret (not a dashboard var: `wrangler deploy` wipes those) | Overrides the recipient | `CF_ACCOUNT_ID` already exists as a var. Without the two tokens the page checks still run. ## Changing the checks * New public site: add a row to `PAGE_TARGETS` with a marker that only the real page has (its `<title>` is usually best; check with `curl -s <url> | grep -o '<title>[^<]*'`). * New Worker: nothing to do; it is picked up from `.github/app-paths.json`. The API Worker must be redeployed to see the change, since the file is bundled at build time. * Reset all state: delete the KV key `synthetic:state:v1`. The next run starts from healthy and alerts only what is failing then (after the two-run rule). Removing a target needs no reset: its state is pruned on the next run after the api deploy.