Test and quality-control system

Owner-facing guide in Vietnamese: harness.nemo12.com (SRC-1434, VitePress on Cloudflare Pages, source in apps/harness/docs). This page stays the technical source.

Owner, 18.10.2026: redesign testing and QC to be very fast, minimal and cheap; audit every place a test runs. This page is the design and the audit it rests on. Numbers are measured, not estimated, unless marked.

Principles

1. Each stage checks only what it can still catch cheaply. Commit time catches mistakes in the files you touched, CI catches the whole-tree view, the weekly check catches slow things, bookkeeping and dependency upgrades, production checks catch what no test sees.
2. A blocking gate must name what breaks for a user, a deploy or the data when it is skipped. Everything else is a report: it tells you, it does not stop anyone.
3. Pay for work, not for overhead. GitHub rounds every job up to a minute and every job re-installs, so CI is one job. Gates run in parallel and only for the paths they cover.
4. One gate list. CI runs the same groups as the hook (scripts/run-checks-groups.mjs); there are no CI-only one-line gates (check-ci-gates-local keeps it that way).

Stages and budgets

Stage	What runs	Budget	Measured (18.10.2026)
pre-commit	autofix (last_reviewed, gen:reference, skills catalog); eslint + typecheck of staged workspaces; run-checks code --staged (parallel, path-aware); run-checks docs --staged, no-clobber, SRC canonical on docs commits	< 5 s	~4 s for a learn or api commit (was 18-20 s)
pre-push	unit tests of touched workspaces and their consumers (test-affected); web mobile audit when apps/web/src changed	< 60 s	14 s for learn + api
main CI (trunk-based: no PRs)	ONE job: scope, install from cache, typecheck + lint + unit tests + API suite in parallel, check:code, check:docs, ledgers, bindings; on main also migrations and deploy	< 5 min, 1 job	see below
weekly (npm run weekly, Sunday 22:00, checks main; owner 18.10.2026: "Mỗi tuần 1 lần thôi cho tiết kiệm")	check:report (non-blocking), minor/patch dependency upgrades, npm audit fix and removal of unused dependencies (knip) tried together and pushed when green (scripts/deps-upgrade.mjs, replaces Dependabot, SRC-1444), npm audit --omit=dev, E2E: the whole suite of every app (owner 18.10.2026: free on the Mac, so 10 times the old 10% set; --quick-e2e keeps the list in scripts/lib/weekly-e2e.mjs); macOS notification	< 30 min	learn full suite 5.4 min (903 cases); quick set 49 s
production	synthetic checks every 15 min (every public site, deployed Worker vs main), check-production after each main run and daily, Sentry	continuous

Gate tiers

npm run check:code and npm run check:docs are blocking: types, builds, seeds matching their source, the /v1 contract, migration safety and numbering, bindings, owner-mandated text rules (em dash, date format, English identifiers, design stack), content gates that guard what learners see.

npm run check:report is not blocking: docs bookkeeping (REQ and SRC canonical, traceability structure, coverage, freshness), one-way ratchets (shadcn adoption, learn UI debt, strict UI audit), calendar-driven checks (test quarantine expiry) and harness links. The weekly check of main reports the red ones. Before the split, a stale last_reviewed or an SRC row without a canonical doc held every app's deploy.

CI cost, before and after

Before: four jobs (scope, gates, api-test, main), 5.9 billed minutes per PR run on average, 4.4 per main push, 3 for a docs-only PR that checks for 5 seconds; about 9,000 billed minutes a month (estimate from 24 sampled runs; the billing API is not readable with the session token). Seven gates ran twice in the same job, one step never ran (it read steps.scope in a job without that step), and PR runs saved their own 195 MB node_modules cache, pushing the repo to 10.6 of its 10 GB cache and evicting main's copy.

After: one job. A PR run bills its own duration rounded up once (expected ~2-3 minutes), a docs-only PR ~1 minute. Only main saves the node_modules cache; the npm download cache (setup-node cache: npm, another ~208 MB per lockfile) is gone. 124 dead caches were deleted (10.6 GB to 1.4 GB).

A package-lock.json change no longer retests and redeploys all 21 apps: scripts/lock-affected.mjs walks each workspace's dependency closure in the old and new lockfile and scope.sh takes only the workspaces it reaches (root package.json still means all; an unreadable case prints all). On the last six lockfile commits of main: one reached a single workspace, one five, and the design-system dependency changes still reached every app.

Local gate timings, before and after

Command	Before	After
check:code (whole group)	15.9 s, 51 gates in a row	5.7 s in parallel
check:code --staged, learn commit	15.9 s	1.9 s (37 gates, 9 skipped by path)
check:docs	12.2 s (check-docs-fresh alone 10.5 s)	0.5 s (4 blocking gates)
check-docs-fresh	10.5 s, one git log per doc	0.4 s, one git log pass, identical result on 1,090 docs

Hooks and gates audit, 18.10.2026 (SRC-1444)

The owner asked whether the hooks, and the tests and gates they call, are still right. Measured on the shared tree and a fresh worktree:

Finding	Fix
Every Claude hook starts in ~55 ms; SessionStart hooks 1.4 s and 2.6 s	Fine, no change
The Claude hook tests (block-dangerous, project-scope) passed but no gate ran them	In check:code with paths: .claude/hooks/
Three gates red on the shared tree with misleading errors ("Could not resolve @nemo12/walk", an esbuild crash): node_modules lagged package-lock.json by 180 packages	scripts/check-node-modules.mjs (40 ms) in pre-commit and the SessionStart notice
pre-push ran the whole typecheck twice when it failed (stdout went to /dev/null, so it ran again to print)	One run, output captured; temp files removed by a trap
pre-push tested workspaces one after another	In parallel; a red one prints its log
The E2E list named two specs that moved to apps/grammar and apps/sat in the learn split; the night run would have gone red	List per app, program apps included; renamed scripts/lib/weekly-e2e.mjs
Dependabot kept opening PRs nobody merges in trunk-based work	Removed; scripts/deps-upgrade.mjs in the weekly run, holds in scripts/deps-hold.json
Nothing found unused dependencies: knip found 43 (Radix single packages replaced by radix-ui, unused fonts, uqr)	knip (knip.jsonc) removes them weekly; check:report lists what is left
The production-health notice expected a daily cron	Warns when a commit older than 2 h was never checked, or after 8 days

Gates that still read the whole repo, merge candidates (four IELTS content gates into one, the two claim gates into one test, the two i18n key gates into one), and wording left from the PR flow in skills are follow-ups.

Audit findings kept for later

• Speed to production was set by the nightly batch, not by the gates. Resolved the same day: the owner moved to trunk-based work (commit, git pull --rebase, git push origin HEAD:main); land, ready/* and the nightly merge are gone, and the nightly job only checks main.
• Low-value tests: 6 source-regex tests (5 in apps/web, moving to nemo12-www), a byte-identical useReferralClaim hook and test in learn and marlins, content-length checks dressed as unit tests (apps/learn/src/books/chapters.test.ts). The suites are fast (api 3,144 cases in 10 s), so these cost clarity more than time.
• E2E: never on GitHub (gate check-no-ci-e2e, SRC-1444); the weekly check runs the whole suite on the Mac (owner 18.10.2026: "Nếu e2e test chạy trên local, ko tốn tiền thì có thể mở rộng gấp 10 lần"). The full learn suite measured 5.4 minutes, not the ~35 minutes once quoted.
• Not adopted: TypeScript 7 native, oxlint or Biome. Typecheck is 6-10 s and cached lint 2 s; the switch would add risk for seconds.

Sources of the design

DORA continuous integration (build in a few minutes); Fowler's practical test pyramid (few end-to-end tests, order stages by speed); Google's test sizes and flakiness data (Software Engineering at Google, ch. 11); the CI workflows of Vite, Astro, Hono and Cloudflare workers-sdk (cancel stale runs, skip docs-only work, one aggregate gate); Cloudflare rollbacks and gradual deployments for shift-right safety.