Total blocking time is CPU time, and the hosted runner's CPU is not a constant: Lighthouse's own benchmark of the runner moves by more than 2x between consecutive runs of one build on a two-core runner, and blocking time moves with it. A median of three raw measurements against a fixed budget therefore fails on the runner rather than on the page. Rescale each run's main-thread tasks by that run's benchmark index to a reference CPU, assert the lowest run instead of the median, and measure an over-budget batch once more before failing. Budgets move to their own file so a fork with a heavier client changes numbers only. |
||
|---|---|---|
| .. | ||
| artifacts.mts | ||
| audit.ts | ||
| budgets.mts | ||
| librechat.yaml | ||
| load.spec.ts | ||
| measure.mts | ||
| process.mts | ||
| README.md | ||
| regression.cjs | ||
Serial database latency Lighthouse CI
Run from the repository root with Node 24 and Chrome installed:
npm ci
E2E_CHROMIUM_CHANNEL=chrome npm run lighthouse
# Reuse the production build:
E2E_CHROMIUM_CHANNEL=chrome npm run lighthouse:run
# Negative control: this MUST exit nonzero with an LCP assertion failure:
E2E_CHROMIUM_CHANNEL=chrome npm run lighthouse:regression
E2E_BASE_URL=http://localhost:3098 selects another local port. Set CHROME_PATH if
chrome-launcher picks the wrong browser — on WSL it prefers the Windows install, whose
debugging port is unreachable from Linux — and LIGHTHOUSE_CHROME_FLAGS to append Chrome
flags such as --no-sandbox. Each run starts a disposable MongoDB and the real Express
server, registers a local user, and seeds a conversation. No model inference is needed.
Do not point this test at a deployed service. Playwright refuses to reuse an existing
server.
The existing benchmarks/mongoose-latency-hook.cjs adds 250 ms per Mongoose
Query/Aggregate execution in the server process. Independent queries can overlap;
serial queries compound. This is a deterministic approximation of remote database
latency, not a replica topology or network emulator. It also delays query-based
writes; native driver calls, bulk operations and cursor batches are outside its
coverage. Use a TCP latency proxy if those paths need coverage.
The runner makes three cold browser navigations to a populated conversation,
using the production client build and real authentication, config, file and message
routes. It uses desktop settings with throttlingMethod: provided so Lighthouse
does not replace the measured server delays with simulated network timing.
Budgets are LCP 4,500 ms, CLS 0.1, and TBT 500 ms (budgets.mts). These are
lab regression budgets, not field web-vitals percentiles; Lighthouse does not measure INP.
LCP and CLS are asserted against the median of the runs. LCP is mostly the injected
database delay, so it barely moves with the runner.
TBT is CPU time, and a hosted runner's CPU is not a constant: three consecutive runs of one build have measured 613, 191 and 159 ms on a two-core runner. The gate therefore asserts blocking time at a reference CPU, on the quietest run:
- Each run's main-thread tasks are rescaled by Lighthouse's own CPU benchmark of that run
(
environment.benchmarkIndexagainstREFERENCE_BENCHMARK_INDEXinmeasure.mts) and blocking time is summed again, so a runner at half speed measures the same page. - The lowest run is asserted, not the median. Contention only adds blocking time, so the quietest run is the closest to what the page itself costs.
- A batch that is over any budget is measured once more, and the verdict uses all six runs. A regression is over budget in every run; a busy stretch on a shared runner is not.
The runner prints both the measured and the reference-CPU blocking time for every run. The test also requires the seeded transcript to be the LCP element, so a fast login page, spinner, or empty shell cannot pass.
When the gate fails
- Read the failed audit's value and limit in the budget table the runner prints
immediately before asserting, and the per-run table above it. A run whose
benchmarkIndexis far below the others was measured on a busy runner; a blocking time over budget at the reference CPU in all six runs is the page, not the runner. - Open a
.lighthouse/lhr-*.report.htmlreport. The console also prints API request start/end times. A late request start suggests a browser dependency; a long request suggests server work or serial database reads. A failed audit keeps each run's measured.lighthouse/lhr-*-0.trace.jsonfor inspection in Chrome's Performance panel; a passing one discards them. Raw network logs are removed and cookie values redacted before upload. Failed audits also save.lighthouse/startup.cpuprofilefrom a separate navigation to identify expensive startup functions, including three seconds after the transcript appears so later catalog callbacks are sampled. Profiling runs after the measured audits and does not change their metrics or budgets. - Inspect the relevant path before changing the budget:
| Request / symptom | Code to inspect | Performance change this protects |
|---|---|---|
| Config waits for repeated user lookups | packages/api/src/app/service.ts, api/server/middleware/config/app.js |
#14101 |
| Message authorization and read run serially | api/server/routes/messages.js, packages/api/src/middleware/messageValidation.ts |
#14101 |
| Messages wait for the file map | client/src/data-provider/Messages/queries.ts, client/src/components/Chat/ChatView.tsx |
#14188 |
| Startup repeats auth user reads | api/server/controllers/AuthController.js, packages/api/src/auth/userDocCache.ts |
#14187, #14343, #14747 |
Reuse already-loaded user/config data. Start independent reads together, but keep every read scoped to the authenticated user/tenant and wait for authorization before returning data. Do not raise a threshold to hide added round trips.
Adding another scenario
auditPage({ url, cookies, runs, budgets }) in audit.ts invokes the Lighthouse CLI
once per run, redacts authentication headers, prints API timings and asserts the
budgets. It returns the reports so each scenario can assert its expected final URL and
LCP content; read the LCP element through the exported lcpElement(report) helper rather
than an audit id, because Lighthouse renames those between majors. load.spec.ts owns
local login state, conversation seeding and transcript assertions; the runner does not
depend on them.
A downstream fork that ships a different client keeps its own numbers in budgets.mts
and leaves the runner alone; calibrate them from the per-run tables of the fork's own
runs, since a heavier client carries more blocking time on every runner. A fork can also
import this runner from a separate launch spec, supply cookies from its own
authentication fixture and pass its own budgets for launch thresholds. The database delay hook can be preloaded by that fixture's server
configuration. Keep provider-specific authentication and provisioning in that spec.
For launch flows that navigate across documents, also measure entry-to-ready time
with Playwright: LCP resets on a new document, so final-page LCP alone does not
cover the entire launch. This lane does not exercise OpenID/Redis cache priming.
lighthouse:regression preloads a test-only hook that adds 16 real, sequential
user reads before message retrieval (at least four extra seconds). It uses the
same page and budget assertions, and never modifies production source. Confirm
that largest-contentful-paint fails, rather than treating any process error as
proof. Run the normal command again to restore baseline reports. Reports stay
local or in GitHub job artifacts; session cookie values are redacted before upload.
Same-repository pull requests also receive the last 80 log lines as a failure comment.