LibreChat/e2e/lighthouse/README.md
Danny Avila 445668a371
🧭 ci: Assert Lighthouse Blocking Time at a Reference CPU on the Quietest Run (#18406)
Total blocking time is CPU time, and the hosted runner's CPU is not a constant: Lighthouse's
own benchmark of the runner moves by more than 2x between consecutive runs of one build on a
two-core runner, and blocking time moves with it. A median of three raw measurements against
a fixed budget therefore fails on the runner rather than on the page.

Rescale each run's main-thread tasks by that run's benchmark index to a reference CPU, assert
the lowest run instead of the median, and measure an over-budget batch once more before
failing. Budgets move to their own file so a fork with a heavier client changes numbers only.
2026-11-18 07:47:01 -05:00

8.2 KiB

Serial database latency Lighthouse CI

Run from the repository root with Node 24 and Chrome installed:

npm ci
E2E_CHROMIUM_CHANNEL=chrome npm run lighthouse
# Reuse the production build:
E2E_CHROMIUM_CHANNEL=chrome npm run lighthouse:run
# Negative control: this MUST exit nonzero with an LCP assertion failure:
E2E_CHROMIUM_CHANNEL=chrome npm run lighthouse:regression

E2E_BASE_URL=http://localhost:3098 selects another local port. Set CHROME_PATH if chrome-launcher picks the wrong browser — on WSL it prefers the Windows install, whose debugging port is unreachable from Linux — and LIGHTHOUSE_CHROME_FLAGS to append Chrome flags such as --no-sandbox. Each run starts a disposable MongoDB and the real Express server, registers a local user, and seeds a conversation. No model inference is needed. Do not point this test at a deployed service. Playwright refuses to reuse an existing server.

The existing benchmarks/mongoose-latency-hook.cjs adds 250 ms per Mongoose Query/Aggregate execution in the server process. Independent queries can overlap; serial queries compound. This is a deterministic approximation of remote database latency, not a replica topology or network emulator. It also delays query-based writes; native driver calls, bulk operations and cursor batches are outside its coverage. Use a TCP latency proxy if those paths need coverage.

The runner makes three cold browser navigations to a populated conversation, using the production client build and real authentication, config, file and message routes. It uses desktop settings with throttlingMethod: provided so Lighthouse does not replace the measured server delays with simulated network timing. Budgets are LCP 4,500 ms, CLS 0.1, and TBT 500 ms (budgets.mts). These are lab regression budgets, not field web-vitals percentiles; Lighthouse does not measure INP. LCP and CLS are asserted against the median of the runs. LCP is mostly the injected database delay, so it barely moves with the runner.

TBT is CPU time, and a hosted runner's CPU is not a constant: three consecutive runs of one build have measured 613, 191 and 159 ms on a two-core runner. The gate therefore asserts blocking time at a reference CPU, on the quietest run:

  • Each run's main-thread tasks are rescaled by Lighthouse's own CPU benchmark of that run (environment.benchmarkIndex against REFERENCE_BENCHMARK_INDEX in measure.mts) and blocking time is summed again, so a runner at half speed measures the same page.
  • The lowest run is asserted, not the median. Contention only adds blocking time, so the quietest run is the closest to what the page itself costs.
  • A batch that is over any budget is measured once more, and the verdict uses all six runs. A regression is over budget in every run; a busy stretch on a shared runner is not.

The runner prints both the measured and the reference-CPU blocking time for every run. The test also requires the seeded transcript to be the LCP element, so a fast login page, spinner, or empty shell cannot pass.

When the gate fails

  1. Read the failed audit's value and limit in the budget table the runner prints immediately before asserting, and the per-run table above it. A run whose benchmarkIndex is far below the others was measured on a busy runner; a blocking time over budget at the reference CPU in all six runs is the page, not the runner.
  2. Open a .lighthouse/lhr-*.report.html report. The console also prints API request start/end times. A late request start suggests a browser dependency; a long request suggests server work or serial database reads. A failed audit keeps each run's measured .lighthouse/lhr-*-0.trace.json for inspection in Chrome's Performance panel; a passing one discards them. Raw network logs are removed and cookie values redacted before upload. Failed audits also save .lighthouse/startup.cpuprofile from a separate navigation to identify expensive startup functions, including three seconds after the transcript appears so later catalog callbacks are sampled. Profiling runs after the measured audits and does not change their metrics or budgets.
  3. Inspect the relevant path before changing the budget:
Request / symptom Code to inspect Performance change this protects
Config waits for repeated user lookups packages/api/src/app/service.ts, api/server/middleware/config/app.js #14101
Message authorization and read run serially api/server/routes/messages.js, packages/api/src/middleware/messageValidation.ts #14101
Messages wait for the file map client/src/data-provider/Messages/queries.ts, client/src/components/Chat/ChatView.tsx #14188
Startup repeats auth user reads api/server/controllers/AuthController.js, packages/api/src/auth/userDocCache.ts #14187, #14343, #14747

Reuse already-loaded user/config data. Start independent reads together, but keep every read scoped to the authenticated user/tenant and wait for authorization before returning data. Do not raise a threshold to hide added round trips.

Adding another scenario

auditPage({ url, cookies, runs, budgets }) in audit.ts invokes the Lighthouse CLI once per run, redacts authentication headers, prints API timings and asserts the budgets. It returns the reports so each scenario can assert its expected final URL and LCP content; read the LCP element through the exported lcpElement(report) helper rather than an audit id, because Lighthouse renames those between majors. load.spec.ts owns local login state, conversation seeding and transcript assertions; the runner does not depend on them.

A downstream fork that ships a different client keeps its own numbers in budgets.mts and leaves the runner alone; calibrate them from the per-run tables of the fork's own runs, since a heavier client carries more blocking time on every runner. A fork can also import this runner from a separate launch spec, supply cookies from its own authentication fixture and pass its own budgets for launch thresholds. The database delay hook can be preloaded by that fixture's server configuration. Keep provider-specific authentication and provisioning in that spec. For launch flows that navigate across documents, also measure entry-to-ready time with Playwright: LCP resets on a new document, so final-page LCP alone does not cover the entire launch. This lane does not exercise OpenID/Redis cache priming.

lighthouse:regression preloads a test-only hook that adds 16 real, sequential user reads before message retrieval (at least four extra seconds). It uses the same page and budget assertions, and never modifies production source. Confirm that largest-contentful-paint fails, rather than treating any process error as proof. Run the normal command again to restore baseline reports. Reports stay local or in GitHub job artifacts; session cookie values are redacted before upload. Same-repository pull requests also receive the last 80 log lines as a failure comment.