mirror of
https://github.com/danny-avila/LibreChat.git
synced 2026-11-18 14:08:21 +00:00
Total blocking time is CPU time, and the hosted runner's CPU is not a constant: Lighthouse's own benchmark of the runner moves by more than 2x between consecutive runs of one build on a two-core runner, and blocking time moves with it. A median of three raw measurements against a fixed budget therefore fails on the runner rather than on the page. Rescale each run's main-thread tasks by that run's benchmark index to a reference CPU, assert the lowest run instead of the median, and measure an over-budget batch once more before failing. Budgets move to their own file so a fork with a heavier client changes numbers only.
108 lines
8.2 KiB
Markdown
108 lines
8.2 KiB
Markdown
# Serial database latency Lighthouse CI
|
|
|
|
Run from the repository root with Node 24 and Chrome installed:
|
|
|
|
```sh
|
|
npm ci
|
|
E2E_CHROMIUM_CHANNEL=chrome npm run lighthouse
|
|
# Reuse the production build:
|
|
E2E_CHROMIUM_CHANNEL=chrome npm run lighthouse:run
|
|
# Negative control: this MUST exit nonzero with an LCP assertion failure:
|
|
E2E_CHROMIUM_CHANNEL=chrome npm run lighthouse:regression
|
|
```
|
|
|
|
`E2E_BASE_URL=http://localhost:3098` selects another local port. Set `CHROME_PATH` if
|
|
chrome-launcher picks the wrong browser — on WSL it prefers the Windows install, whose
|
|
debugging port is unreachable from Linux — and `LIGHTHOUSE_CHROME_FLAGS` to append Chrome
|
|
flags such as `--no-sandbox`. Each run starts a disposable MongoDB and the real Express
|
|
server, registers a local user, and seeds a conversation. No model inference is needed.
|
|
Do not point this test at a deployed service. Playwright refuses to reuse an existing
|
|
server.
|
|
|
|
The existing `benchmarks/mongoose-latency-hook.cjs` adds **250 ms per Mongoose
|
|
Query/Aggregate execution** in the server process. Independent queries can overlap;
|
|
serial queries compound. This is a deterministic approximation of remote database
|
|
latency, not a replica topology or network emulator. It also delays query-based
|
|
writes; native driver calls, bulk operations and cursor batches are outside its
|
|
coverage. Use a TCP latency proxy if those paths need coverage.
|
|
|
|
The runner makes three cold browser navigations to a populated conversation,
|
|
using the production client build and real authentication, config, file and message
|
|
routes. It uses desktop settings with `throttlingMethod: provided` so Lighthouse
|
|
does not replace the measured server delays with simulated network timing.
|
|
Budgets are LCP **4,500 ms**, CLS **0.1**, and TBT **500 ms** (`budgets.mts`). These are
|
|
lab regression budgets, not field web-vitals percentiles; Lighthouse does not measure INP.
|
|
LCP and CLS are asserted against the median of the runs. LCP is mostly the injected
|
|
database delay, so it barely moves with the runner.
|
|
|
|
TBT is CPU time, and a hosted runner's CPU is not a constant: three consecutive runs of
|
|
one build have measured 613, 191 and 159 ms on a two-core runner. The gate therefore
|
|
asserts blocking time **at a reference CPU, on the quietest run**:
|
|
|
|
- Each run's main-thread tasks are rescaled by Lighthouse's own CPU benchmark of that run
|
|
(`environment.benchmarkIndex` against `REFERENCE_BENCHMARK_INDEX` in `measure.mts`) and
|
|
blocking time is summed again, so a runner at half speed measures the same page.
|
|
- The lowest run is asserted, not the median. Contention only adds blocking time, so the
|
|
quietest run is the closest to what the page itself costs.
|
|
- A batch that is over any budget is measured once more, and the verdict uses all six runs.
|
|
A regression is over budget in every run; a busy stretch on a shared runner is not.
|
|
|
|
The runner prints both the measured and the reference-CPU blocking time for every run.
|
|
The test also requires the seeded transcript to be the LCP element, so a fast
|
|
login page, spinner, or empty shell cannot pass.
|
|
|
|
## When the gate fails
|
|
|
|
1. Read the failed audit's value and limit in the budget table the runner prints
|
|
immediately before asserting, and the per-run table above it. A run whose
|
|
`benchmarkIndex` is far below the others was measured on a busy runner; a blocking
|
|
time over budget at the reference CPU in all six runs is the page, not the runner.
|
|
2. Open a `.lighthouse/lhr-*.report.html` report. The console also prints API request
|
|
start/end times. A late request start suggests a browser dependency; a long
|
|
request suggests server work or serial database reads.
|
|
A failed audit keeps each run's measured `.lighthouse/lhr-*-0.trace.json` for
|
|
inspection in Chrome's Performance panel; a passing one discards them. Raw network
|
|
logs are removed and cookie values redacted before upload. Failed audits also save `.lighthouse/startup.cpuprofile` from a separate navigation to
|
|
identify expensive startup functions, including three seconds after the transcript
|
|
appears so later catalog callbacks are sampled. Profiling runs after the measured audits and
|
|
does not change their metrics or budgets.
|
|
3. Inspect the relevant path before changing the budget:
|
|
|
|
| Request / symptom | Code to inspect | Performance change this protects |
|
|
| ------------------------------------------- | ----------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
|
| Config waits for repeated user lookups | `packages/api/src/app/service.ts`, `api/server/middleware/config/app.js` | [#14101](https://github.com/LibreChat-AI/LibreChat/pull/14101) |
|
|
| Message authorization and read run serially | `api/server/routes/messages.js`, `packages/api/src/middleware/messageValidation.ts` | [#14101](https://github.com/LibreChat-AI/LibreChat/pull/14101) |
|
|
| Messages wait for the file map | `client/src/data-provider/Messages/queries.ts`, `client/src/components/Chat/ChatView.tsx` | [#14188](https://github.com/LibreChat-AI/LibreChat/pull/14188) |
|
|
| Startup repeats auth user reads | `api/server/controllers/AuthController.js`, `packages/api/src/auth/userDocCache.ts` | [#14187](https://github.com/LibreChat-AI/LibreChat/pull/14187), [#14343](https://github.com/LibreChat-AI/LibreChat/pull/14343), [#14747](https://github.com/LibreChat-AI/LibreChat/pull/14747) |
|
|
|
|
Reuse already-loaded user/config data. Start independent reads together, but keep
|
|
every read scoped to the authenticated user/tenant and wait for authorization
|
|
before returning data. Do not raise a threshold to hide added round trips.
|
|
|
|
## Adding another scenario
|
|
|
|
`auditPage({ url, cookies, runs, budgets })` in `audit.ts` invokes the Lighthouse CLI
|
|
once per run, redacts authentication headers, prints API timings and asserts the
|
|
budgets. It returns the reports so each scenario can assert its expected final URL and
|
|
LCP content; read the LCP element through the exported `lcpElement(report)` helper rather
|
|
than an audit id, because Lighthouse renames those between majors. `load.spec.ts` owns
|
|
local login state, conversation seeding and transcript assertions; the runner does not
|
|
depend on them.
|
|
|
|
A downstream fork that ships a different client keeps its own numbers in `budgets.mts`
|
|
and leaves the runner alone; calibrate them from the per-run tables of the fork's own
|
|
runs, since a heavier client carries more blocking time on every runner. A fork can also
|
|
import this runner from a separate launch spec, supply cookies from its own
|
|
authentication fixture and pass its own `budgets` for launch thresholds. The database delay hook can be preloaded by that fixture's server
|
|
configuration. Keep provider-specific authentication and provisioning in that spec.
|
|
For launch flows that navigate across documents, also measure entry-to-ready time
|
|
with Playwright: LCP resets on a new document, so final-page LCP alone does not
|
|
cover the entire launch. This lane does not exercise OpenID/Redis cache priming.
|
|
|
|
`lighthouse:regression` preloads a test-only hook that adds 16 real, sequential
|
|
user reads before message retrieval (at least four extra seconds). It uses the
|
|
same page and budget assertions, and never modifies production source. Confirm
|
|
that `largest-contentful-paint` fails, rather than treating any process error as
|
|
proof. Run the normal command again to restore baseline reports. Reports stay
|
|
local or in GitHub job artifacts; session cookie values are redacted before upload.
|
|
Same-repository pull requests also receive the last 80 log lines as a failure comment.
|