Commit graph

45 commits

Author SHA1 Message Date
Marco Beretta
e36598f019
⏩ ci: Record Playwright Shard Timings Automatically and Run Fifteen Shards (#18115)
* ci: Record Playwright Shard Timings From Green Dev Runs
* ci: Run the Playwright Mock Suite on Fifteen Memory Shards
* ci: Run the Playwright Workflow Script Tests Once per Run
* test: Make the Due Schedule E2E Wait for One Engine Tick Only
* test: Cover the Shard Timings Hand-off With Acceptance Scenarios
2026-11-14 01:25:48 +01:00
Danny Avila
bd9108a87a
🧯 ci: Stop Diagnostic Artifact Uploads From Failing or Stalling Test Jobs (#18068)
On 2026-11-11 the artifact service answered every request with an HTML
page. In Playwright run 43092661470 eight memory shards failed in
"Upload Playwright HTML report" after retrying for 9 to 10.5 minutes;
in six of them every test had passed.

Uploads that no job downloads (Playwright reports, traces and
screenshots, the Lighthouse report, the advisory quality reports) are
now `continue-on-error` with a three minute step timeout, so a failed
upload stays out of the job result and a stalled one cannot run the job
into its own timeout. The Jest result uploads were already
`continue-on-error` and gain the same bound. Build outputs and mutation
shards that a later job downloads stay fatal.

merge-gate.test.mjs holds the gated workflows to that split, and
workflow.test.mts holds Change Quality to it.
2026-11-11 19:17:15 -05:00
lia-by-librechat[bot]
d434417089
🦠 chore: Upgrade Node.js and Undici to Security Releases (#18066)
* 🦠 chore: Upgrade Node.js and Undici to Security Releases

* test: Wait for Restored Composer Readiness and Reuse Wrapping Config

---------

Co-authored-by: Lia <lia@librechat.ai>
2026-11-11 19:17:04 -05:00
Danny Avila
9f0646c00d
🚦 ci: Make Dev Workflows Ready for a Merge Queue (#18054)
* 🚦 ci: Make Dev Workflows Ready for a Merge Queue

* ♻️ ci: Evaluate Merge Gate Expressions With the Shared Workflow Evaluator

* 🧺 ci: Share the Installed Tree in Merge Gate Tests the Way ci-setup Requires
2026-11-11 12:06:31 -05:00
Danny Avila
d32403f851
🍃 ci: Provision the mongod Binary Before Any Test Starts (#18053)
Some checks failed
Codegraph E2E Vote Gate / Gate (push) Waiting to run
Codegraph E2E Votes / vote (shard 1/10) (push) Waiting to run
Codegraph E2E Votes / vote (shard 10/10) (push) Waiting to run
Codegraph E2E Votes / vote (shard 2/10) (push) Waiting to run
Codegraph E2E Votes / vote (shard 3/10) (push) Waiting to run
Codegraph E2E Votes / vote (shard 4/10) (push) Waiting to run
Codegraph E2E Votes / vote (shard 5/10) (push) Waiting to run
Codegraph E2E Votes / vote (shard 6/10) (push) Waiting to run
Codegraph E2E Votes / vote (shard 7/10) (push) Waiting to run
Codegraph E2E Votes / vote (shard 8/10) (push) Waiting to run
Codegraph E2E Votes / vote (shard 9/10) (push) Waiting to run
Docker Dev Branch Images Build / publish (push) Waiting to run
Frontend Unit Tests / Client build recovery regression (push) Waiting to run
Frontend Unit Tests / Codegraph select (push) Waiting to run
Frontend Unit Tests / Build packages (push) Waiting to run
Frontend Unit Tests / TypeScript type checks (client) (push) Blocked by required conditions
Frontend Unit Tests / Tests: @librechat/chat (push) Blocked by required conditions
Frontend Unit Tests / Tests: @librechat/client (push) Blocked by required conditions
Frontend Unit Tests / Tests: Ubuntu (shard 1/2) (push) Blocked by required conditions
Frontend Unit Tests / Tests: Ubuntu (shard 2/2) (push) Blocked by required conditions
Frontend Unit Tests / Vite build verification (push) Blocked by required conditions
Playwright E2E Tests / Codegraph select (push) Waiting to run
Playwright E2E Tests / e2e () (push) Blocked by required conditions
Playwright E2E Tests / MCP list_changed (replica count 1) (push) Blocked by required conditions
Playwright E2E Tests / MCP list_changed (replica count 2) (push) Blocked by required conditions
Playwright E2E Tests / MCP Apps (browser and server) (push) Waiting to run
Playwright E2E Tests / e2e (push) Blocked by required conditions
Main Promotion Policy Tests / Main promotion denial and fast-forward tests (push) Waiting to run
Sync Locize Translations & Create Translation PR / Sync Translation Keys with Locize (push) Has been cancelled
Sync Locize Translations & Create Translation PR / Create Translation PR on Version Published (push) Has been cancelled
* 🍃 ci: Provision the mongod Binary Before Tests Start

mongodb-memory-server downloads its ~122 MB mongod the first time a test
starts a server, inside a 15 s beforeAll. Only backend-review.yml cached
~/.cache/mongodb-binaries (the same step pasted four times), and a cache
alone still leaves the download inside the first hook on a miss.

Integration Tests downloaded on every run: 6 failures in 274 runs with
"Exceeded timeout of 15000 ms for a hook" in secrets.integration.spec.ts,
followed by "spawn ETXTBSY" in the suites that started mongod while the
timed-out download was still writing it.

Add .github/actions/mongodb-binary: restore the cache with the existing
key scheme, ask the installed mongodb-memory-server for the binary it
wants (MongoBinary.getPath), make sure that binary sits in the home cache
every working directory resolves, run it once, and save the cache right
away. The step fails the job when no runnable binary can be put in place.

Use it in every job that starts mongodb-memory-server, after dependencies
are installed and before tests, and add the action to the path filters of
the workflows that list their own file.

* 🧪 test: Cover the mongod Provisioning Script Without the Network

Run the real script against a stand-in resolver that keeps the properties it relies on: home cache first, download when missing, and a memoised path until the cache is cleared. Integration Tests runs the suite before it uses the action.

* 🧪 test: Run mongod Provisioning Against the Real Resolver

The suite replaced mongodb-memory-server with a hand-written stand-in, so
it checked the script against a description of the resolver rather than the
installed one. It now runs the script against the real library and
replaces only the network: MONGOMS_DOWNLOAD_MIRROR points at a local
server that serves an archive whose bin/mongod is a shell script, with the
.md5 the library verifies it against.

Recovery also no longer charges a broken copy that was already on disk to
the download: with a broken binary in both the home cache and the root
node_modules/.cache, the first attempt removed one, the retry met the
other, and the step failed without ever downloading. Existing copies are
now discarded one at a time and the download keeps its own two attempts.

* 🧪 test: Reach the Exit-Status Check Behind the Library's System Binary Probe

A system binary that prints nothing is rejected by the library's own version probe, so the case never reached this script's check or the rule that a file the job named is not deleted. Add one that answers the probe and exits non-zero.
2026-11-11 10:31:46 -05:00
Danny Avila
c6b7a5ac12
🎲 ci: Fail Playwright Runs When a Test Passes Only on a Retry (#18051)
Playwright reports a test that fails and then passes on a retry as "flaky" and exits 0, so a
regression that fails two attempts in three merges green. Set `failOnFlakyTests` in CI on the
mock profile (inherited by the Redis transport, registered-email and other derived profiles) and
on the base profile. Retry counts are unchanged: the later attempts still record the trace and
video and show whether a failure repeats.

A run failed this way still prints "N flaky" beside a passed count, so add a reporter that names
each retried pass in the job log and the job summary and says that it is what failed the run.
2026-11-11 10:31:06 -05:00
Danny Avila
97f7fcdbac
🥅 ci: Run Playwright After Merges to Dev and Guard Specs Under Test Selection (#18049)
* 🥅 ci: Run Playwright After Every Merge to Dev

The mock e2e suite ran on pull requests, nightly on main and on demand, so
when a spec broke on dev no run on dev could go red: the vote workflow that
runs the suite after merges cannot fail by design.

Add a push trigger for dev with the pull request path filter. A push runs
the pull request profile with no lane narrowed (10 memory shards, 2 Redis
transport shards, both MCP lanes and MCP Apps), and the gate job reports it.

Pushes take their own concurrency group that never cancels: the run in
progress finishes, and GitHub keeps one pending run per group, so merges
that land meanwhile collapse into one run of the newest commit. Pull
request, nightly and manual runs keep their per-ref cancelling group.

The matrix test now evaluates the workflow's own expressions per event
rather than matching their text.

* 🔭 ci: Always Run Source-Scanning Guard Specs Under Test Selection

Codegraph test selection narrows a pull request run to the test files an
import graph reaches. A spec that reads source with fs has no import edge
to what it reads: toolCapabilityGates.spec.js scans four source trees, was
not selected when #17864 changed one of the files it counts, and failed on
dev afterwards.

List every such spec, with the paths it reads, in .github/guard-specs.json.
Both selectors now add a workspace's guards to a FILES decision and turn a
NONE into a guards-only run when the change touches a path one of them
reads, renames a file, or edits the list. A list that cannot be read runs
the workspace FULL. FULL decisions and every earlier fail-open path are
unchanged.

The rule keys off the decision, not the event. A selection with no file
left in the checkout gets no guards, so the test job still runs FULL on a
stale selection instead of running only the guards.

guard-specs.test.mjs fails when a spec in a gated workspace reads the
repository through fs without being listed, when a listed path is missing,
and when a workflow would not run for a path one of its guards reads. That
last check adds the five backend and one frontend paths the triggers
lacked. The test also runs both real select steps against canned codegraph
responses.
2026-11-11 10:30:09 -05:00
Danny Avila
bb3cac9d95
🧰 ci: Provision Pinned Redis Binaries Without a Distro Mirror (#18055)
* 🧰 ci: Provision Pinned Redis Binaries Without a Distro Mirror

The MCP list_changed jobs and Cache Integration Tests installed redis-server and redis-tools with apt-get at test time. When the runners could not resolve the Ubuntu mirrors, apt-get update spent 3m20s giving up, the install stalled into the step timeout, and the next step found no redis-cli. The version was whatever the mirror served that day.

A composite action now restores redis-server and redis-cli 8.0.5 from the Actions cache and, on a miss, builds them once from the pinned upstream commit with bounded retries. The step fails with one clear error when Redis cannot be provisioned.

* 🔦 ci: Keep Lighthouse Running When the Ubuntu Mirror Is Unreachable

The Lighthouse job installed its pinned Chromium with `playwright install --with-deps`, which runs apt-get first. With the Ubuntu mirrors unresolvable, apt exited 100 and the job failed before any audit ran.

On the runner every library Chromium links is already present with the preinstalled Chrome; the only packages apt added were nine optional fonts. The browser download stays fatal and keeps the lockfile-pinned build, while the system packages go through the bounded best-effort script the Playwright jobs already use.

* ⏱️ ci: Give the Cold Redis Build Room on a Slow Runner

The first hosted runs built Redis in 49, 58 and 103 seconds. A 180 second cap left the slowest of those less than twice its own time, so a slower runner would have failed a job for being slow. The build may now take 300 seconds and the calling steps eight minutes; an unreachable host still fails within 98 seconds. The ready line reports the fetch and the build separately.
2026-11-11 10:29:25 -05:00
Dustin Healy
5081850280
🛠️ feat: Add Slack Gateway Transport Adapters (#17616)
* 🛠️ feat: Add Slack Gateway Transport Adapters

* fix: Share Gateway Execution Wire Contracts and Preserve Stop Scope

* fix: Align Gateway Transport and Installation Contracts

* fix: Centralize Gateway Policy and Durable Stop Observations

* docs: Clarify Transport Adapter and Runtime Ownership

* fix: Separate Stop Event Identity From Execution Authority

* build: align gateway lock with current workspace toolchain

* 🛠️ fix: Cache Gateway Dependencies Across Shared CI Jobs
2026-11-10 21:04:05 -08:00
Danny Avila
38c96a5e3a
🚦 ci: Throttle Codegraph E2E Votes to One Run per Window (#17380)
* 🚦 ci: Throttle Codegraph E2E Votes to One Run per Window

A vote runs the whole mock suite on ten runners, about 154 job-minutes, and
every merge to dev started one: 2,227 job-minutes on 2026-10-22, queued beside
PR CI for the same runners.

With repo variable CODEGRAPH_E2E_VOTE_MIN_HOURS set to N > 0, the votes
workflow skips its push runs (in a concurrency group of their own) and the new
Codegraph E2E Vote Gate decides instead: a merge starts a vote on dev
(workflow_dispatch) once the newest started vote is N hours old, and an hourly
catch-up votes the newest merge no vote covers once the window opens. A merge
votes as before while the catch-up has not run for three hours or the history
cannot be read. Unset, empty or 0 keeps today's push-triggered votes and skips
the gate; CODEGRAPH_E2E_VOTES=off still stops both.

The decision is a pure function in .github/scripts/vote-gate.mjs, tested in
vote-gate.test.mjs and run beside the shard tests in playwright-mock.yml.

* 🚦 fix: Keep an Unfinished Handoff Run from Covering Its Merge

With the throttle on, a merge creates the votes workflow's push run, whose
vote job will be skipped, and the gate's push run at about the same time.
Until that handoff run completed, the gate read it as a queued trial: it
matched the merge on head_sha, so the merge started no vote and waited for a
later catch-up tick, and it also held the window as a started vote.

While throttled, a push run that has not completed is now read as a handoff:
it neither covers a merge nor holds the window. A finished push run that
voted still counts as before.
2026-10-24 08:50:20 +02:00
lia-by-librechat[bot]
53ae22a6a3
💉 chore: Upgrade Node.js to Security Release 24.22.1 (#17332)
Some checks are pending
Publish `@librechat/client` to NPM / pack (push) Waiting to run
Publish `@librechat/client` to NPM / publish-npm (push) Blocked by required conditions
Docker Dev Images Build / publish (push) Waiting to run
Sync Helm Chart Tags / Ignore non-main push (push) Waiting to run
Sync Helm Chart Tags / Sync chart tags (push) Waiting to run
* 💉 chore: Upgrade Node.js to Security Release 24.22.1

* test: Enable Enter Submission in Native Acceptance

* test: Use the Machine Picker in Native Acceptance

---------

Co-authored-by: Lia <lia@librechat.ai>
2026-10-22 09:34:02 +02:00
Danny Avila
27e8930d52
⏱️ ci: Balance Playwright Shards by Recorded Duration (#17019)
Plan the complete mock suite's shards from recorded per-file durations instead of Playwright's equal-count split, widen the memory lanes from six to ten, and give the nightly full suite ten shards per store so it fits its 30-minute job timeout again.
2026-10-16 07:00:43 -04:00
Marco Beretta
c00c776e0e
📦 feat: Scaffold the @librechat/chat Package (#16964)
* feat: Scaffold the @librechat/chat package

Adds packages/chat with core, react and components entries, a tsdown build,
a node-environment jest config, turbo and root build wiring, import boundary
rules, design-lint scope, and core types for the chat transport, stream,
host and status. No code moves yet.

* test: Cover the @librechat/chat boundaries as scenarios

* fix: Run the chat suite in test:turbo and build it for the mock lane

test:turbo now selects @librechat/chat, the mock e2e job builds the package the
chat-package scenarios load, and the synthetic repository in the suppression
scenarios carries the packages/chat/src root lint:design:record names.

* chore: Build the chat package in frontend:ci too

* chore: Run the chat suite in test:all

* fix: Hold core TSX to the chat boundary and scan the package for cycles

* fix: Apply the chat import boundary to JavaScript sources

* fix: Keep nested core files off the components entry

* fix: Close the remaining chat package gates and fence aborts by generation

Blocks imports through the @librechat/frontend workspace name, adds the chat
sources to the i18n scan, the CSS color gate and the client Tailwind content,
and lets ChatAbortRequest carry the generation it means to stop.

* fix: Keep JSX out of the chat core and carry abort and reconnect outcomes

Rejects JSX under the core, copies the chat manifest before the Docker
install, keys the client app build caches on the chat sources, fences
reconnects by generation, and types the abort result the transport returns.

* test: Give the CSS gate's scratch tree the chat root it now scans

* fix: Close the remaining @librechat/chat gates and wiring

Rejects import() and require() across the package boundaries, holds chat
components to the literal-copy rule, compiles TSX in the package's jest
transform, builds the package in Frontend Unit Tests, requires its test job for
main promotion, adds it to the unused-dependency scan, and lists it wherever a
sibling workspace is listed: Dockerfile.multi, the docker smoke paths, the
remaining node_modules caches, the reinstall helper, the bun build chain and
AGENTS.md.

* fix: Tighten the chat boundaries and carry steering and Assistants aborts

Template-literal import() and require() and self-imports through the package
name are rejected, user-facing JSX attributes need localized text, the cycle
scan resolves self-imports, Dockerfile.multi builds the package for the client
stage, and the transport carries steer, cancelSteer, armSteer and the
Assistants abort key.

* feat: Match the chat transport and host contracts to the client's

The transport now reports settled and replaced starts, the stream's protocol
version and resumed flag, reconnect replay, generation status, paused-run
resume with approvals or answers, abort and steer protocol versions, and the
request error shape. The host config and auth carry what the client's
ChatSettings now supplies: rename-while-running, model specs, retention mode
and the user.

* fix: Keep lazy React-entry loads out of the core and type the Assistants stop

Relative import() and require() of the /react entry from core are rejected,
and an abortKey stop resolves to the terminal conversation and run messages the
Assistants route returns.

* fix: Fence start controls by protocol and close extension-bearing entry loads

Settled and replaced start results carry the protocol version, core loads of
the /react or /components entry with a file extension are rejected, and
test:packages:chat builds data-provider first.
2026-10-10 21:00:10 +02:00
lia-by-librechat[bot]
19092e55bd
🖲️ fix: Enable Running Chat Menus and Preserve Renamed Titles (#16697)
Reconciles current dev while preserving manual title ownership, existing menus, and configuration regressions.

Co-authored-by: Lia <lia@librechat.ai>
2026-10-03 23:17:15 -04:00
Lia
9dc8db0ea8 🛶 fix: Preserve Canary Sync Resolutions (#16361)
Retain the tracked-file integration outcomes recorded by original canary sync PR #16361.

No new implementation is introduced. Shared and ambiguous corrections are carried by the final sync-resolution commit so the resulting tracked tree equals frozen canary f66ad88e1c.

Original-PR: https://github.com/LibreChat-AI/LibreChat/pull/16361
Resolution-Sources: 6d7ba72b89, 241c692b63, 6113242b7b, 3c8a5703e7, a45be88578, 3789cbf590, 81e038972f, c505db2256, 420760d8b2
2026-10-01 03:46:19 +00:00
lia-by-librechat[bot]
e396ea9c7b 🫸 fix: Gate MCP App Views and Actions on User Intent (#16297)
* 🫸 fix: Gate MCP App Views and Actions on User Intent

* 🧪 ci: Run MCP Apps Browser Checks on Repository Bot PRs

* 🧲 test: Open App Before Checking Tool-Call Frames

* 🔎 fix: Wait for Stable App Views in Restored Browser Tests

* 🧪 test: Trace Grouped App View Activation in Browser CI

* 🧪 test: Diagnose Grouped App Click Propagation

* 🧪 test: Distinguish Native and React App Click Delivery

* 🧪 test: Inspect Grouped App Button Hit Target

* 🎟️ test: Open Grouped App Views After Scrolling Into Place

* 🧯 fix: Revoke App Approval Before Teardown Handshake

* 🫸 fix: Configure App Limits and Suspend Denied Actions

* 🧾 test: Expect Configured MCP App Limits in API Suites

* 🧱 fix: Isolate Approved App Messages From Composer Drafts

* 🛟 fix: Suspend Aborted App Actions and Restore View Focus

* 🦮 fix: Preserve Focus on App-Initiated Teardown

* 🧳 fix: Exclude Draft Agent State From App Messages

* 🧭 test: Preserve Legacy Null File Overrides

* 🧰 fix: Retain Selected MCP Tools Across App Messages

* 🧾 fix: Acknowledge Accepted App Turns and Disclose Selected Tools

---------

Co-authored-by: Lia <lia@librechat.ai>
(cherry picked from commit 6866732a88)

Original-PR: https://github.com/LibreChat-AI/LibreChat/pull/16297
2026-10-01 03:46:17 +00:00
lia-by-librechat[bot]
627ded714b 🥼 ci: Gate MCP Apps on Executed Browser Tests (#16283)
Co-authored-by: Lia <lia@librechat.ai>
(cherry picked from commit 1c54ffa411)

Original-PR: https://github.com/LibreChat-AI/LibreChat/pull/16283
2026-10-01 03:46:17 +00:00
Marco Beretta
4ffe3825ec 📧 feat: add email address change (#14597)
(cherry picked from commit 9ff251dd96)

Original-PR: https://github.com/LibreChat-AI/LibreChat/pull/14597
2026-10-01 03:46:16 +00:00
lia-by-librechat[bot]
24570b68cf
🐙 ci: Spread Playwright Coverage Across Isolated Shards (#16358)
* 🐙 ci: Spread Playwright Coverage Across Isolated Shards

* ci: Allow Manual Validation of the PR Shard Profile

---------

Co-authored-by: Lia <lia@librechat.ai>
2026-09-25 09:54:03 -04:00
Marco Beretta
fdd5f9ce65
🖼️ fix: Precache PWA Icons and Let Them Refresh (#15654)
* 🖼️ fix: Precache PWA Icons and Let Them Refresh

Production builds set `publicDir: false`, so `client/scripts/post-build.cjs`
copied `public/assets` into `dist/assets` only after `vite build` had finished.
vite-plugin-pwa globs `dist/` from its `closeBundle` hook, inside that build, so
the icon patterns in `workbox.globPatterns` matched zero files and the generated
`sw.js` shipped without a single icon, silently: workbox treats an empty glob as
a no-op. `build:ci`, used by the Playwright lanes, never ran the post-build step
at all and produced a `dist` with no icons or `robots.txt`.

Move the copy into a `copyPublicAssets()` build plugin that runs in
`writeBundle`, which completes for every plugin before `closeBundle`, and drop
the post-build script with its lone `fs-extra` dependency.

Ordering alone was not enough. The plugin defaults `dontCacheBustURLsMatching`
to `/^assets\//`, true only for hashed bundle output. The icons live in
`assets/` under stable filenames, so they entered the manifest with
`revision: null`, permanently immutable, meaning an installed PWA would never
pick up a rebranded icon: the other half of the report. Match Vite's
`[name].[hash].[ext]` shape instead so hashed chunks stay revision-free while
the icons carry content revisions.

Also drop `manifest.webmanifest` from `globPatterns`, since the plugin always
appends it as an additional manifest entry and globbing it duplicated the
precache entry.

A clean `npm run build` now yields 55 precache entries with all five icons
revisioned, against 0 icon entries before.

* 📝 docs: Describe the Full Copy Set in copyPublicAssets

* 🔒 chore: Drop fs-extra From bun.lock
2026-09-06 11:32:09 -04:00
Danny Avila
b88cb3e361
👐 ci: Cancel the Playwright E2E Gate With Its Run (#15415)
The `e2e` fan-in gate guarded itself with `always()`, so a run superseded by
`concurrency: cancel-in-progress` still dispatched it onto a fresh runner after
every other job had gone `cancelled`. It then read
`needs.e2e_shards.result == 'cancelled'`, took the `!= 'success'` branch, and
exited 1 — leaving the required `e2e` check red on a run that was never
adjudicated, and burning a runner slot to do it.

`!cancelled()` keeps the property the gate actually needs, running when a shard
fails so the failure is reported rather than skipped, and drops the one it never
wanted. It also matches `e2e_shards` and `mcp_tool_list_changed`, which already
guard themselves this way; run 33391962579 shows both lanes running normally
past a skipped `codegraph_select`, so a skipped dependency still reaches the
gate.


Claude-Session: https://claude.ai/code/session_012jJbt9x1sCu8ggJQmdjoMy

Co-authored-by: Claude <noreply@anthropic.com>
2026-08-31 09:54:53 -04:00
Danny Avila
3df046f29a
🎓 ci: Graduated E2E Spec Skipping, Wired Dark Until Armed (#15172)
Select emits e2e_skip from the decision's e2e.graduated (skippable-tier specs
whose clean-trial streaks meet the pre-registered bar, 2x where history-
coupled, >=3 distinct days — computed server-side from the shadow's per-spec
ledger). Dark by default: the output is empty unless repo var
CODEGRAPH_E2E_SKIP=on, and even armed it accepts only a well-typed pool-path
list from a non-fail-open decision (traversal segments rejected). Shard steps
subtract the skips from a git-derived run list — unknown names match nothing,
skip-everything falls back to full, and skipped specs still execute post-merge
in every full-suite vote run.

15 verbatim guard tests, scripts extracted from this YAML and executed against
fixtures: arm/disarm, fail-open, malformed/missing/non-array lists, traversal,
out-of-pool paths, unknown names, all-skips fallback, and the 2-real-skips
59->57 arg case. The traversal case caught a real regex gap pre-commit.
2026-08-24 11:53:50 -04:00
Danny Avila
f7b65cb7ae
🎭 ci: Gate Playwright Lanes and Docker Smokes on Codegraph Selection (Stage 2) (#15136)
* 🎭 ci: Gate Playwright Lanes and Docker Smokes on Codegraph Selection (Stage 2)

* ci: surface the fail-open reason in the stage-2 select summaries

* ci: log the stage-2 select decision for harvesting

* ci: fail open on fetch failure or truncated file list; type-strict skip decisions (Codex)

* ci: check curl's exit status before honoring a selection (Codex r2)
2026-08-23 14:58:45 -04:00
Danny Avila
061e4b02a3
🎞️ ci: Fix Playwright ffmpeg Install Hang and Cache the Download (#15065)
Every Playwright job spent a flat 90s on `npx playwright install ffmpeg`,
and none of them ended up with a usable ffmpeg.

The 2.3MB download finishes in under a second; extraction then hangs until
`timeout -k 10 90` reaps it (exit 124, masked by `continue-on-error`). That
is a Node 24.16.0 readable-stream change (nodejs/node#62557) colliding with
yauzl/fd-slicer never firing `close` after EOF, which hangs extract-zip.
It leaves a truncated `ffmpeg-linux` — 5,055,201 bytes against the zip's
declared 5,101,056, segfaulting on exec — and no INSTALLATION_COMPLETE
marker, so Playwright treated ffmpeg as uninstalled. `video: 'on-first-retry'`
has therefore never worked in CI, and every first retry of a flaky test died
in browserContext.newPage: exactly the failure the step existed to prevent.

Upstream fixed it in Playwright 1.60.0 (microsoft/playwright#40747) and Node
reverted it in 24.18.0 (nodejs/node#63834). Node 24.16.0 is pinned in 17
places including the Dockerfiles, so bump Playwright instead — it is a dev
dependency, and `^1.56.1` already permitted 1.62.1; only the lockfile pinned
it. Staying at or above 1.62.1 also avoids the tsconfig-resolution
regressions in 1.62.0.

Caching alone could not have fixed this: a cold cache still hangs, and what
would have been cached is the corrupt binary. So the ffmpeg download is now
restored from cache keyed on the resolved playwright-core version, the
install is skipped outright on a hit, and the cache is only saved once the
binary is verified to actually execute — a partial extraction can never be
promoted into a cache that every later job restores.

Per job: 90s to ~0s on a hit, ~2s on a miss.
2026-08-21 01:36:09 -04:00
Danny Avila
d175741010
🪢 ci: Prevent Playwright Apt Lock Leakage (#14993) 2026-08-19 10:21:12 -04:00
Danny Avila
259f1e0c32
🛰️ feat: Route Live Subagent Controls Across Replicas (#14971)
* feat: route live subagent controls across replicas

* fix: initialize task routing in cluster workers

* fix: harden cross-replica task routing

* fix: expire routed task owners independently

* fix: close cross-replica routing edge cases

* fix: bound owner refresh and close routed cancellation gaps

Refresh owned task registrations in bounded parallel batches so a full
heartbeat pass stays well inside the 30-second directory lease instead of
serializing one Redis EVAL per registration.

Route conversation-deletion cancellation through a dedicated owner-side
scope operation. The owner applies the deletion predicate to its complete
local task set, so a scope holding more children than the model-facing
list cap no longer leaves live executors running after their parent is
removed.

Key a consumed claim's retained response by its operation rather than by
one caller's correlation id, so a later poll recovers a terminal result
whose responses were all lost. Live claim statuses stay uncached so a
poll always observes the task's current state.

Type the model-facing `maxLength` bounds with a narrow local string
schema; the SDK's JsonSchemaType does not declare the keyword, and the
runtime checks continue to enforce the same limits.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H1cCMDrTWaRNkmtKjpWELZ

* fix: retain claimed results apart from control replays

A consumed claim is the only routed response whose loss destroys data, so
it no longer shares one bounded cache with control replays that unrelated
command traffic can evict. Claims are retained under their own budget, and
the requester acknowledges a result it received so the owner releases the
copy immediately instead of holding it for the full replay window.

Resolve the post-delete cancellation pass from durable leases. The deleted
conversations cannot be read back, so re-reading each one only scaled the
cascade while probing the owner directory once per removed id; one lease
read now resolves every live child address instead.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H1cCMDrTWaRNkmtKjpWELZ

* fix: never consume a result the owner cannot replay

Retention for consumed claims is bounded, so a burst of undelivered results
could evict an earlier one and lose it for good. The owner now admits a claim
only while it can retain a worst-case result, and refuses the routed claim
otherwise instead of consuming it, leaving the result on the task for a later
poll. Retained claims are never displaced; control replays keep evicting.

Key a control replay by the command itself rather than by one caller's
correlation id. The transport's own retry reuses a single envelope, but a
caller that saw the owner as unavailable reissues the command under a new id,
which steered, queued, or interrupted the child a second time.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H1cCMDrTWaRNkmtKjpWELZ

* fix: own a claimed result until it is acknowledged

A consumed terminal result is task-owned state, not a cache entry. It now
carries no expiry at all: the owner holds it until a caller acknowledges
receipt, and only then is it released. Retention stays bounded by the
existing admission gate, which refuses a claim the owner could not keep
rather than consuming a result it might drop.

Identify a control by the caller's invocation instead of by its content.
The tool mints one id per invocation and routing carries it, so a routed
retransmission of that invocation replays the owner's result while two
deliberate identical commands arrive under distinct ids and both apply.
Content-derived identity could not tell those apart and would have
answered the second from a stale snapshot.

Wait for the dpkg frontend lock in the best-effort Playwright font step.
Its timeout kills npx while the apt-get it spawned keeps the lock, which
then failed the fatal Redis install and ended the MCP replica jobs before
any test ran (#14983).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H1cCMDrTWaRNkmtKjpWELZ

* fix: treat acknowledgement as part of delivering a result

Publishing an acknowledgement once and ignoring the outcome meant a result
could be reported as delivered while the owner never learned it could let
go, and since that retention neither expires nor evicts, enough lost
acknowledgements would fill it and refuse every later remote claim.

An acknowledgement is now confirmed: publishing to zero subscribers is not
success, it retries inside the ordinary request window, and a claim whose
acknowledgement cannot be confirmed reports the retryable unavailable path
instead of handing back a result the owner still holds. A later poll
recovers that result and acknowledges it, and releasing is idempotent.
Owner registration also outlives the task while a result is unacknowledged,
so the retained result cannot become unreachable.

Take the control invocation identity from the provider's tool-call id
rather than minting one per execution, so replaying the same tool call
stays idempotent while two distinct calls with identical payloads both
apply.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H1cCMDrTWaRNkmtKjpWELZ

* style: sort the widened node:crypto import

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H1cCMDrTWaRNkmtKjpWELZ

* fix: own control invocations and cancellation plans at the task seam

Applies one logical control exactly once for its owning task rather than
in the transport, so a local caller and a routed caller of the same
invocation agree, and reusing an invocation id for different content is
refused instead of silently applied. Invocation identity now comes from
the run, agent, and provider tool-call id hashed to a bounded 32
characters, so a repeated `call_0` never bleeds across tasks and no id can
overrun the routed bound.

Cancellation for conversation deletion is now resolved into a plan while
those rows are still readable, then replayed against the owner directory
after the cascade is deleted. Owner registration is awaited before any
provider work, so a child that cannot be addressed never starts.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H1cCMDrTWaRNkmtKjpWELZ

* style: separate the control invocation map from the next member

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H1cCMDrTWaRNkmtKjpWELZ

* fix: close subagent deletion, claim, and control invocation gaps

Bulk conversation deletion now runs behind a durable owner admission fence.
Draining alone could not close the race: a child admitted on another replica
after the drain read its leases would start provider work against a parent
about to disappear. The fence is written before any lease is read and each
child revalidates it after its own lease is written, so one of the two always
observes the other. It expires on its own, so a process lost mid-deletion
cannot leave an account unable to run subagents.

A terminal child result is no longer kept alive in the owning replica's
memory until someone acknowledges it. Collection is recorded durably on the
child's own message against the polling invocation, so the poll whose
response was lost recovers its own result while a different invocation is
told the result was already collected. Owner-side retention returns to an
ordinary bounded cache that expires, which is what abandoned polls needed:
they can no longer occupy claim capacity until the process restarts.

The deletion drain now cancels each task under one invocation held for the
whole drain, stops re-sending once the owner answers, and retries only
deliveries it could not confirm. A routed control replay also validates the
command fingerprint, so one invocation id carrying different content reaches
the owner to be refused instead of collecting the earlier command's success.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H1cCMDrTWaRNkmtKjpWELZ

* test: assert the drain's calls before restoring its spies

Restoring a spy also clears its recorded calls, so the drain assertions
ran against an emptied mock. Formats the durable claim method tests.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H1cCMDrTWaRNkmtKjpWELZ

* fix: close the follow-on gaps in the deletion fence and result claim

The admission fence now carries an ownership token, so an overlapping
deletion's fence is never lifted by the one that finishes first, and both
fence writes invalidate the cached auth user document. It also covers the
other bulk-delete path: `DELETE /` with no conversation filter removes every
conversation, so it runs behind the same fence rather than a bare drain.

The durable record now decides who holds a one-shot result. An owner replaying
a retained response could hand the same terminal claim to a second invocation;
that invocation is told the result was already collected, while the one that
consumed it still recovers its own. A task with no durable record to arbitrate
keeps whatever the owner answered.

Drain cancellation treats `not_found` as unconfirmed: a missing registration
while the durable lease is still live means the child may be running, so the
command is retried under its invocation once the owner republishes itself.
Control fingerprints are hashed, so retaining one per invocation costs a fixed
few bytes instead of a bounded message.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H1cCMDrTWaRNkmtKjpWELZ

* fix: hold every deletion fence and keep live idempotency records

An owner now holds one admission fence per concurrent bulk deletion instead
of one at a time, so admission reopens only when the last deletion finishes
regardless of completion order. Expired fences are pruned as new ones arrive
and the set is bounded, so an abandoned fence cannot accumulate or lock an
account out.

A failed durable claim write is no longer read as an absent record. Handing a
terminal result over without recording its claimant would let another
invocation collect the same one-shot output once the database recovered, so
the collection reports the retryable unavailable path and leaves the result
for a later poll.

Control invocation records now evict tasks the store no longer holds before
live ones, over a bounded scan. Dropping a live task's record would let a
caller retry apply its queue, steer, or interrupt a second time once the
transport replay had also expired.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H1cCMDrTWaRNkmtKjpWELZ

* fix: keep the deletion fence portable and never drop a live record

The admission fence is written with plain update operators again. DocumentDB
rejects pipeline-form updates, and this runs before any deletion, so the
pipeline form would have failed both bulk-delete endpoints outright on a
supported database target.

An excess deletion is now refused rather than silently displacing the oldest
active fence, which would have reopened admission for a deletion still
running. Expired fences are pruned before the cap is tested, so only genuinely
concurrent deletions count against it.

Control invocation records now sweep every settled task's entry when the
window fills, and a window of entirely live records refuses the new control
before touching the child instead of evicting one. Applying a command with no
room to record it would let the caller's own retry apply it twice.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H1cCMDrTWaRNkmtKjpWELZ

* fix: hold the fence, bound recovered results, and expire stale commands

The admission fence is renewed for as long as its deletion runs, so a very
large account or a stalled database cannot let it lapse while conversations
are still being removed. Only the deletion's own fence is renewed, and the
renewal stops with the operation.

Cancellation now covers every conversation the cascade removed, not only the
ones a plan named: a grandchild lives in its own parent's scope, which a plan
naming the deleted root never reaches.

A routed request carries the deadline its caller waits for, and an owner drops
one that arrives past it. A publisher disconnected mid-request queues the
envelope offline and delivers it after the caller was told the owner was
unavailable, which would otherwise steer a child the caller believes untouched.

A result recovered from its durable child message is bounded like a routed one.
The message keeps the child's untruncated output, so recovery could otherwise
return far more than the routed result limit allows.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H1cCMDrTWaRNkmtKjpWELZ

* test: size the fence window so a renewal can be observed

The renewal test set a 30ms drain timeout but the five-minute grace window
dominates it, so the interval was 100 seconds and no renewal could fire
inside the test's deletion. The grace window is an option now, matching the
store's other timings, and the test sizes the window to 90ms.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H1cCMDrTWaRNkmtKjpWELZ

* fix: wire the durable claim method and close the fence follow-ons

The production store never received `claimSubagentTaskResult`, so every
terminal result would have surfaced as unavailable once a task settled. The
host wires that object from JavaScript, where the factory's parameter type
checks nothing, so the factory now refuses a store missing any method it
calls rather than failing at the first claim.

The routing transport takes a dedicated publisher with the offline queue
disabled. The shared client held commands issued during a disconnect and
delivered them after the caller had given up, which the request deadline
narrowed but could not close inside the clock-skew allowance.

Fence renewal invalidates the cached auth document like the fence and release
paths, and a renewal reporting its entry gone re-takes the fence instead of
letting the deletion run on unfenced. The post-delete cancellation retries a
transiently unreachable owner: the conversations are already gone, so it is
the only pass that can still stop a late-admitted child.

A replaced replay entry no longer leaves its bytes counted, which would have
inflated the cache's total until unrelated responses were evicted.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H1cCMDrTWaRNkmtKjpWELZ

* test: wait on observed lease renewal instead of a fixed delay

The shared-lease renewal test held a 60ms lease and slept 100ms before
asserting an overlapping worker was refused, so a loaded runner that
starved the 10ms heartbeat past the TTL let the lease lapse and the
second worker run. Spy on acquisition and renewal, then wait until a
renewal succeeds past the acquired lease's own deadline — direct
evidence the heartbeat carried it past expiry, with no timing
assumption — and give the lease enough headroom that a stalled timer
no longer decides the outcome.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H1cCMDrTWaRNkmtKjpWELZ

* fix(agents): close the routing, fence, and cache gaps found in review

Five separate seams, each with its own failure:

`Cluster.duplicate` reads its first argument as a startup-node list and
its second as the overrides, unlike `Redis.duplicate`, so the publisher's
`enableOfflineQueue: false` was silently dropped under
`USE_REDIS_CLUSTER` and a command issued mid-disconnect could still
reach a child after its caller was told `unavailable`. Route both
through `duplicateIoRedisClient`.

The control window's capacity refusal ran before the store knew whether
it owned the task, so unrelated local load could veto a cancellation
bound for another replica. Establish that the task is local first and
leave a remote one to its owner's window.

`clearInterval` stops only future fence renewals. One already waiting on
the database could resolve after the release, read its own lifted fence
as expiry, and write a replacement that nothing remained to lift —
closing subagent admission for the account until it aged out. Track the
in-flight renewal, refuse overlapping passes, and await it before
releasing.

Every owner bounds its own task list, but the aggregation appended each
batch whole, so the model-facing list grew with the number of replicas
holding the scope. Cap the merged list while still reading every reply
for the stale-registration sweep.

The admission-fence prune commits independently of the fence that
follows it, so a refused or failed push left the cached auth document
describing entries the collection no longer held. Invalidate whichever
way the second write goes.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H1cCMDrTWaRNkmtKjpWELZ

* fix(agents): cap the merged task list the poll tool actually reads

Each owner bounds its own reply and the remote aggregation bounds their
sum, but `listTasks` merged that bounded remote list with however many
children this replica owns and returned it whole. `check_background_task`
could therefore still receive roughly twice the advertised cap. Bound the
deduplicated, sorted result and export the cap so both seams share one
number.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H1cCMDrTWaRNkmtKjpWELZ

* test: admit every task the merged-list cap test starts

The base store admits ten concurrent runs per scope by default, so
starting 150 at once left most refused for capacity and the assertion
never reached the merge it was written to check. Raise the cap for this
store only; admission is a different invariant with its own tests.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H1cCMDrTWaRNkmtKjpWELZ

* fix(agents): let a deletion notice its admission fence lapsing

Renewal failures were logged and swallowed, so a run of rejected writes
let the last confirmed `fencedUntil` pass while the deletion carried on
believing admission was still closed — long enough for another replica
to admit a child against conversations about to be removed. Track the
deadline only a confirmed write advances, and check it after the drain,
before anything is deleted: nothing has been removed at that point, so
the operation fails closed and the caller retries once the fence can be
held. A lapse detected after the rows are gone is logged instead, since
reporting failure there would invite a retry against conversations that
no longer exist.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H1cCMDrTWaRNkmtKjpWELZ

* test: raise both concurrency caps the merged-list test trips

Raising the per-scope limit left the store-wide `maxRunningTotal` at its
default hundred, so fifty of the hundred and fifty starts were still
refused. Verified against the base store directly this time: with only
the per-scope cap raised it admits a hundred, and with both raised it
admits all hundred and fifty.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H1cCMDrTWaRNkmtKjpWELZ

* fix(agents): close the fence renewal gap and keep running tasks listed

A renewal that started before its deadline but landed after it was still
credited with extending the fence from its own start time, so a window in
which admission stood open was papered over: a child could take a lease
the drain had already read past and the deletion would proceed without
cancelling it. The deadline now only advances when the write lands while
the previous one still holds; anything later records a lapse the fence
cannot be restored backwards over.

The model-facing cap sorted oldest-first and sliced, which dropped the
newest tasks — including children that had only just started running,
and which the poll tool offers no other way to discover. Bound by status
instead: running children first, then the most recent settled results.
Both caps share one helper, and the routed aggregation now bounds after
its loop so the choice is made across every owner's reply rather than by
whichever answered first.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H1cCMDrTWaRNkmtKjpWELZ

* fix(agents): finish the cap and the fence at the seams they still missed

The status-aware cap only reached the requester: an owner's own reply
still sliced positionally, so a replica holding more than the cap dropped
its running children before the requester could bound anything. Both
sides now share `boundedTaskList`.

A fence that lapsed during the deletion itself was only logged. The rows
are gone by then, so failing is still wrong, but the child another
replica admitted while the fence was down is not: the fence is retaken
and the drain repeated to cancel it.

A child's lease renewal had the same retroactive hole the admission fence
had — Mongo filters on the `now` captured before the call, so a write
landing after the lease expired still moves the row forward, while an
owner drain reading active leases in that gap saw the thread as free. The
lease now carries its own deadline and a late renewal stops the executor.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H1cCMDrTWaRNkmtKjpWELZ

* test: cover the lease lapse and the post-deletion re-drain

The owner-side cap shipped with a regression test; these two did not.
One drives a lease renewal that succeeds only after the lease it was
extending had expired and asserts the executor stops; the other lets the
fence lapse during the deletion itself and asserts a second drain runs
while the request still reports success.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H1cCMDrTWaRNkmtKjpWELZ

* fix(agents): close live-task lifecycle gaps

* test(redis): exercise cluster node discovery

* fix(test): type cluster discovery seam

* fix(ci): wait for orphaned apt processes

* fix(ci): reserve time for apt drain

* fix(ci): skip optional fonts in MCP jobs

* fix(agents): recover tasks after owner loss

* fix(agents): preserve local task discovery

* fix(agents): initialize fail-fast cluster publisher

* style(agents): sort routing test imports

---------

Co-authored-by: Claude <noreply@anthropic.com>
2026-08-19 02:16:35 -04:00
Danny Avila
e1ac7d2bda
⚓ ci: Settle the E2E Reply Before the Double-Click Quote (#14840)
* test(e2e): grant MULTI_CONVO.USE in the mock e2e config

`agent-skills-added.spec.ts` drives the composer's `+` command, which opens the
added-model popover. That path is gated on MULTI_CONVO.USE:

    if (!hasMultiConvoAccess || !plusCommandEnabled || isAssistantsEndpoint(endpoint)) return;

The mock config never sets `interface.multiConvo`, so the permission falls
through to the seeded role default and `handlePlusCommand` returns before
opening the popover. The spec then fails on a popover that is absent from the
DOM entirely, which reads as a selector or timing problem rather than a missing
permission.

Set it explicitly, the same way `contextCost` is set just above for the usage
gauge — the mock config's job is to make each exercised feature's gate explicit
rather than inherit a default.

* test(e2e): wait for the reply to settle before the double-click quote

`quotes.spec.ts` › 'summons the popup from a native double-click word
selection' double-clicks a word as soon as `mockReply` becomes visible. But
`sendMessage` resolves on the stream *response*, not on the final render, so the
reply can still be re-rendering.

A streaming markdown re-render swaps out the text node the selection points at,
which collapses the selection — the same mechanism the sibling
`selectionchange` test documents deliberately. A double-click landing mid-stream
therefore loses its selection before the popup can be clicked, and because the
whole gesture is wrapped in `toPass`, every retry re-runs into the same
still-streaming reply rather than recovering from a one-off.

This is a different race from the one #14777 fixed. That one is the *selection*
still settling (touch long-press, native handle drags, block-granularity
gestures) and is handled inside QuoteButton. This one is the *reply* still
streaming, which no amount of component-side settling can absorb.

Observed on a downstream fork running this suite on slower hardware: the test
fails all three attempts, deterministically on the in-memory stream store while
the Redis lane passes the same shard — the in-memory store's final re-renders
land late enough to outlive the gesture. Four separate runs, same split.

Wait for the reply text to hold steady before selecting.

* ci(e2e): install ffmpeg so first-retry video actually records (#14841)

`playwright.config.mock.ts` sets `video: 'on-first-retry'`, but the runner only
installs `install-deps chrome`, which does not include ffmpeg. Without it the
first retry fails inside `browserContext.newPage` while setting up video
recording — before the test body runs.

The cost is the retry itself: a genuinely flaky test loses the attempt that
would have recovered it, and the reported failure is a video-setup error rather
than the original symptom.

Bounded and non-fatal on purpose. The CLI has been observed hanging after the
download completes on these runners, so the step is wrapped in `timeout` and
its failure is swallowed — if ffmpeg cannot be installed the job proceeds
exactly as it does today, and no lane is blocked on it.

Applied to both jobs that run Playwright (`e2e_shards` and
`mcp_tool_list_changed`), since both configure retries.
2026-08-15 10:48:11 -04:00
Danny Avila
6f05f2427b
👷 ci: Stop Optional Playwright Fonts From Failing E2E (#14852)
`npx playwright install-deps chrome` is the third-most-common e2e failure:
three of the last twenty-five Playwright runs died on it, taking the whole
aggregate gate with them. The step is not installing anything CI needs.

The runner's Chrome is an apt package, so apt has already satisfied every
library Playwright lists — the log shows each one "already the newest version".
All `install-deps` adds are decorative CJK/Thai/Cyrillic font packages, ~21MB
pulled from azure.archive.ubuntu.com by seven jobs on every PR. No CI assertion
depends on them: the only spec that screenshots gates its comparison behind
`E2E_VISUAL_SNAPSHOTS`, which no workflow sets, and no baselines are committed.

Keep the install, but demote it. `google-chrome --version` becomes its own
fatal step so a genuinely missing browser still fails loudly and immediately,
while the font install retries with a per-attempt cap and degrades to a warning.
The Redis install in the list_changed job stays fatal — that one is required.
2026-08-14 20:35:37 -04:00
Danny Avila
ee21066590
🏎️ ci: Focus Redis E2E Coverage (#14842) 2026-08-14 12:54:30 -04:00
Danny Avila
dfd4d9dd81
🧩 ci: Close Workflow Path-Filter Gaps (#14728)
* 🧩 ci: Close Workflow Path-Filter Gaps

Six trigger-filter gaps found by reading all workflows against the live
dependency graph (AI-1755, codegraph FINDINGS §6l):

- backend-review/frontend-review: root package.json/package-lock.json now
  trigger unit tests — a lockfile-only dependency bump previously ran zero
  backend or frontend unit tests while every test job installs from it
- agents-integration-tests: widen to the three package src trees it builds
  and imports (was only src/agents/**)
- cache-integration-tests: same shape — verified live that
  packages/api/src/flow/manager.ts (imported by mcp/oauth) matched neither
  integration filter
- docker-smoke: plain Dockerfile had no PR-time validation despite shipping
  via dev-images/tag-images; new node-image-smoke job builds it, gated by
  paths-filter to Dockerfile/.dockerignore changes
- dev-images/dev-branch-images: add config/**, skill/**, .dockerignore —
  the single-stage image COPYs the full build context
- static-checks: eslint.config.mjs now re-triggers the lint job (gap carried
  over from eslint-ci.yml in the #14716 consolidation)
- delete generate_embeddings.yml: fired on docs/**, which no longer exists,
  and its docs-root-path pointed at the same missing directory

* 🧩 ci: Address Codex Review Findings

- Build caches: all 26 build-* keys across 8 workflows now lead with
  root package.json + package-lock.json so manifest-only bumps cannot
  restore stale dists (data-provider embeds the root version); unifies
  the split key families (playwright already hashed the lockfile)
- static-checks: config changes now gate on the ESLint config loading
  and applying to representative files, plus a report-only full-tree
  sweep (70 pre-existing errors at dev HEAD block a hard gate for now)
- docker-smoke: the workflow file itself triggers the plain-Dockerfile
  build so job edits are validated
- dev-images/dev-branch-images: re-include skill/**/*.md after !**.md
  so shipped deployment-skill Markdown rebuilds images

* 🧩 ci: Gate Config Lint Sweep on Regression vs Base Config

Second-round codex finding: the report-only sweep swallowed config-wide
breakage in scoped blocks the representative files don't exercise. The
sweep now lints the same tree under the PR's config and the base ref's
config and fails only when the PR's config produces more diagnostics for
some (file, rule) pair — pre-existing debt never fails the gate, and
fixes are never penalized. Base-config unavailability degrades to the
load gate with a notice. Outcome surfaced in the failure summary.

* 🧩 ci: Harden Config Lint Gate per External Review

- Coverage direction: fail when the PR config stops linting files the
  base config covered (set difference on linted files) — a mis-scoped
  ignores previously only removed diagnostics and passed both gates
- Severity-aware fingerprints: (file, rule, severity) so warn->error
  escalations gate on a clean tree for that rule; downgrades still free
- Hard-fail when the base commit is missing so a future shallow-checkout
  change cannot silently disable the gate; annotate fetch-depth: 0
- Tab-separated fingerprint keys (space-in-path proof), --config on
  both sweeps, EXIT trap for the base config copy, comment on why it
  must live at the repo root (flat-config pattern base paths)
- Narrow skill md re-include with !skill/README.md: top-level README is
  documentation-only; runtime skill Markdown still rebuilds images
2026-08-10 17:26:50 -04:00
Danny Avila
c3a429ddcd
🎨 feat: Add Versioned Theme Foundation (#14709)
* 🎨 feat: Add Versioned Theme Foundation

* 🧩 fix: Keep Theme-Aware Chip Actions Consistent

* 🎛️ fix: Preserve Default Theme Geometry

* 🪪 fix: Keep Theme Identity in Sync

* 🧭 docs: Define Theme Styling Policy

* 🧹 chore: Sort Theme Imports

* 🧵 fix: Preserve Theme Compatibility Contracts

* 🛡️ fix: Harden Theme Compatibility Boundaries

* 🧵 fix: Publish Theme Appearance Preset

* 🐳 fix: Include Theme Preset in Docker Build

* 🪢 fix: Preserve Legacy Theme Compatibility

* 🧭 fix: Harden Theme Lifecycle Boundaries

* 🧱 fix: Align Theme Appearance Defaults

* 🧬 fix: Record Persisted Theme Provenance

* 🧭 fix: Preserve Theme Transition State

* 🧷 fix: Preserve Legacy Theme Contracts
2026-08-10 13:41:03 -04:00
Marco Beretta
82de3bc422
🚦 ci: Reduce GitHub Actions Runner Pressure (#14716)
* ci: reduce GitHub Actions runner pressure

* ci: enforce static check path exclusions
2026-08-09 07:41:00 -04:00
Danny Avila
1bccc2bc18
📡 fix: Refresh MCP Tools After List-Changed Notifications (#14686)
* fix(mcp): handle dynamic tool list changes

Co-authored-by: Pascal Garber <pascal@artandcode.studio>

* test(mcp): fix CI validation

* fix(mcp): keep dynamic tool catalogs live

* fix(mcp): harden dynamic catalog lifecycle

* test(mcp): use typed startup connection

* test(mcp): isolate dynamic e2e fixtures

* fix(mcp): refresh tools after reconnect

* fix(mcp): close dynamic catalog cache gaps

* test(mcp): update OAuth connection mocks

* fix(mcp): preserve app snapshot ownership

* style(mcp): sort connection imports

* fix(mcp): close review race conditions

* fix(mcp): preserve cache ownership edges

* fix(mcp): harden recovery lifecycle

* fix(mcp): guard tool-less app refresh

* fix(mcp): fence distributed cache races

* fix(mcp): retire stale connection state

* fix(mcp): keep tool snapshots authoritative

* fix(mcp): fence stale app tool publications

* style(mcp): sort repository test imports

* test(mcp): mock empty startup publication

* fix(mcp): preserve app publication generations

* fix(mcp): harden publication recovery races

* fix(mcp): address tool catalogs by runtime config

* fix(mcp): load scoped catalogs for assistant writes

* fix(mcp): harden catalog publication recovery

* fix(mcp): serialize forced connection replacement

* fix(mcp): serialize ordinary creation with replacements

* fix(mcp): harden catalog fallback boundaries

* fix(mcp): close lifecycle fencing gaps

* fix(mcp): preserve catalog authority on failures

* fix(mcp): compensate failed catalog mutations

* fix(mcp): fence catalog refresh ordering

* style(mcp): sort agent loader imports

* fix(mcp): cancel stale connection creation

* fix(mcp): fence catalog coordination

* fix(mcp): close catalog race windows

* fix(mcp): harden cross-pod catalog fencing

* fix(mcp): close catalog lifecycle edges

* style(mcp): sort assistant imports

* fix(mcp): reject stale recovery authority

* fix(mcp): restore static catalog on every startup

* fix(mcp): order app catalog publications

* style(mcp): sort catalog revision imports

* fix(mcp): separate catalog allocation and commit fences

---------

Co-authored-by: Pascal Garber <pascal@artandcode.studio>
2026-08-08 13:50:21 -04:00
Marco Beretta
9e6d677751
⚙️ ci: Bump GitHub Actions to Node.js 24 Runtimes (#14689)
* ci: bump GitHub Actions to Node.js 24 runtimes

Clear Node 20 deprecation warnings on runners by moving workflow
actions to majors that declare node24 (checkout, cache, setup-node,
artifacts, Docker buildx/build-push/qemu/login, github-script,
setup-go, Azure login/helm, create-pull-request, axe-linter).

* fix: release leader lock via ioredis on Redis Cluster

@keyv/redis EVAL can surface unhandled MOVED redirects on cluster,
so resign() logged failure and left LeadingServerUUID set. Use ioredis
for leader election SET NX / GET / Lua (same pattern as principals and
concurrency locks) so cluster redirects are retried and resignation
clears the lock.
2026-08-07 11:25:29 -04:00
Danny Avila
52b2ebf948
🧪 test: Run mock E2E against Redis in shards (#14551)
* 🧪 test: Run mock E2E against Redis in shards

* 🧪 test: Isolate local Redis E2E data
2026-07-31 12:10:43 -04:00
Danny Avila
5af12c722e
🎭 ci: Scope Playwright Runs to Relevant Changes (#14388)
Some checks are pending
Docker Dev Branch Images Build / build (Dockerfile, lc-dev, node) (push) Waiting to run
Docker Dev Branch Images Build / build (Dockerfile.multi, lc-dev-api, api-build) (push) Waiting to run
GitNexus Index / index (push) Waiting to run
GitNexus Index / post-index (push) Blocked by required conditions
2026-07-22 12:23:39 -04:00
Danny Avila
dbb771aa7c
🚦 ci: Gate Playwright Runs to Maintainers (#14385)
* ci: gate Playwright runs to maintainers

* ci: guard pull request context explicitly
2026-07-22 12:23:02 -04:00
Danny Avila
a3c92b83c8
🧾 ci: Skip Workflows for Markdown-Only Changes (#14378) 2026-07-21 20:35:59 -04:00
Danny Avila
cb6dbc8f60
⚡ refactor: Migrate @librechat/client build to tsdown (#13596)
* ⚡ refactor: Migrate @librechat/client build from Rollup to tsdown

Mirrors the data-schemas migration. Replaces Rollup (rpt2 + postcss) with
tsdown (rolldown + oxc); the package build drops from tens of seconds to ~0.3s.

- Emit isolated-declaration .d.ts via oxc (dts.oxc) and enforce
  isolatedDeclarations in tsconfig for editor DX (source made clean: explicit
  export type annotations added across src, no `any`).
- Extract component CSS to dist/style.css so the CJS output stays valid
  CommonJS (the prior postcss runtime-injection produced an ESM import in the
  CJS bundle that breaks jest/require). Imported once in the client app entry;
  Vite bundles it for the app.
- Repoint package.json to dual .mjs/.cjs + .d.mts/.d.cts and add ./style.css
  and ./package.json exports.
- Update CI build-cache keys to hash tsdown.config.mjs; remove rollup.config.js.

* 🔧 chore: address Codex review on client tsdown migration

- Add tsdown.config.mjs to turbo.json build `inputs` so changes to the new
  bundler config invalidate the Turbo cache (the shared inputs only listed the
  rollup configs). Also covers the already-migrated data-schemas.
- Name the memoized default export (ControlComboboxMemo) instead of the
  codefix-generated `_default_1`, for clearer stack traces / grepping.
2026-06-08 11:20:03 -04:00
Danny Avila
fbc51f6e66
⚡ refactor: Migrate data-provider Build to tsdown (split tsc dts) (#13597)
Replace the Rollup + `rollup-plugin-typescript2` build with a split
pipeline: tsdown (rolldown) bundles the JS in ~0.2s, and plain `tsc`
emits the declarations to `dist/types` (~2s). Full cold build drops from
~9.2s to ~2.5s (~3.6x) with zero source changes.

Unlike data-schemas, the fast oxc/isolated-declarations dts path isn't
viable here: the package's 78 exported zod schemas produce 374
`isolatedDeclarations` errors (TS9013/TS9038) and a `z.ZodType<T>`
annotation would break the 76 downstream `.extend`/`.shape`/`.pick`
usages. Plain `tsc` keeps the rich zod types intact, and since dts was
never the bottleneck (rollup-plugin-typescript2 was), the win stands.

- dts stays unbundled in `dist/types/` — identical to the prior output,
  so the existing deep `dist/types` imports and the exports `types`
  paths are unchanged.
- ESM output renamed `index.es.js` -> `index.mjs` (via the exports map;
  no consumer hardcodes the old path). cjs/types paths unchanged.
- `./react-query` now emits a real cjs build + types — the exports map
  already promised them, but Rollup only ever built the esm file.
- Kept `rollup` + the plugins used by `server-rollup.config.js`
  (the `rollup:api` server-bundle smoke test in backend-review.yml);
  removed only the deps used solely by the deleted `rollup.config.js`.
- Repointed CI build-cache keys from `rollup.config.js` to
  `tsdown.config.mjs`.
2026-06-08 11:09:16 -04:00
Danny Avila
6bc75d24c8
⚡️ refactor: Migrate @librechat/api build to tsdown (#13595)
* ⚡️ refactor: Migrate @librechat/api build to tsdown

Replace Rollup with tsdown (rolldown + oxc isolated-declarations) for the
@librechat/api package build, mirroring the merged data-schemas migration.

- Add tsdown.config.mjs (cjs output, oxc dts, externalize all bare deps,
  bundle first-party `~/` + relative imports)
- Annotate exports for isolatedDeclarations (codefix-driven). Collapse the
  tokens.ts model->token maps to Record<string, Record<string, number>> and
  switch validation.ts's runtime `files` field from z.any() to z.unknown()
  so no explicit `any` is introduced
- Repoint package.json main/types/exports to tsdown's .cjs/.d.cts output
- Add src/telemetry.ts entry shim so the two index.ts entries don't collide
  in oxc's flat dts output (stable dist/telemetry.{cjs,d.cts})
- Delete rollup.config.js

Build time ~36s -> ~0.5s. No runtime behavior change: 5712 unit tests pass,
both entries load via require(), legacy /api consumes them unchanged.

* 👷 ci: Hash packages/api/tsdown.config.mjs in build-api cache keys

The build-api cache keys hashed `packages/api/server-rollup.config.js`,
which never existed (api used `rollup.config.js`, now removed) — a copy-paste
artifact from the data-provider key that matched no file. Replace it with the
new `packages/api/tsdown.config.mjs` so edits to the build config (entry,
format, externals) bust the api build cache, matching the data-schemas key.
2026-06-08 10:54:48 -04:00
Danny Avila
192703e041
⚡ perf: Migrate data-schemas Build to tsdown with isolatedDeclarations (#13578)
* ⚡ perf: Migrate data-schemas Build to tsdown with isolatedDeclarations

Replace Rollup with tsdown (rolldown + oxc) for @librechat/data-schemas. With the source made isolatedDeclarations-clean, oxc emits .d.ts without tsc, dropping the package build from ~5.8s to ~0.8s (~7x).

- Annotate exported model/method factories for isolatedDeclarations (TypeScript's fixMissingTypeAnnotationOnExports codefix plus hand-authored interfaces); type the ~44 mongoose `any`s and add an explicit PromptMethods interface (previously its declaration was silently dropped by the Rollup build).
- Repoint package.json exports/main/module/types to tsdown output; drop rollup config.
- Config lives in tsdown.config.mjs (native ESM) so CI without a TS-config loader can build it; bundle `dotenv` so the package stays self-contained for its env-loading side effect.
- Fix a latent token `metadata` mismatch the accurate types surfaced: widen TokenCreate/UpdateData inputs to accept plain objects, flatten OAuthMetadata at the api boundary.
- Update mongoMeili/aclEntry specs to the precise model types; drop redundant terser minification from data-provider's library build.

All data-schemas tests pass; api builds clean against the new output.

* 🔧 chore: Hash tsdown.config.mjs in data-schemas CI build-cache keys

The data-schemas build switched from rollup to tsdown, but the build-data-schemas / build-api cache keys in backend-review, config-review, and playwright-mock still hashed the (now-deleted) rollup.config.js. Hash tsdown.config.mjs instead so a config-only change invalidates the cached dist/api builds. (Found by Codex review.)

* 🔧 chore: Replace deprecated tsdown `external` with `deps.neverBundle`

tsdown 0.22 deprecated the top-level `external` option in favor of `deps.neverBundle`. Migrate the data-schemas config and set `deps.onlyBundle: false` to silence the (intentional) dotenv bundling hint. Build output and externalization are unchanged — dotenv bundled, all peers external.
2026-06-07 21:40:48 -04:00
Danny Avila
a1bfa3b298
🎭 test: Run Mock E2E Suite Through createRun With In-Process Fake Model (#13508)
* 🎭 test: Run Mock E2E Suite Through createRun With In-Process Fake Model

Replace the standalone HTTP mock LLM server with an in-process fake model
injected into the real createRun -> Run.create pipeline via
run.Graph.overrideTestModel, so the mock suite exercises the agents
integration end-to-end without a live provider or a separate server.

- Bump @librechat/agents to 3.2.2 for the FakeChatModel/createFakeStreamingLLM exports
- Add an env-gated applyTestRunHook seam in packages/api createRun (no /api changes)
- Add e2e/setup/fake-model.js to drive default replies + the skill-authoring tool-call flow
- Drop the mock-llm webServer from playwright.config.mock.ts and set LIBRECHAT_TEST_RUN_HOOK

* 🧹 test: Retire Standalone Mock LLM Server From E2E Recorder

Migrate the `--profile=mock` recorder onto the same in-process fake model
as the Playwright mock suite, then delete the now-unused HTTP mock server
so the fake-LLM logic lives in a single place.

- Point record.js mock profile at the fake model via LIBRECHAT_TEST_RUN_HOOK
- Remove the mock-llm-server spawn/wait and MOCK_LLM_PORT plumbing from record.js
- Delete e2e/setup/mock-llm-server.js (e2e/setup/fake-model.js is now the only source)
- Update e2e/README.md to describe the in-process fake LLM

* 🏷️ ci: Rename Playwright Mock E2E Check to Playwright E2E Tests
2026-06-04 08:33:28 -04:00
Danny Avila
d680763db3
🧭 ci: Use System Chrome for Mock E2E (#13481)
Some checks are pending
GitNexus Index / index (push) Waiting to run
GitNexus Index / post-index (push) Blocked by required conditions
Sync Locize Translations & Create Translation PR / Sync Translation Keys with Locize (push) Waiting to run
Sync Locize Translations & Create Translation PR / Create Translation PR on Version Published (push) Blocked by required conditions
Sync Helm Chart Tags / Ignore non-main push (push) Waiting to run
Sync Helm Chart Tags / Sync chart tags (push) Waiting to run
2026-06-02 22:07:10 -04:00
Teresa Blanco
b45e4aeae5
🎭 feat: Add Credential-Free Playwright Smoke Suite with a Local Mock LLM (#13472)
* 🧪 feat: add e2e playwright tests

* 🧪 feat: Add Playwright Recording Harness

* test: fix mock playwright config

* test: harden mock e2e environment

* test: preserve mock dotenv secrets

* test: harden mock isolation setup

* ci: cache mock e2e builds

* test: harden e2e cache and recorder checks

* test: preserve data-provider exports in oauth route test

* test: isolate mock auth logout state

* test: allow isolated logout smoke setup

* test: prepare logout smoke auth via api

* test: isolate oauth route module mock

---------

Co-authored-by: Danny Avila <danny@librechat.ai>
2026-06-02 16:36:39 -04:00