LibreChat/e2e
Marco Beretta 971e076015
🎛 refactor: Replace Agent Skills Toggles With a Three-Way Mode Control (#15520)
* refactor: replace agent skills toggles with a three-way mode control

The Skills section in the agent builder exposed one dependent choice as two
peer switches. "Use all skills" only meant anything while "Enable skills" was
on, and turning it on hid the skill list, so the user could not see what "all"
covered. Two Add affordances competed for the same action and both labels
leaned on a tooltip to explain themselves.

Replace both switches with a single segmented control over the SkillsScope
enum that already existed. Off, All, and Selected now write skills_enabled and
skills_scope outright, so the mode is the only source of truth for catalog
exposure and the allowlist survives a round trip through All.

Extract the section into SkillsSection. That drops the badgeText, showAdd,
showBody, and children props SelectedSection only carried to bend the tools
layout into a skills layout, and it retires skillsSelectionTransition, which
inferred the capability flags from the allowlist length and flipped skills off
whenever the last skill was removed.

All mode reports how many skills the deployment exposes and expands a
read-only list, so the catalog is never hidden. Selected mode keeps one Add
affordance: the dashed card while nothing is chosen, a header button once rows
exist.

Legacy agents keep loading through resolveAgentSkillsScope. The one ambiguous
persisted shape, skills_enabled with no explicit scope, is normalized on load
without dirtying the form so the next save records the mode explicitly.

Also give the shared Radio the roving tabindex and arrow key handling a
radiogroup owes its users; every segment was previously its own tab stop with
no way to change the selection from the keyboard.

* chore: add temporary PR comparison screenshots

* chore: drop temporary PR comparison screenshots

* fix: address PR review bot findings on the skills mode control

Codex (P1):
- Compare the resolved mode, not the raw persisted scope, before treating a
  segment click as a no-op. An agent left with `skills_enabled: false` and a
  retained `skills_scope` renders as Off, so matching on the raw field
  swallowed the click that re-enables it while the shared Radio moved its own
  selection, leaving the form clean and the disabled state persisted.
- Exempt explicit scopes from the legacy allowlist normalization in
  AgentSelect. Off deliberately keeps the allowlist so returning to Selected
  restores it, and flipping `skills_enabled` there persisted skills-enabled on
  an agent shown as Off, which skillDeps reads as permission to inject the
  skill-authoring tools.

Codex (P2):
- Report a failed catalog request instead of rendering it as an empty catalog.
  All mode showed "0 skills available" over an empty list when /api/skills
  failed; it now renders the picker's alert with a retry.
- Stop paging the whole catalog before the list is opened. Merely viewing an
  All-scoped agent walked every cursor at 100 records a page and retained each
  summary. Pagination now waits for the expansion, and the header reports the
  loaded count as a floor while more pages remain.
- Honor an explicit `skills_scope` in both skill cleanup paths. Deleting the
  last allowlisted skill, or pruning a dangling id on save, forced
  `skills_enabled` to false without consulting the scope, so an All-scoped
  agent that had once picked a single skill was silently switched off. The
  fail-closed inference now applies only to legacy agents that persist no
  scope at all.

Copilot:
- Skip the onChange when Home or End lands on the already-checked segment.
  Focus still follows the key; only a selection that does not move stops
  firing a form write.

* fix: address second round of PR review bot findings

Codex (P1):
- Normalize authoring left on under a resolved Off. An agent carrying
  `skills_enabled: false` with `skill_authoring_enabled: true` renders as Off,
  yet skillDeps enables the skill-authoring tools for either flag, so it ran
  with skills the section reported as disabled. Clicking the already-selected
  Off segment is a no-op, so nothing ever cleared it. The section now clears
  the flag on load without dirtying the form, matching what selecting Off
  already writes.

Codex (P2):
- Apply the scope-aware pruning in createAgent and revertAgentVersion too, not
  only updateAgent. Restoring an All-scoped version whose skills have since
  been deleted silently came back as Off, and createAgent had the same
  unconditional branch for ids that go stale before the write lands.
- Collapse the duplicate first-page skills request. ToolsSection listed skills
  through useListSkillsQuery while the section and the picker used
  useSkillsInfiniteQuery, so the same endpoint was fetched under two React
  Query keys and could not be deduplicated. All three consumers now share the
  infinite query, which leaves one cache entry instead of two.

Also sorts the imports the repo's checker flagged in SkillsSection,
ToolsSection and the section's spec.

* fix: clear the skills master flag under a resolved Off

`skills_enabled: true` with `skills_scope: none` is a shape the API accepts,
and the resolver reports it as Off. `skillDeps` reads the master flag on its
own as permission to expose the skill-authoring tools, so the agent ran with a
capability the section said was disabled, and because clicking the
already-selected Off segment is a no-op there was no way to clear it without
cycling through another mode.

The normalization that already cleared `skill_authoring_enabled` now clears
both flags whenever the resolved mode is Off, still without dirtying the form.
Neither write fires while a catalog mode is active.

* fix: disable skills for an explicit none scope in allowlist cleanup

The scope-aware fail-closed rule exempted every explicit scope, which spared
one shape it should not have: `skills_enabled: true` with `skills_scope: none`.
The API accepts that combination, the builder renders it as Off, and skillDeps
reads the master flag on its own as permission to expose the skill-authoring
tools. Cleanup runs server side without the builder mounted, so the frontend
normalization cannot repair it.

Only `all` and `selected` opt out now, since each already defines what an
empty allowlist means. The three self-heal paths in agent.ts share a
`requiresSkillsDisable` predicate, and the Mongo filter in skill.ts uses
`$nin: [all, selected]`, which still matches an absent field for legacy
agents.

* test: align the skills e2e spec with the explicit-mode contract

Three assertions still encoded the removed toggle behavior:

- The remove button is labelled `Remove {name}` in the new section, not
  `Remove from agent`, so both removal steps waited 180s for a button that no
  longer exists. That was the timeout, not a failed expectation.
- Emptying the allowlist no longer infers Off. The save writes `skills: []`
  and leaves the mode at Selected, which resolves to no skills at runtime, so
  the spec now asserts `skills_enabled: true` with `skills_scope: 'selected'`
  and keeps the existing proof that the picker and the run expose nothing.
- All leaves `skills` untouched, so a fresh agent serializes no allowlist at
  all rather than an empty one: `composeAgentUpdatePayload` always carries the
  four skill fields and `JSON.stringify` drops the undefined ones. Create and
  Off now assert `skills_scope` instead of an empty array.

Adds `skills_scope` to the spec's payload and detail types.

* fix: address third round of PR review bot findings

All three are P2 accuracy and cost issues in All mode:

- Count only the skills the runtime would inject. The list endpoint returns
  every skill the user can VIEW, while catalog resolution filters each through
  `resolveSkillActive`, so a shared skill left inactive was counted and listed
  as available even though the agent never receives it. The section now shares
  the `isActive` predicate the chat-side picker already filters on, so the
  count cannot promise more than the runtime delivers.
- Report loading instead of an empty catalog. With a cold query `data` is
  undefined and the count derived to "0 skills available", presenting an empty
  deployment while it was merely loading.
- Skip the allowlist lookups outside Selected. `useResolvedSkills` issues one
  `getSkill` per allowlisted id missing from the first catalog page, and since
  All now retains previous picks, merely viewing such an agent could fire
  hundreds of requests for rows that mode never renders. The fallback is gated
  on Selected, where the rows exist.

* fix: gate the All catalog on resolved skill states

`useSkillActiveState` substitutes an empty override map while its own request
is in flight and after it fails, so `isActive` reports every skill active in
both states. The section consumed the predicate but discarded the status, so a
deliberately deactivated owned or deployment skill was counted and listed as
available: briefly during load, and permanently after an error.

The count now waits for both requests, and a failure in either renders the
alert instead of a count the runtime would not honor. Retry refetches both,
which needed the hook to expose its `refetch`.

* fix: resolve the config behind defaultActiveOnShare before filtering

`useSkillActiveState` reads `defaultActiveOnShare` from the interface config
and falls back to `false` while that request is pending or after it fails, so
a deployment configured with `true` had its shared skills filtered out of the
All count until the config landed, and permanently on error. The section only
gated on the skills and skill-states queries.

Both dependencies feed `isActive`, so the hook now reports them as one
signal: `isLoading` and `isError` cover the override map and the config
alike, and `refetch` retries both. A consumer gating on the pair can no
longer miss a dependency, which is how the previous gap arose. The section is
the only consumer of those flags.

Also composes the retry from the shared `Button` (`outline`, `sm`), matching
the picker's own retry instead of a feature-local recipe.

* fix: report what the All catalog loaded, not a lower bound

`hasNextPage` tracks raw pagination while the rows are filtered to the
runtime-active skills, so the two disagree: 30 active skills followed only by
deactivated ones made the header read "30+ skills available" when 30 was the
true total, and it stayed wrong until the user expanded the list and finished
paging.

While pages remain the header now describes what has been loaded rather than
claiming a floor on what is available, so it cannot overstate the catalog. The
exact "N skills available" phrasing still lands once the catalog is complete.
`com_ui_skills_available_count_more` is replaced by singular and plural loaded
keys.
2026-09-06 03:32:56 +02:00
..
benchmarks ⚡ perf: Reduce Agent Chat Startup Latency (#14423) 2026-07-25 07:58:20 -04:00
benchmarks-mobile-chat 📱 test: Add Mobile Chat Performance Benchmark (#15510) 2026-09-02 10:36:33 -04:00
benchmarks-navigation 🔀 perf: Swap the Transcript With the URL on Conversation Switch (#15054) 2026-08-21 11:28:18 -04:00
benchmarks-reasoning 🔀 perf: Swap the Transcript With the URL on Conversation Switch (#15054) 2026-08-21 11:28:18 -04:00
bombadil 🤖 feat: make Event Actor HITL durable (#15305) 2026-08-28 08:05:11 -04:00
config 🪢 test: Prove MCP OAuth Resume End to End (#15485) 2026-09-01 18:38:39 -04:00
fixtures 🎬 test: Record-Once/Replay-Forever Model Fixtures for Mock E2E (#15210) 2026-08-25 18:29:55 -04:00
perf 📱 test: Add Mobile Chat Performance Benchmark (#15510) 2026-09-02 10:36:33 -04:00
recordings 🎭 feat: Add Credential-Free Playwright Smoke Suite with a Local Mock LLM (#13472) 2026-06-02 16:36:39 -04:00
setup 🛰️ feat: Show BYOM Worker Readiness (#15618) 2026-09-04 19:02:35 -04:00
specs 🎛 refactor: Replace Agent Skills Toggles With a Three-Way Mode Control (#15520) 2026-09-06 03:32:56 +02:00
config.local.example.ts
jestSetup.js ✨ v0.8.8-rc2 (#14845) 2026-09-02 22:18:48 -04:00
playwright.config.a11y.ts ⚖️ feat: Add Violation Scores (#8304) 2025-07-07 17:08:40 -04:00
playwright.config.benchmark.ts ⚡ perf: Reduce Agent Chat Startup Latency (#14423) 2026-07-25 07:58:20 -04:00
playwright.config.bombadil.ts 🧪 test(e2e): add Bombadil property exploration (#14462) 2026-08-10 02:11:06 +02:00
playwright.config.local.ts 🛟 test: Restore Playwright Smoke E2E (#13020) 2026-05-14 09:49:26 -04:00
playwright.config.mermaid.ts 🧜 feat: Open Mermaid Diagrams as Artifacts with SVG/PNG Export (#14713) 2026-08-09 09:02:42 -04:00
playwright.config.mobile-chat-perf.ts 📱 test: Add Mobile Chat Performance Benchmark (#15510) 2026-09-02 10:36:33 -04:00
playwright.config.mock.ts 🕹️ feat: Configure BYOM Tool Permissions (#15504) 2026-09-02 10:38:00 -04:00
playwright.config.navigation-perf.ts 🔀 perf: Swap the Transcript With the URL on Conversation Switch (#15054) 2026-08-21 11:28:18 -04:00
playwright.config.real.ts 🎯 feat: Tool Intent Label Capability (tool_intents) (#14499) 2026-07-29 15:40:52 -04:00
playwright.config.reasoning-perf.ts 🧪 test: Reasoning-Stream Render Perf Benchmark via react-scan (#14494) 2026-07-28 22:18:24 -04:00
playwright.config.redis.ts 🎬 test: Cover Detached Subagent Activity Lifecycle (#15117) 2026-08-23 01:16:25 -04:00
playwright.config.ts 🎭 feat: Add Credential-Free Playwright Smoke Suite with a Local Mock LLM (#13472) 2026-06-02 16:36:39 -04:00
README.md 🏎️ ci: Focus Redis E2E Coverage (#14842) 2026-08-14 12:54:30 -04:00
types.ts

LibreChat e2e

The mock e2e profile is the safest default for generated tests. It starts LibreChat with e2e/config/librechat.e2e.yaml, injects an in-process fake LLM (via LIBRECHAT_TEST_RUN_HOOK), creates an authenticated e2e user, and avoids real provider credentials.

Stream Stores and Shards

The mock profile uses the in-memory generation stream store by default. To exercise the same browser scenarios through a real Redis job store and pub/sub transport, start Redis on port 6379 and run:

npm run e2e:mock:redis

Memory mode explicitly disables Redis. Redis mode defaults to database 15 with a LibreChatE2E key prefix, and fails closed: the test server pings Redis and verifies that the generation job manager did not silently fall back to memory. Override REDIS_URI or E2E_REDIS_KEY_PREFIX when needed.

Pull request CI runs the complete mock suite in memory mode across three shards, plus a focused Redis transport suite. The Redis suite covers streaming fidelity, steering, interrupts, resumptions, HITL approvals, completion, thread folding, model icons, and usage:

npx playwright test --config=e2e/playwright.config.mock.ts --shard=1/3
npm run e2e:mock:redis:transport

The nightly schedule and manual workflow dispatch run the complete mock suite in both stream modes across two shards per mode. Every shard keeps one worker so tests do not contend for its authenticated user and database.

Property-based browser testing

Bombadil explores randomized sequences across the core chat loop, message branches, parallel multi-conversation responses, model changes, reloads, and sidebar conversation lifecycle operations:

npm run e2e:bombadil

Set BOMBADIL_TIME_LIMIT for longer local or scheduled runs. Failures leave a reproducible trace under e2e/.generated/bombadil-output; rerun it with:

BOMBADIL_REPRODUCE=e2e/.generated/bombadil-output npm run e2e:bombadil:run

Reproducing a real violation is expected to fail the Playwright test. Before a new run overwrites the active output, the harness archives it under e2e/.generated/bombadil-history/. Reproduction can diverge when streaming timing changes; Bombadil reports that explicitly.

The harness uses the credential-free mock-LLM profile, so exploration never sends billable provider requests.

CI runs the broad property exploration for five minutes in the non-blocking Bombadil Property Exploration workflow. If a property fails, download the bombadil-reproduction-* artifact into e2e/.generated/bombadil-output/, then reproduce it locally:

BOMBADIL_REPRODUCE=e2e/.generated/bombadil-output npm run e2e:bombadil:run

The accompanying bombadil-diagnostics-* artifact contains the captured CI log, Playwright HTML report, and Playwright test results. A Bombadil failure produces a workflow warning but does not block merge.

The default instruments inline JavaScript only because instrumenting LibreChat's full Vite bundle can exceed Bombadil's driver timeout during stateful runs. Set BOMBADIL_INSTRUMENT_JAVASCRIPT=files,inline for shorter coverage-guided experiments.

The branch reload, fork submission, model/conversation, HITL pause/resume, and mid-run steering lifecycle properties can be run independently:

npm run e2e:bombadil:branch-reload
npm run e2e:bombadil:fork-lifecycle
npm run e2e:bombadil:model-lifecycle
npm run e2e:bombadil:hitl
npm run e2e:bombadil:steering

These focused commands are diagnostic properties: they exit nonzero when they reproduce a product invariant violation. Reproduce a focused trace with its matching :run script and output directory, for example:

BOMBADIL_REPRODUCE=e2e/.generated/bombadil-output-hitl npm run e2e:bombadil:hitl:run

HITL drives a real ask_user_question checkpoint through the answer/resume controller, reloads while the question is paused, answers it once, and reloads the completed conversation. Steering submits an in-flight steer during a slow MCP-backed run, checks that it moves exactly once from the composer anchor into the response at the tool boundary, and reloads the applied state. The model lifecycle property is the passing control. The branch reload and fork properties preserve their minimal failing traces.

Recording Tests

Use Playwright codegen when you want to turn an exploratory browser session into a draft test:

npm run e2e:record

That command builds the app, starts the LibreChat test server (with an in-process fake LLM) when needed, writes e2e/storageState.json, and opens Playwright codegen at /c/new. The npm script uses http://localhost:3333 so it does not collide with a normal dev server on 3080. Raw recordings are written to e2e/recordings/ and ignored by git.

For a real local LibreChat config instead of the mock profile:

npm run e2e:record:local

Useful direct options:

node e2e/setup/record.js --url=http://localhost:3080/c/new
node e2e/setup/record.js --profile=local --no-output
node e2e/setup/record.js --auth-only
node e2e/setup/record.js --output=e2e/recordings/settings-draft.spec.ts

LLM-Assisted Loop

  1. Start npm run e2e:record.
  2. Let the LLM use Computer Use to operate the headed Playwright browser.
  3. Stop codegen after the workflow is captured.
  4. Move the useful parts from e2e/recordings/ into a committed spec under e2e/specs/mock/.
  5. Replace brittle generated selectors with role, label, text, or data-testid locators.
  6. Add assertions that prove the behavior, not just the clicked path.
  7. Run the finished spec with npm run e2e:mock -- <spec name>.

Generated recordings are a draft, not the final test. The committed version should use the shared helpers in e2e/specs/mock/helpers.ts where possible, wait on network or visible UI state instead of fixed sleeps, and keep test data deterministic.