Commit graph

1339 commits

Author SHA1 Message Date
Danny Avila
7347cfc195
🍡 feat: Batched User Questions With A Single Bounded Answer Form (#14737)
* feat: support batched user questions

* test: align batched question fixtures

* fix: harden batched question lifecycle

* test: submit batched HITL answers in e2e

* fix: address batched question review findings

* fix: preserve invoke return typing
2026-08-11 01:06:16 -04:00
Danny Avila
a3cec67e08
🪆 feat: Add Parent Activity Phase Summaries (#14721)
* feat: add activity phase summaries

* fix: preserve activity phase lifecycle semantics

* fix: satisfy activity phase type checks

* fix: simplify activity phase status mapping

* style: format activity phase changes

* fix: rebase activity phase bounds after shaping

* fix: link activity phase trace ancestry

* fix: reconcile activity phase bounds

* style: format activity phase reconciliation test

* style: align activity phase assertion

* fix: retain reasoning across commentary

* fix: preserve activity phase boundary state

* fix: detect renderable phase children

* test: type parallel phase assertion

* chore: bump agents SDK for activity phases

* fix: retain unphased lane reasoning

* fix: preserve tool group expansion across phases

* style: format phase expansion regression

* fix: preserve phase interaction state efficiently

* perf: skip sparse phase segment holes

* perf: partition phase segments with offsets

* fix: preserve phase boundaries and cursor state

* test: align activity phase regressions with CI

* test: keep phase context mock hoist-safe
2026-08-10 13:41:37 -04:00
Dustin Healy
87a8b9aa12
📡 feat: Route Web Search and Scrape Egress through the SSRF-safe Agent (#14606)
* feat(web-search): route outbound search and scrape requests through the SSRF-safe agent

Build the SSRF-safe agents at the web-search tool-assembly site and pass them into the
search tool config so outbound search and scrape connections are validated at connect
time against their resolved IP, on every hop including redirects, consistent with the
other outbound clients.

Add allowedAddresses to webSearchSchema, reusing allowedAddressesSchema, so self-hosters
can permit a deliberately-private search or scrape endpoint (for example a private
SearXNG instance). The field is resolved directly from the webSearch config at the
createSearchTool call site, not through loadWebSearchAuth, because it is config and not
an auth credential. webSearchSchema is flat (providers are chosen by enums, not by
counting keys), so the field is inert with respect to provider selection.

Document the field and its operator warning in librechat.example.yaml, and assert the
wiring in handleTools.test.js: the SSRF-safe agents are threaded into the search tool
config, allowedAddresses is passed through when set, and omitting it still threads the
agents with no exemptions.

TODO awaits @librechat/agents release with the httpAgent hook: this consumes optional
httpAgent/httpsAgent fields on the search-tool config that are not yet in a published
@librechat/agents. package.json is intentionally left at the current version; bump it to
the release that ships the hook before this lands. Validated locally against a revendored
@librechat/agents build, not a published release.

* 🛡️ fix: Apply allowedAddresses to the Web Search SSRF Preflight

The connect-time SSRF agent already honors webSearch.allowedAddresses, but
loadWebSearchAuth ran the isSSRFUrl preflight without it, so an admin-permitted
private search or scrape URL was stripped before the agent could ever use it.

Thread allowedAddresses and the URL's effective port through isSSRFTarget and
resolveHostnameSSRF so the exemption is consistent across both SSRF layers.

* 🛡️ fix: Validate Web Search Destinations and Defer to Configured Proxies

Handing agents to createSearchTool covered only the connect-time DNS lookup,
which Node skips for IP-literal hosts, and a configured proxy connects on our
behalf without running that check. A literal private target such as
http://169.254.169.254 could therefore reach the network.

Route every resolved web-search destination through the existing
applySSRFSafeAgentIfDirect contract so a blocked literal target throws before
any request is made, and withhold the agents when a proxy owns egress, since one
agent pair is shared by every provider and a direct-connect agent on a proxied
connection would break the request while asserting protection the proxy's
network context cannot provide.

* 🛡️ fix: Keep Web Search SSRF Agents Under a Proxy and Restore Pooling

Withholding the agents whenever a proxy was configured removed protection from
every direct and NO_PROXY destination in exchange for preventing a failure that
cannot occur: for an https target Axios substitutes its own CONNECT tunnel, so
the injected agent is never used for the proxy connection. Only a plaintext http
target keeps our agent and repoints it at the proxy, and only a proxy whose
hostname resolves private then trips the connect-time check.

Always pass the agents and exempt the proxy endpoint instead, deriving host:port
from the same PROXY, HTTP_PROXY, and HTTPS_PROXY resolution the rest of LibreChat
uses so the proxy hop stays reachable while destinations remain guarded. Axios
already applies NO_PROXY per request, so bypassed routes keep enforcement with no
extra logic.

Drop the load-time destination validation. It duplicated the isSSRFTarget
preflight for user-provided URLs, rejected admin values that were previously
legal, and threw from inside loadTools, where both loader wrappers swallow the
error and drop every tool for the turn rather than degrading web search alone.

Build the agents with keepAlive and cache them per exemption list. A bare
http.Agent does not pool, so the previous code replaced the pooled global agents
for every search, scrape, and rerank call and allocated a fresh pair per turn.

* 🛡️ fix: Reject IP-Literal Private Targets on Web Search Connections

Node resolves nothing for a literal host, so the connect-time lookup never saw
one: a destination or a redirect target given as http://169.254.169.254 reached
the network. Redirect hops pass through the same createConnection, so checking the
literal there covers both cases and removes the need for a maxRedirects control
that createSearchTool cannot accept.

Gate it behind blockLiteralHosts so only web search opts in. A caller that
reaches a proxy or a deliberate private service by literal address must exempt it
first, and the merged consumers of createSSRFSafeAgents have no such exemption, so
enabling this everywhere would break configurations that work today.

* 🛡️ fix: Keep IPv6 Brackets on Derived Proxy Exemptions

The exemption parser accepts an IPv6 entry only as [ipv6]:port, so stripping the
brackets produced fd00::1:3128, which carries three colons and is dropped as
malformed. An IPv6 proxy therefore stayed unexempted and the connect-time check
rejected it, failing every web-search request routed through it.

Use the URL hostname as parsed, which already carries the brackets.

* 🛡️ fix: Exempt Proxies Configured Through ALL_PROXY

Axios resolves a proxy through proxy-from-env, which falls back to all_proxy in
either case after <protocol>_proxy, so ALL_PROXY on its own is enough to route a
request through a proxy. Exemptions were derived from PROXY, HTTP_PROXY, and
HTTPS_PROXY only, leaving such a proxy unexempted and rejected with ESSRF.

Derive the exemptions from the full set of variables that can put a proxy in
front of these requests instead. The installed proxy-from-env 2.1.0 reads no
npm_config variables, so those are deliberately not included.

* 🛡️ fix: Drop the Unearned PROXY Exemption and Harden the Web Search Guard

Nothing on this path consumes PROXY: Axios resolves proxies through
proxy-from-env, which reads only <protocol>_proxy and all_proxy, and web search
never calls applyAxiosProxyConfig. Exempting it therefore granted a bypass rather
than preserving a working route, and a user-settable search URL that redirects to
that address reached it and returned the body. Remove PROXY and proxy, and skip a
socks endpoint for the same reason, since Axios cannot proxy through one.

Tolerate a non-array allowedAddresses instead of spreading it, which threw out of
loadTools and dropped every tool for the turn. The YAML path is schema-validated
but the admin override path merges without parsing, so the value is reachable.

Separate cache keys with NUL rather than a newline, so an entry containing a
newline cannot collide with two separate entries, and bound the cache. Give the
agents the idle timeout the global agents carry, which keepAlive alone did not
restore. Reject a unix socket, which carries no host to validate. Also treat
fec0::/10 site-local as private, matching the fe80::/10 handling beside it.

Exercise the real resolver in handleTools.test.js rather than mocking it, so the
wiring test now fails if the agents it threads do not actually block a private
target.

* 🛡️ fix: Derive Proxy Exemptions Through Axios's Own Resolver

Unioning every populated proxy variable exempted addresses that never carry a
request. proxy-from-env picks a protocol-specific variable before all_proxy and
lowercase before uppercase, so an ignored value became a trusted host:port that a
redirect onto a direct route could reach. It also normalizes a scheme-less value
such as proxy.internal:3128 to an http URL, where parsing the raw string yielded
an empty hostname and no exemption at all, breaking the proxy hop.

Resolve through getProxyForUrl, the entry point Axios itself calls, so precedence,
scheme normalization, and NO_PROXY match exactly and cannot drift. NO_PROXY
covering everything now yields no exemption, since nothing is proxied. Declared
locally rather than adding a types package, alongside the existing declaration in
the same directory.

Also revert the fec0::/10 site-local change. domain.spec asserts that boundary
deliberately to prove the fe80::/10 mask does not over-reach, and the shared
address schema still classifies fec0 as public, so a runtime block there would
leave operators unable to configure the exemption. It belongs with those two
together, not in this PR.

* 🛡️ fix: Resolve Proxy Exemptions Against the Real Destinations

Resolving against placeholder probe hosts applied destination-specific NO_PROXY
rules to a host nobody dials. With NO_PROXY matching the probe domain but not a
real provider, no exemption was derived even though Axios still proxied the actual
request, so the agent rejected the private proxy hop with ESSRF.

Resolve per configured destination instead, passing the values loadWebSearchAuth
already resolved. Only plaintext http destinations are considered, since for an
https destination Axios substitutes its own CONNECT tunnel and never uses the
injected agent for the proxy connection, which is also why provider defaults need
no exemption: every one of them is https.

* 🛡️ fix: Accept Embedded-IPv4 IPv6 Forms in the Address Exemption Schema

The runtime guard blocks 6to4, NAT64, and Teredo addresses whose embedded IPv4 is
private, but the schema's local copy recognized only ULA, link-local, and the
dotted IPv4-mapped form, so an entry such as [64:ff9b::a00:1]:8080 was dropped as
a public literal. An operator reaching a private endpoint that way could not
configure the exemption at all.

Mirror hasPrivateEmbeddedIPv4 in the schema helper, which the surrounding comment
already asks to keep in sync. Public embedded addresses stay rejected, since an
exemption there has no defensive purpose.
2026-08-10 10:33:54 -04:00
Marco Beretta
152dcf4721
🔗 feat: Shared Conversation Badge and Stable Share Links (#14712)
* feat: improve shared conversation links

* test: Cover Shared Link Lifecycle

* test: Cover Shared File Snapshots

* fix: address review findings on shared links

Stop double-decoding the conversation search term. Express already decodes
req.query, so the route's extra decodeURIComponent threw URIError on any term
containing a bare percent sign and mangled percent-escape-looking text. The
sidebar already sent the term raw, so this failed there too.

Advance a share's stored target to its branch tail when an update omits one.
Updating from the conversation list could not resolve the tail and reused the
stored target verbatim, silently republishing the same snapshot instead of the
turns added since.

Require revalidation on shared files. Updates now keep the shareId, so the file
URL no longer changes and a cached response could outlive a revoked share-files
choice; an ETag over the pinned snapshot fields keeps unchanged files on 304.

* fix: keep the shared badge across conversation cache replacements

isShared is derived per list request and absent from single-conversation
payloads, so rename, pin, and the SSE conversation updates dropped it when they
swapped a server response into the sidebar cache, hiding the badge until an
unrelated list refetch. Carry the cached value forward in updateConvoInAllQueries
so every replacing caller is covered, while an explicit value still wins.

* test: mock syncStaticTools in server boot specs

initializeMCPs now calls syncStaticTools when no MCP servers are configured, but the server boot specs stub ~/server/services/Config without it. Post-listen initialization threw, hit process.exit(1), and took the jest worker down until it exceeded the retry limit.

* fix: address codex findings on the shared DataTable and file ETag

Keep the published DataTable export bound to the legacy component and ship the design-system table as VirtualizedDataTable, so external consumers of @librechat/client keep the props they compile against.

Fold the snapshot's stored location into the shared-file ETag, so a re-published output that keeps its size and revision but moves its object no longer revalidates to a stale 304.

Auto-fill the table when a first page is too short to overflow its container, since pagination is otherwise only reachable through the scroll handler.

* fix: re-scope share grants before publishing and retry stalled auto-fill

Move the shared-link ACL expiration write ahead of the content update. The shareId survives an update, so a failed ACL write after the write-through left the new messages and file snapshot readable at the same URL while the owner saw a 500.

Retry a rejected auto-fill fetch up to three times: an unscrollable table has no scroll event to fall back on, and the sentinel alone would strand it on the first page.

* fix: follow regenerated branches and pin forks to the payload they saw

advanceTargetToBranchTail only walked descendants, so a target replaced by a regeneration (a sibling, not a child) left the update parked on the obsolete branch and published none of the turns that followed. A childless target now hops once to the newest sibling the conversation continued under.

A shareId survives an update, so an owner republishing between a viewer's load and their Continue click would resolve targetMessageIndex against different messages. The fork request now carries the payload's updatedAt and is rejected with 409 when it no longer matches; the viewer gets the current version pulled in and can retry.

* fix: keep table sorting and legacy backfills from breaking share flows

Restore the union formatting a local lint-staged prettier collapsed in data-provider types, which broke the CI lint run.

Header clicks now toggle direction instead of cycling through an unsorted state, which the controlled tables translated straight back into the default and made one direction unreachable.

Re-arm the auto-fill guard on a sort change: a re-sorted first page arrives with the same row count, and the guard would otherwise suppress paging on a container that still cannot scroll.

Lazy fileSnapshots backfills no longer touch updatedAt. That timestamp is the revision a viewer's fork is validated against, so a legacy share's first read would have made the Continue click that followed it fail with a 409.

* fix: break pagination ties by id and reset share state per conversation

Both list cursors marked a page boundary with values that repeat: conversations by (sort field, updatedAt) and shared links by the sort field alone. Imported chats share a title and a timestamp, so every row tied with the boundary was skipped. Both now carry the boundary row's _id and sort by it last, and the shared-links cursor is an opaque composite the route still validates before querying.

The share dialog outlives a switch between conversations, so a link with files disabled left the next conversation's dialog showing the switch off and quietly published without files. The stored choice now falls back to the enabled default, and a stale link no longer sits in the copy field.

* fix: keep titleless shared links in the paginated list

A share has no title default, and BSON orders a missing title before every string, so encoding the boundary as an empty string skipped the remaining untitled links when sorting Name ascending and re-admitted all of them descending.

The cursor now carries the boundary's null rather than flattening it, and because $lt and $gt are type-bracketed against a string, descending adds an explicit clause for the untitled tail that a string comparison can never reach.

* style: sort share method imports

* fix: fail closed on orphaned share targets and guard snapshot backfills

getMessagesUpToTarget walked levels from the roots and returned everything it had accumulated when the target was never reached. An imported or partially deleted branch whose parent is missing therefore published the whole conversation instead of the selected branch; the walk now returns nothing unless it actually reaches the target.

A lazy backfill wrote fileSnapshots unconditionally, so a viewer's first read of a legacy link could land after a republish and restore the snapshot it replaced, re-authorizing the stable URL of a file the owner had just removed. The write is now conditional on the link still having no snapshot, and the stored one wins any race.

Regenerating a message above the shared tail leaves the whole stored branch childless, so the target walk now climbs to the closest ancestor the conversation continued under instead of stopping at the stored tail's own siblings.

Changing the search or sort also returns the table's viewport to the top, since the query holds the previous rows while it refetches.

* fix: page through titleless rows on both sides of the cursor

The route validator still required a string primary, so the composite cursor the data layer issues for a titleless boundary came back as a 400 and the shared-links table stopped at that page.

Conversations had the same type-bracketing gap the shared links just closed: a name-sorted page could not reach conversations with no title, since a comparison against a string never matches a missing field. The cursor now carries the null and the filter spells out the titleless clauses for both directions.

* fix: keep the share badge read-only and refresh rows on cell changes

ensureLinkPermissions re-granted the owner ACL entry on every call, so rendering the header's shared badge turned ordinary navigation into a permission write. It now checks for the grant first and only migrates a link that still lacks one.

A fork carrying a positional target but no revision falls back to the whole share, since nothing proves which payload the index was counted against.

The memoized table row compared row data and selection only, so a cell rendering external state (the archived list's pending Restore, for one) kept its stale rendering until the row object itself moved; rows now also compare a marker that moves with the column definitions.

* fix: keep the shared badge honest when a delete fails or a link remains

A failed delete left the conversation looking unshared: the optimistic snapshot covered only the shared-link queries, not the conversation caches the badge reads. The cleared conversations are now restored with the rest.

A conversation can hold one link per target message, so clearing the badge on delete is a guess. The conversation list is invalidated once the mutation settles, letting the server decide from the links that are actually left.

* fix: refetch every cached conversation page after deleting a link

The invalidation was pinned to page zero, so a conversation cached further down the sidebar kept the badge the optimistic update had already cleared even when another targeted link survived.

* fix: treat a failed page fetch as a failed auto-fill

React Query resolves fetchNextPage with an error result instead of rejecting, so the rejection handler never ran: the guard stayed armed on the unchanged row count and an unscrollable table could never reach the next page.

* refactor: move the share request helpers into the typed backend

Cursor validation, page-size clamping and the shared-file cache validator were plain backend logic sitting in the legacy JS route. They now live in packages/api with their own tests, and the route keeps only the Express-side wiring: reading query params, mapping domain error codes to status codes, and writing the response.

Also carries the requested file choice into the cache entry the create and update mutations synthesize, since the response never echoes it and the dialog reads a resolved entry with no choice as the enabled default.

* fix: hold auto-fill while the replacement page is in flight

A search or sort swap keeps the previous rows and hasNextPage on screen while the new first page loads, so the re-armed auto-fill asked for page two against a query that was still fetching its first. An infinite query runs one fetch at a time, so that request could cancel or interfere with the one already out. Both tables now pass their fetching state and the guard waits for it.

* fix: stop advertising links a deployment no longer serves

The sidebar badge rendered from the derived flag alone, so a deployment that turned shared links off still told owners a link was live while the public routes were unregistered.

The share dialog called the link a snapshot, but the payload populates the referenced message documents on every request: an edit to an already-shared message is visible immediately, and Update only adds newly referenced ones. The copy now says that.

Scroll pagination inspects a resolved error the way auto-fill already does, since React Query reports a failed page that way instead of rejecting.

* a11y: gate the shared conversation label on the feature flag

The icon stopped rendering when a deployment turns shared links off, but the row still announced the conversation as shared to screen readers. Both now read the same condition.

* fix: accept long title cursors and stop badge work the feature disables

The cursor cap was tight enough that a Name-sorted page ending on a long title produced a nextCursor the next request rejected, stranding the rest of the list. It now sits well clear of anything the server can issue.

The conversation list skipped straight into the shared-link lookup even where ALLOW_SHARED_LINKS is off, paying a round trip on the sidebar's first page for a badge that is never rendered.

A regeneration is newer than what it replaced, so only a newer sibling counts: an older one that still has follow-ups is the branch the target was regenerated away from, and resuming there published turns the target had excluded.

* fix: hold scroll pagination while a replacement page loads

Resetting the viewport to the top after a search or sort change fires a scroll event, and the retained previous rows still report another page, so the handler asked for page two of a query that was still loading page one.

* fix: keep the legacy share migration ahead of the owner-grant shortcut

A legacy row keeps its marker until every grant it needs exists, so an owner grant on its own is not proof the migration finished. Reading the marker first means a half-migrated public link still gets its public grant, while a fully migrated one keeps the read-only settled path the badge lookup depends on.
2026-08-09 08:14:54 -04:00
Danny Avila
5c939d129b
🔌 feat: Add Agent Plugins (Experimental) (#14704)
* 🔌 feat: Add Agent Plugins v1.0.0 Support

Implements the Agent Plugins 1.0.0 specification so LibreChat can load
portable plugin packages: a `plugin.json` manifest, `skills/` holding Agent
Skills, `mcp.json` describing MCP servers, and reverse-domain extension
directories.

- Validate the closed `plugin.json` schema, selecting rules from `$schema`
  without retrieving it. Unknown top-level fields and a non-object
  `extensions` field are reported and ignored; every other violation rejects
  the plugin.
- Enforce plugin-root containment through realpath, including for paths whose
  leaf does not exist, and apply the narrowest failure boundary per component.
- Map `mcp.json` onto LibreChat MCP options across stdio, Streamable HTTP, and
  legacy HTTP+SSE, bypassing the config loader's `${VAR}` process-env
  expansion so plugin values never resolve against the server environment.
- Expand only `${PLUGIN_ROOT}` and `${PLUGIN_DATA}`, once and non-recursively,
  in `args`, `env` values, and `cwd`; supply both variables to the subprocess
  after configured `env`, and reject entries that declare them.
- Discover skills from the immediate children of `skills/` only, reusing the
  deployment skill loader so plugin skills are ordinary deployment skills with
  a distinct id namespace.
- Read LibreChat's `ai.librechat` extension directory and hand
  `hooks/hooks.json` to the Claude hook compatibility layer.
- Load operator-installed plugins from `DEPLOYMENT_PLUGINS_DIR` at startup,
  merging their skills into the deployment skill registry and their MCP
  servers into the app config. Plugins never displace a configured server or
  deployment skill.
- Add `cwd` to the stdio MCP transport, which the specification requires and
  LibreChat did not previously support.

Component failures stay isolated: a malformed `mcp.json`, an invalid skill, or
a bad hooks document never prevents the rest of a plugin from loading.

* 🔒 fix: Contain Agent Plugins config at the runtime boundary

Review of #14704 surfaced that every real finding sat where the loader's
output crosses into LibreChat's existing runtime, not in the specification
logic. The loader deliberately left plugin placeholders literal, but
downstream layers re-processed the same fields and undid it.

- Mark plugin MCP configuration with `source: 'plugin'` and return it verbatim
  from `processMCPEnv`. Without this a remote plugin could declare
  `Authorization: Bearer ${OPENAI_API_KEY}` and receive host credentials at its
  own origin. The gate reads the configuration rather than a caller-supplied
  flag, so no future call site can reintroduce the leak by omitting it.
- Skip `preProcessGraphTokens` for plugin configuration as well; it resolves
  placeholders into headers, url, and args on the same path.
- Reject plugin server names that change under `normalizeServerName`. Tool keys
  embed the normalized name while request-time resolution uses the raw name, so
  an unstable name published tools that nothing could resolve.
- Reject `__proto__`, `constructor`, and `prototype` as server names, and merge
  plugin servers with `Object.defineProperty` and an own-property conflict
  check, so a package cannot reach a prototype setter or collide with an
  inherited member.
- Enforce manifest-name uniqueness before components are accepted; two packages
  sharing a name would share one `PLUGIN_DATA` directory.
- Isolate a failed data-directory creation to the single plugin instead of
  rejecting the whole scan.
- Prefix rejected-plugin diagnostics with the directory, which is the only
  identifier a package without a valid manifest has.
- Type extension namespace contents as JSON rather than `unknown`, and correct
  the header field-value comment to name obs-text.

Verified end to end from the built package: a plugin declaring an environment
placeholder in a header reaches the transport with the placeholder intact while
operator-authored configuration still resolves normally.

* 🔇 fix: Report Agent Plugin hooks that will not run

The loader reads `ai.librechat/hooks/hooks.json`, but nothing registers the
resulting plan, and startup supplies no hook capabilities. A package declaring
hooks was therefore accepted in silence, leaving an operator to believe the
hooks ran.

Detect the document when no capabilities are registered and report it as
unsupported, so the limitation is visible in startup diagnostics rather than
inferred from behavior that never happens.

* 🧯 test: Restore MCP startup test mocks

Carries the two mock additions from #14711 so this branch can prove itself
green. `initializeMCPs` now calls `syncStaticTools`, which the server startup
specs do not stub, so they fail on every branch that has not picked this up.
Drops out of the rebase once #14711 lands.

Co-authored-by: Danny Avila <danny@librechat.ai>
2026-08-09 08:10:22 -04:00
Danny Avila
ef38f362ec
📦 chore: bump @librechat/agents to v3.4.2 and npm audit (#14702)
* 📦 chore: bump `@librechat/agents` to version 3.4.2

* 📦 chore: bump `mermaid` to version 11.16.1 and update related dependencies

* 📦 chore: bump `js-yaml` to version 4.3.1 in package-lock and data-provider

* 📦 chore: bump `nanoid` to version 3.3.18 in package.json and package-lock.json across multiple packages

* 🔧 fix: Remove stray `api/tsconfig.json` breaking e2e `~` alias

An empty `api/tsconfig.json` was accidentally committed with the agents bump.
Playwright's require hook resolves path aliases from the nearest path-config,
checking `tsconfig.json` before `jsconfig.json` in each folder, so the empty
file shadowed `api/jsconfig.json` — the only place `"~/*": ["./*"]` is defined.

Every e2e spec that calls `cleanupUser` then failed on
`Cannot find module '~/cache/getLogStores'` from `api/models/index.js`.

- delete the stray file and gitignore it so tooling can't re-commit it
- register `module-alias` in `cleanupUser` so backend requires resolve
  regardless of which path-config Playwright happens to find

* 📦 chore: bump `@librechat/agents` to version 3.4.3 in package.json and package-lock.json
2026-08-08 12:24:06 -04:00
Marco Beretta
39f5f9d846
⚡ perf: Agent List and Model Selector at Scale (#14601)
* perf: cut serial round trips from the agent list query path

The agent list was the slowest path on first page load. Three separate
problems compounded:

- `getListAgentsHandler` chained its reads: two ACL lookups, the avatar
  refresh cache probe and the viewer skill scope all resolved serially
  ahead of the list query, and `attachOwnerContacts` added two more hops
  after it. The four independent reads now resolve together, and the
  avatar refresh runs alongside the list query instead of before it -
  refreshed paths reach the response through `urlCache`, not through
  whatever the list query happened to read. Serial hops per request drop
  from 7 to 4 on a warm cache.

- The avatar refresh loaded the user's whole accessible agent set (up to
  MAX_AVATAR_REFRESH_AGENTS) to discover which entries were S3-backed.
  Scoping the query to `avatar.source` means deployments on any other
  file strategy match nothing instead of walking the full set.

- `fetchAllAgentPages` walked cursor pages at the server's default size
  of 100, and callers consume the flattened result, so every extra page
  was a serial round trip for no benefit. It now requests the server
  maximum. Measured over a 2,860 agent account: 29 requests / 1.65s
  before, 3 requests / 0.29s after.

Also parallelizes the conversation file reads in `initializeAgent`. The
convo file refs and the execute_code thread walk share no inputs, and the
two code-file lookups depend only on `threadFileIds`, so the chain of six
serial reads on every turn collapses to two. This one is time to first
token the user waits through.

* perf: virtualize the model selector agent list

Opening the agents submenu with a large agent set froze the tab and could
kill it outright. With ~10k accessible agents the submenu blocked for over
15 seconds and took the heap from 96MB to 911MB. Four per-row costs were
being multiplied by the full list, which rendered unwindowed:

- `useIsActiveItem` allocated a MutationObserver per row (10,016 of them
  for one dropdown). Replaced with an Ariakit store subscription, which
  needs no observer at all and returns a boolean so a row only re-renders
  when its own active state flips.

- `useFavorites` ran per row, opening a jotai subscription, a query
  subscription and a mutation each time. Hoisted to one call per endpoint.

- Each row rescanned `endpoint.models` to recover `isGlobal`, a field the
  parent had already discarded from the array it was mapping. The parent
  now passes it down from a lookup map.

- The list itself is now windowed above 100 rows. Ariakit's composite only
  knows about mounted rows, so arrow-keying to the window edge previously
  found no next item and let focus escape the nested menu, closing it;
  `handleBoundaryNavigation` scrolls the next index in, waits for it to
  mount, then moves the composite onto it. Navigation inside the window is
  left to Ariakit.

Open drops from >15s to 96ms, mounted rows from 10,028 to ~18, DOM nodes
from 123,346 to ~1,000, and the heap no longer grows. Verified in browser:
arrow keys track 1:1 to index 238 and back, and click selection works.

* perf: serve the model selector from the shared VIEW agent query

The model selector asked for EDIT-scoped agents whenever the marketplace
is enabled, while `useAgentsMap` and `useMentions` asked for VIEW. Since
the cache key includes the params, that was two distinct entries, so first
page load ran the paginated walk twice and held two copies of the whole
agent list in memory. Measured against a 10k agent account: 22 list handler
invocations per page load, now 11.

Collapsing the two by asking for the same permission everywhere would have
changed what the selector shows - under the marketplace the EDIT scope is
what makes it "My Agents", with discovery handled by the marketplace entry.
So the list endpoint now marks each row with `isEditable`, resolved from an
ACL read folded into the existing parallel batch (no extra serial hop), and
the selector filters the shared VIEW response instead of refetching. A
VIEW-scoped list for a user with 2861 visible / 361 editable agents returns
exactly 360 rows flagged editable, matching what the EDIT query returned.

`AgentSelect` deliberately keeps its own EDIT query: it reads `skills` and
`skills_enabled`, which `sanitizeViewerSkillScope` strips from VIEW-scoped
responses. It also only mounts when the builder panel is open, so it is not
part of the first-load cost.

The field is set unconditionally rather than omitted when false so that a
client talking to an older server sees `undefined`, keeps every agent, and
degrades to showing too many rather than none.

* fix: address review findings on the agent list at scale

Three issues from review, all confirmed against the code before fixing.

Avatar refresh no longer runs alongside the list query. `updateAgent` writes
through `findOneAndUpdate` on a `timestamps: true` schema, so refreshing an
avatar advances `updatedAt` — the field `getListAgentsByAccess` sorts and
cursors on. A write landing after the first page's snapshot moved that agent
ahead of the returned cursor, dropping it from every later page and silently
truncating the caller's flattened list. This was a regression introduced when
the two were parallelized; serializing them costs nothing on the common path,
because a cache hit returns without issuing any query, so only the
once-per-30-minutes miss pays for the ordering. The new test asserts the write
lands before the list snapshot and fails against the parallel version.

The virtualized list no longer inserts a focusable grid into the combobox.
`List` spreads its props onto `Grid`, whose defaults are `role="grid"`,
`containerRole="row"` and `tabIndex={0}`; inside Ariakit's listbox that added a
tab stop ahead of any row and put grid/row semantics between the listbox and its
options. All three are now neutralized so focus and ARIA stay with the combobox
items.

The list also resets to the top when the filter changes. `Grid` keeps its scroll
offset across prop changes and clamps an out-of-range offset to
`totalRowsHeight - height`, the end of the shorter list. Scrolling deep and then
searching landed on the tail: measured at row 626 of 667 matches, with only
those rows mounted and reachable by keyboard. Keying the list on the search
value restores row 0.

* fix: declare option position and set size for the virtualized model list

Once the model list is windowed, only the mounted slice exists in the listbox,
so a screen reader infers position and total from ~19 elements instead of the
real set — announcing "3 of 19" partway through 10,014 agents.

Model rows now carry aria-posinset and aria-setsize. The marketplace entry and
any model specs share the same numbering, because they are options in the same
listbox: declaring the values on some options while leaving others to be
inferred from the DOM would make the set internally inconsistent. Both are
omitted entirely when the list is short enough to render unwindowed, where the
DOM holds every option and the implicit values are already correct.

Verified against a 10,014 agent account: the marketplace entry reports 1 of
10015, the first models 2 and 3, and after scrolling to row 4999 the leading
mounted model reports 5001 of 10015 with 19 options in the DOM.

* 🩹 fix: Address Follow-Ups on the Agent List at Scale

Corrects residual issues in the agent-list perf work, all inside its own scope.

- Forward `idOnTheSource` through `PermissionService.findAccessibleResources`
  so `getUserPrincipals` skips the user-document read. The list handler resolves
  three permission sets per request and each was paying its own `User.findById`;
  the auth strategies already normalize the field to a value or null.
- Gate the editable-set lookup on its own predicate instead of borrowing
  `canReturnSkillConfig`. The two answer unrelated questions and only coincide
  today, so redefining the skill flag would have marked every agent editable.
- Log mapping failures in the list response instead of swallowing them.
- Apply the walk page size after the caller's params in `fetchAllAgentPages`.
  A caller limit only changed page size, never what the flattened walk returned,
  so `defaultAgentParams`' `limit: 10` would have turned one request into 301.
- Carry `isEditable` on the agent rows the create and update mutations write
  into the list cache. Mutation responses omit the field, so those rows lost it.
- Document `isEditable` as list-only, ACL-derived, and fail-open on absence.
- Restore the truthiness guard on the thread walk in `initializeAgent`. Widening
  it to `!= null` made an empty `parentMessageId` issue a full-conversation read
  against an anchor that can never match.
- Await `getConvoFiles` directly rather than calling `.then()` on it, restoring
  tolerance for synchronous test doubles.
- Correct the avatar-refresh comment: the projection was never full documents,
  and the real reason to filter is that an unfiltered budget is self-reinforcing.

Tests: both new `initialize` tests and both new backend tests are
mutation-verified; the concurrency test fails under either serialization order.

* fix: preserve ACL isEditable when merging agent mutation responses

Mutation responses omit list-only isEditable. Inferring true from write
success promoted VIEW-only rows into the editable subset for MANAGE_AGENTS
callers who can PATCH agents their ACL marks non-editable.

* fix: sort imports in agent mutations test

ESLint import-order check failed on the isEditable cache-preservation test.

* 🧷 fix: Carry isEditable Onto Duplicated Agent List Rows

`useDuplicateAgentMutation` prepended the raw duplicate response to the cached
list, and mutation responses omit the list-only `isEditable` field. The row
survived the "My Agents" filter only by failing open on `undefined`, so it would
disappear the moment a consumer read the flag strictly.

Duplicating grants the caller ownership, so the new row is editable outright;
this is the create case rather than the merge case `mergeAgentListRow` handles.
Last cache write on this path that did not carry the field.

---------

Co-authored-by: Danny Avila <danny@librechat.ai>
2026-08-07 21:04:55 -04:00
Danny Avila
5ff46d8c67
🛟 fix: Stop Agents When Code Resources Cannot Recover (#14651)
* fix: Block Agents When Code Resources Cannot Recover

* fix: Preserve Resource Recovery Failures Across Agent Paths

* fix: Centralize Fatal Agent Initialization

* chore: sort agent imports
2026-08-07 00:33:28 -04:00
Dustin Healy
c6bb77325a
🔗 feat: Admin Panel Link in Settings for Admins (#14662)
Expose ADMIN_PANEL_URL through the startup config for users holding the
access:admin capability, and render an Admin section in Settings > General
with an external link to the admin panel. The URL is omitted server-side
for unauthenticated requests and users without admin access.
2026-08-06 12:40:44 -04:00
Danny Avila
33c8801ba6
🌊 feat: Wire Adaptive Stream Smoothing Across Google, Bedrock, and Zero-Disable Semantics (#14660)
* 🌊 feat: Wire Adaptive Stream Smoothing Across Google, Bedrock, and Zero-Disable Semantics

With @librechat/agents 3.4.0 smoothing defaults ON (25ms adaptive) for
every provider; this completes the LibreChat side:

- google: fix the unguarded endpoints.all clobber and wire streamRate
  into llmConfig._lc_stream_delay — previously read from config and
  silently dropped, making google streamRate a no-op end to end
- bedrock: wire streamRate (endpoint + endpoints.all) — previously
  absent entirely, so bedrock smoothing was unreachable from config
- anthropic/openai: nullish guards so streamRate: 0 survives as the
  explicit smoothing disable; drop the azure 30/17 hardcoded fallback
  the SDK default now supersedes
- delete dead createHandleLLMNewToken (no call sites since #6886, and
  LangChain backgrounds callbacks so a sleep there never paced anything)
- schema: streamRate gains .min(0) and docs; example docs updated,
  including pairing STREAM_DELTA_COALESCE_MS with the smoothing tick

* 🩹 fix: Address Review — Keep Published Shim, Typed Delay Access, Scoped Docs

- restore createHandleLLMNewToken as a @deprecated compatibility shim:
  it ships in the public @librechat/api root, so removal is reserved for
  a major release
- assign llmConfig._lc_stream_delay via the SDK's typed property (3.4.0
  StreamSmoothingOptions) instead of Record<string, unknown> casts at
  all five sites
- scope the 25ms-default wording to agents SDK-backed providers (legacy
  Assistants/Ollama still per-chunk sleep at DEFAULT_STREAM_RATE=1) and
  clarify coalescing guidance when streamRate: 0
2026-08-06 12:39:12 -04:00
Dustin Healy
0e14d91ed9
⏱️ fix: Compile admin file-config MIME patterns on a linear-time engine (ReDoS) (#14555)
* ⏱️ fix: Compile admin file-config MIME patterns on a linear-time engine

convertStringsToRegex compiled admin-configured supportedMimeTypes with the native RegExp engine, and checkType runs those patterns against an uploaded file's Content-Type on the server event loop, so a catastrophic-backtracking pattern in fileConfig could ReDoS the whole process on upload.

The MIME-pattern compiler is now swappable. It defaults to native RegExp, which browser builds keep so no engine is added to the client bundle, and the server injects a linear-time engine (RE2JS) at startup. Only test is ever called on these matchers, so the shared type widens to a structural RegexLike with no behavior change for valid patterns. The browser stays on native because a client-side stall would only affect that one tab.

* ⏱️ fix: Wire the linear MIME compiler in the experimental entry point

api/server/experimental.js mounts the same upload routes and calls mergeFileConfig but never set the linear-time compiler, so admin MIME patterns still compiled with native RegExp there. Mirror the setup, and widen the client-side supportedMimeTypes type to the shared RegexLike so the browser typechecks against the same structural matcher.

* 🧹 refactor: Configure the file-config linear engine from a shared helper

Move the RE2 wiring out of both JS server entry points into a single
configureFileConfigRegexEngine helper exported from @librechat/api, so /api stays a thin
caller and the setup no longer has to be kept in sync across index.js and experimental.js.

Also warn loudly when compiling an endpoint's supportedMimeTypes drops every pattern (an
empty allowlist would reject all uploads), and correct the isMimeTypeSupported docstring to
say RegexLike rather than RegExp.

* fix: fail closed when every MIME pattern fails to compile

convertStringsToRegex returned [] when all configured patterns failed to
compile, and filter.ts reads an empty allowlist as no restriction, so a
restrictive config whose patterns all fail allowed every attachment.
Return a single reject-all matcher instead so every consumer fails closed.
2026-08-06 09:05:42 -04:00
Danny Avila
56175af0b5
🎟️ fix: Reconcile MCP OAuth Readiness Across Pods (#14629)
* fix: stabilize MCP OAuth readiness across pods

* fix: harden MCP readiness review findings

* fix: resolve CI type check and terminal OAuth polling

* fix: address MCP OAuth readiness review

* fix: align MCP OAuth readiness state

* test: stabilize MCP OAuth readiness assertion

* fix: reject stale MCP OAuth callbacks

* fix: close distributed MCP OAuth readiness gaps

* style: sort Redis MCP test imports

* fix: preserve MCP OAuth polling across rolling pods

* fix: finalize distributed MCP OAuth readiness

* fix: preserve runtime-detected MCP OAuth

* fix: report runtime MCP OAuth readiness

* fix: preserve live MCP OAuth classification

* style: sort MCP connection imports
2026-08-05 19:42:26 -04:00
Dustin Healy
4cec1a675f
🔒 feat: Add allowedAddresses Exemption to Speech (STT/TTS) and OCR Config Schemas (#14559)
* feat: add allowedAddresses exemption to speech (STT/TTS) and OCR config schemas

Add the existing allowedAddressesSchema as an optional field on sttSchema,
ttsSchema, and ocrSchema, reusing the schema already attached to endpoints,
mcpSettings, and actions so port scoping and normalization stay identical.

STT and TTS resolve a single provider by counting non-empty section keys, so
exclude the allowedAddresses key from that scan. Without the exclusion a
configured exemption list would be counted as a second provider and trip the
"Multiple providers are set" guard. The field is inert on its own: nothing
reads it for SSRF yet, and provider detection now ignores it.

* fix: preserve allowedAddresses through the OCR config loaders

loadOCRConfig rebuilt the ocr config with only apiKey, baseURL,
mistralModel, and strategy, dropping allowedAddresses before it reached
req.config.ocr. Pass it through in both the AppService loader
(packages/data-schemas/src/app/ocr.ts) and the duplicate at
packages/api/src/files/ocr.ts so the exemption survives config load.
2026-08-05 13:42:43 -04:00
Dustin Healy
3f0a1ec8d9
🛡️ fix: Run message-filter PII patterns on a linear-time regex engine (ReDoS) (#14554)
* 🛡️ fix: Run message-filter PII patterns on a linear-time regex engine

The messageFilter.pii middleware compiled admin-configured customPatterns with the native RegExp engine and ran them synchronously against every message on the shared event loop, so a catastrophic-backtracking pattern such as (a+)+$ could stall the entire process (native RegExp takes tens of seconds at roughly 32 characters) and take the instance down for every user.

Compile these patterns with RE2JS, a linear-time RE2 port with no native addon, so catastrophic backtracking is impossible regardless of the pattern rather than something the code tries to detect. Patterns using features RE2 does not support, such as backreferences, fail to compile and are dropped and logged exactly as an invalid pattern already is. The filter only tests for a match, so this is a drop-in engine swap with no behavior change for valid patterns.

* 🛡️ fix: Reject RE2-incompatible messageFilter patterns at config load

The customPatterns regex was validated with native RegExp at config load, but the runtime now compiles it with a linear-time engine (RE2) that does not support backreferences or lookaround. Such a pattern passed validation, then failed to compile and was silently dropped at request time, quietly removing PII protection after upgrade.

Reject backreferences and lookaround during config validation with an explicit message, and document RE2 syntax in the example config instead of "JavaScript-flavor". The runtime engine remains the authoritative boundary and still drops-and-logs anything this load-time check misses.

* 🧹 test: Use direct MessageFilterPiiConfig annotations in the PII specs

The added ReDoS cases satisfy the exported MessageFilterPiiConfig type directly, so the `as unknown as` assertions were unnecessary. Annotate the config objects directly, matching the repo's type-safety guidance.

* 🧹 fix: Reject named backreferences in messageFilter patterns at config load

Extend the config-load check to also reject named backreferences (\k<name>), which are valid JavaScript regex but unsupported by the linear-time runtime engine, so they surface at load rather than being dropped at request time. Together with the existing numeric-backreference and lookaround checks this covers the RE2-incompatible construct set; the runtime engine remains authoritative.

* 🛡️ fix: Preserve Unicode whitespace matching in messageFilter starter patterns

RE2's \s is ASCII-only, so after the engine swap the built-in api-key and Bearer starters no
longer matched a secret separated by non-ASCII whitespace (e.g. a non-breaking space), which
native RegExp did match. Broaden the whitespace classes to [\s\p{Zs}] so those patterns keep
their original coverage, and add a regression test for a non-breaking-space separator.

* 🛡️ fix: Validate messageFilter patterns with the RE2 engine at config load

Replace the syntax blacklist (numeric/named backreferences, lookaround) with authoritative
validation: config load now compiles each custom pattern with the same linear-time engine the
runtime uses, so any RE2-incompatible construct (including control escapes like \cA) is rejected
at load with a clear error instead of being silently dropped at request time.

The validator is swappable and defaults to native RegExp so browser builds add no engine; the
server wires the RE2-backed check at startup via configureMessageFilterRegexValidator in both
entry points.

* 🛡️ fix: Match the full whitespace set in messageFilter starter patterns

RE2's `\s` omits the vertical tab and `\p{Zs}` omits U+2028, U+2029, and
U+FEFF, so a separator built from one of those characters slipped past the
`api-key` and `Bearer` starter patterns and reached the model. Broaden the
starter whitespace class to the full JavaScript whitespace set so those
separators are covered again.

* fix: fail closed when messageFilter.pii compiles to zero patterns

DB and admin config overrides bypass the RE2 schema validation (it only
runs at YAML load), so an override whose only pattern is RE2-incompatible
was dropped at compile time, left zero patterns, and let the request
through. compile() now returns a failClosed flag when a config declared
patterns but every one failed to compile; the middleware returns 400 and
findPiiMatchInMessages returns a distinct misconfigured match that the
OpenAI and Responses controllers surface with an admin-facing message.

* 🛡️ fix: Fail closed when any messageFilter.pii custom pattern drops

compile() previously set failClosed only when every pattern dropped (patterns.length === 0 && dropped > 0). With the default starters present, a single RE2-incompatible custom override incremented dropped but left patterns.length > 0, so the filter silently enforced only the surviving subset and text matching only the dropped rule passed.

failClosed now keys off dropped > 0, so any dropped custom pattern blocks with the misconfigured 400. YAML patterns are RE2-validated at load, so dropped stays 0 for valid configs and only unvalidated DB or admin overrides can trip it. Reframed the two keeps-others-active specs to assert fail-closed and added a default-starters partial-drop regression.

* 🧹 fix: Correct the misconfigured JSDoc and drop redundant casts in the PII specs

The misconfigured flag now means any configured custom pattern failed to compile, not that every pattern failed, so its JSDoc on PiiMatch is updated to match. The partial-drop regressions now use direct MessageFilterPiiConfig annotations instead of as-unknown-as casts, keeping the specs type-checked, consistent with the rest of the suite.
2026-08-05 13:42:18 -04:00
Danny Avila
ccf4301093
🚦 feat: Configurable Circuit Breakers for Runaway Streamed Tool Args (#14613)
* 🚦 feat: Configurable Circuit Breakers for Runaway Streamed Tool Args

* docs: forewarn create_file about the streamed tool-argument limit

The breaker failing a near-limit write should not be the model's first
exposure to the bound. Both create_file variants now state the default
64 KB per-call limit and the incremental pattern (create the first
section, extend with edit_file) in the tool description and the content
parameter description.

* fix: keep skill create_file description under the provider advisory cap

The limit-guidance paragraph pushed the skill-aware description to 1169
chars, past the 1024-char advisory bound where providers may truncate.
The skill variant now carries the guidance only in its content parameter
description, which sits closest to the generated payload and is not at
truncation risk; the shorter code-sandbox variant keeps the full
paragraph.

* 🚦 feat: per-tool streamed-arg limits with a create_file default

Thirty days of production data show create_file is the only tool class
with legitimate near-limit arguments (p99 80.6 KiB; every other tool
p99 under 10 KiB). Rather than loosening the global 64 KiB cap for all
tools, the yaml gains maxToolCallArgBytesByTool (per-tool overrides,
keyed by model-facing tool name, 0 disables that tool's guard) and
LibreChat ships { create_file: 131072 } by default; yaml entries merge
over and can replace it. Pairs with maxToolCallArgBytesByTool support
in the agents SDK and stays inert until the dependency bump.

* test: pass per-tool spec configs as plain Partial literals

The as-TAgentsEndpoint casts fail TS2352 for object-valued fields:
comparability does not grant nested literals the implicit index
signature that plain assignability does, so casts carrying
maxToolCallArgBytesByTool never sufficiently overlap. The mapper
already accepts Partial<TAgentsEndpoint>, so the new cases pass
uncast literals instead.

* chore(deps): bump @librechat/agents to 3.3.12
2026-08-04 23:05:05 -04:00
Danny Avila
178e61b763
⚓ fix: Bind Action Servers to Metadata Ports (#14575)
* fix: bind action server ports

* style: sort action imports

* fix: normalize action port input

* fix: parse action ports consistently
2026-08-02 13:41:31 -04:00
Danny Avila
cdf437dc5b
🪺 fix: Keep Preempt-Abandoned Siblings Nested, Make Message Tree Order-Robust (#14587)
* 🪺 fix: Keep Preempt-Abandoned Siblings Nested, Make Message Tree Order-Robust

* 🔗 fix: Sever Cycle Back-Edges So the Returned Message Tree Is Acyclic

* 🧪 fix: Satisfy TFile in buildTree Spec fileMap Fixture

* 🌲 fix: Assert Repaired Trees in convoStructure Specs, Uncharge Self-Parent Edges

* 📌 feat: Identity-Stable Sibling Selection Across Background Tree Churn

* 🎭 test: E2E Coverage for Thread Fold and Sibling Selection Invariants

* 🔑 fix: Treat Newest-Sibling Re-Key as Hydration, Not a New Branch

* 🧭 fix: Rebind Sibling Selection Per Parent, Detect Appends by Membership
2026-08-02 07:04:52 -04:00
Danny Avila
ad0f72dede
🌀 ci: Deterministic Circular Dependency Checks (#14579)
* 🌀 ci: Deterministic Circular Dependency Checks

* 🌀 ci: Enforce Type-Level Edges in Circular Dependency Scan

* 🌀 ci: Materialize Import-Type Expression Edges in Cycle Scan

* 🌀 ci: Collect Inline Type-Only Specifier Edges in Cycle Scan
2026-08-01 14:43:26 -04:00
Danny Avila
6f45a9e32e
🔗 fix: Normalize MCP Tool Keys at Every Producer, Resolve Raw Names via Aliases (#14553)
* 🔗 fix: Normalize MCP Tool Keys at Every Producer, Resolve Raw Names via Aliases

Tool keys had two spellings that could diverge for any server whose name
contains characters outside [a-zA-Z0-9_.-]: the tool cache (and registry
inspector) built keys with the RAW server name, while runtime instances
are named with normalizeServerName(serverName). Three code comments
already asserted "tool keys embed the normalized server name" - no
producer honored it. For a special-character server that meant:

- definitions-only mode shipped raw def names the model echoed back,
  but the executor's tool map held the normalized instance name, so
  every call failed with "Tool not found";
- per-tool tool_options (defer_loading / allowed_callers /
  run_in_background / describe_intent) were persisted under raw keys
  that never matched the definition names the option passes resolve
  against, so builder settings were silently inert;
- tool-key parsing against normalized candidate lists silently fell
  back to last-delimiter splitting, which mis-parses delimiter-bearing
  tool names.

The reconciliation is one contract enforced in three moves:

1. PRODUCERS NORMALIZE. The tool cache (packages/api/src/mcp/tools.ts)
   and the registry inspector build keys with the normalized server
   name, matching the instance names MCP.js has always assigned. The
   builder's tool ids, agent.tools entries, tool_options keys, and
   definition names all flow from these keys, so every model-facing
   name now agrees. The cache STORE stays keyed by the raw config name.

2. CONFIG LOOKUPS RESOLVE ALIASES. New shared helpers in data-provider
   (buildServerNameAliases, normalizeMCPToolKey) map a parsed
   normalized name back to the raw config name that the registry,
   config maps, tool cache, and plugin-auth rows are keyed by. Applied
   in the definitions loader closure, handleTools grouping,
   createMCPTool's parsing fallback, getUserMCPAuthMap, and the MCP
   tools endpoint - matching both spellings so legacy raw keys keep
   resolving.

3. LEGACY DATA HEALS AT ONE BOUNDARY. initializeAgent rewrites
   raw-keyed agent.tools entries and tool_options keys to the
   normalized form (normalizeAgentToolKeys) before anything consumes
   them, so agents persisted under the old convention load their tools
   AND have all four per-tool options honored. Placeholder and
   server-pin tokens stay raw - they are config-identity references,
   not model-facing names.

Servers whose names are already in the safe character set (the common
case) produce byte-identical keys before and after; the fast path
allocates nothing. Stale Redis-cached raw keys self-heal via the
existing reconnect-on-missing path within one cache cycle.

* 🧯 fix: Deterministic Alias Collisions + Raw Names in Definition Metadata

Two review findings on the normalization contract:

- Two configured server names that normalize to the same segment (e.g.
  'Sales Force' and 'Sales:Force' -> 'Sales_Force') produce inherently
  ambiguous tool keys; the alias map silently resolved last-wins, so a
  tool selected from one server could execute against the other's
  config. buildServerNameAliases now resolves collisions to the FIRST
  configured name deterministically, and resolveMCPServerContext warns
  once per colliding pair per process so the operator can rename one
  server. A collision-resistant identifier would change every existing
  tool key, so detection + stable routing is the right treatment here;
  startup-time config validation can follow separately.

- The definitions loader resolved parsed (normalized) server names to
  raw only inside the ToolService closure, while the definition
  metadata (serverName -> mcpRawServerName) kept the normalized value.
  Server instructions are keyed by raw config names, so a
  special-character server's instructions were silently omitted in
  definitions-only mode. loadToolDefinitions now takes rawServerNames,
  resolves the boundary against both spellings, and stores the RAW
  name in definition metadata - consistent with the instance path.

* 🧯 fix: Heal Stale Caches, Skill Allowed-Tools, and Builder Selectors

Three review findings on the normalization rollout, all in the
transition class:

- Stale cache entries (P1): the definitions-only loader treats the
  per-server tool map as authoritative and never reconnects on a
  per-key miss, so a pre-change raw-keyed Redis entry would make a
  special-character server's tools vanish for up to the cache TTL.
  getMCPServerTools now heals legacy raw-keyed entries to the
  normalized format at read time (keys and function names), covering
  every consumer with no coordinated invalidation; safe names return
  the map untouched.

- Skill allowed-tools: a skill declaring a raw MCP key in
  allowed-tools bypassed the initialize-boundary heal (the union runs
  after it) and would neither dedupe against healed agent tools nor
  match the normalized tool map. The primes' allowedTools now pass
  through the same normalizeAgentToolKeys heal before unioning.

- Builder selectors: matchesMcpServer and useVisibleTools parsed tool
  ids against raw server names only, so an attached special-character
  server rendered as an unselected orphan card. Both now accept the
  normalized spelling and resolve it back to the raw map key, keeping
  legacy raw ids working.

* 🧯 fix: Fail Closed on Normalized Server-Name Collisions

Escalation of the collision finding: a deterministic first-wins alias
plus a warning still let the tools listing publish BOTH colliding
servers, so a tool selected under the shadowed second server would
silently execute against the first server's configuration (their
model-facing keys are identical, so routing cannot ever distinguish
them).

- findShadowedServerNames identifies later-configured names whose
  normalized form an earlier different name claimed.
- getMCPTools excludes shadowed servers from the published listing
  entirely (with a warn naming the collision), so their tools are
  never selectable - nothing ambiguous can be picked.
- Server creation reserves both spellings: a generated slug may not
  collide with a raw config name OR the normalized form its tool keys
  would carry.

Collision-resistant model-facing IDs remain out of scope: changing
normalizeServerName's output would rewrite every existing tool key
(agent documents, caches, instance names) for ALL servers to handle a
misconfiguration that is now blocked from exposure instead.

* ✅ fix: Dedupe Reserved Server-Name Spellings at Creation

The reservation list appended normalized forms unconditionally, which
duplicated every safe name (raw === normalized) and broke the
route-level contract test pinning the exact list. Dedupe via a Set so
safe names contribute one entry, while special-character names still
reserve both spellings; adds the special-character reservation case.

* 🧯 fix: Never Heal a Shadowed Server's Keys; Align Authorization Tie-Break

Persisted references were the remaining collision vector: an agent or
skill saved with the shadowed later server's raw key was HEALED into
the shared normalized key, authorized through a last-wins map, and
routed first-wins - authorized as one server, executed as another.

- normalizeAgentToolKeys now refuses to rewrite keys of shadowed
  servers (findShadowedServerNames): rewriting would produce exactly
  the first server's key. Left raw, the key cannot match the
  normalized-keyed tool map and the tool fails visibly - broken beats
  misrouted. Covers agent.tools, tool_options, and skill
  allowed-tools through the shared heal.

- filterAuthorizedTools (agents/v1.js) builds its normalized-to-raw
  map via the shared buildServerNameAliases instead of a last-wins
  Map constructor, so authorization resolves a colliding key to the
  SAME first server execution routes to.

* 🧯 fix: Direct Identity Wins Over Aliases; Heal Client Forms and Degraded Contexts

Four review findings on the normalization edges:

- Alias hijack (P1): a user-DB server named exactly like an operator
  server's normalized form ('foo' vs YAML 'foo!') had its tools
  rerouted to the operator server by unconditional alias resolution.
  Resolution is now DIRECT-FIRST everywhere: the parsed name is tried
  as-is, and only when nothing resolves is it treated as a normalized
  spelling (definitions loader, handleTools grouping, createMCPTool
  fallback). buildServerNameAliases seats identity entries before
  derived ones so a literal name owns its slot regardless of config
  order, findShadowedServerNames and the collision warning derive from
  the same construction, and getUserMCPAuthMap fetches auth under both
  spellings so either owner finds its rows.

- Builder double-match: a normalized name containing the delimiter
  ('foo mcp bar' -> 'foo_mcp_bar') also suffix-matched a server named
  'bar', selecting both cards and making removal strip the wrong tool.
  matchesMcpServer now resolves the token ONCE against the full
  configured list (longest boundary, both spellings) when the caller
  supplies it; selection and removal share the resolution.

- Builder legacy ids: an agent saved with raw-keyed ids showed its
  tools unchecked while the runtime heal kept them active, and
  selection updates never replaced the legacy entries. McpSection maps
  legacy raw ids to their current normalized ids when deriving and
  rewriting this server's selection.

- Degraded context: a transient ensureConfigServers failure returned
  an entirely empty context, leaving normalized keys unresolvable for
  the request. resolveMCPServerContext now keeps the name lists (they
  derive from the config snapshot alone) and degrades only the
  lazy-init configs.

* 🧯 fix: Collision Detection Sees Accessible Servers; Shadowed Refs Fail Closed End to End

Round follow-ups on the collision design, all in the
DB-server-visibility class:

- The legacy-key heal detected collisions against operator-config
  names only, so healing could still produce a key that direct-first
  resolution routes to an invisible user-DB server. initializeAgent
  gains an optional getAccessibleMcpServerNames dep (wired through
  ToolService for controllers that mock it, directly elsewhere),
  consulted ONLY when a configured name needs normalization - zero
  cost for safe-name deployments. The heal then sees the full
  accessible set and skips shadowed servers' keys.

- Wildcard and legacy raw tokens bypassed catalog filtering, letting a
  shadowed server's instances join a run under the same normalized
  names as the winner's. filterAuthorizedTools rejects tools of
  shadowed servers at authorization (its merged map sees DB + config),
  and handleTools skips them at execution.

- The builder migrated only tool selection, not tool_options: legacy
  raw option keys showed disabled while the runtime honored them, and
  toggles could not clear them. McpSection now migrates option keys to
  the current normalized ids (existing normalized entries win).

- A transient ensureConfigServers failure degraded to an EMPTY server
  context, leaving normalized keys unresolvable for the request.
  resolveMCPServerContext keeps the name lists (derived from the
  config snapshot alone) and degrades only the lazy-init configs.

* 🧯 fix: Complete the Collision Audit at Every Gate; Safer Heal Semantics

Round follow-ups hardening the collision audit:

- Execution guards now consult the FULL accessible set: the caller's
  heal threads its already-fetched names through loadTools, and
  handleTools fetches them itself when a configured name needs
  normalization (never for safe-name deployments) - so a cross-tier
  collision (user-DB 'foo' vs operator 'foo!') fails closed at eager
  execution instead of joining the run under one normalized name.

- Healing is SKIPPED when the collision audit cannot complete
  (transient lookup failure, or no dep): un-healed raw keys still
  resolve through the direct-first candidates, so skipping is safe
  while rewriting against an incomplete audit is not.

- The audit lookup is gated on the agent actually carrying
  delimiter-bearing keys (tools, tool_options, or skill
  allowed-tools), so non-MCP agents never pay a registry round-trip
  even on specially named deployments.

- normalizeAgentToolKeys gives the CURRENT (normalized) entry
  precedence when both spellings carry options, matching the builder's
  migration semantics instead of letting insertion order decide.

- The builder's toCurrentToolId resolves entries boundary-exactly
  against every configured server (longest match, both spellings), so
  a raw suffix shared with a LONGER server name can no longer reassign
  that server's selection or options while another dialog is open.

* 🧯 fix: Shared Collision Audit for Definitions Loading; Fail Closed on Audit Failure

Round follow-ups closing the remaining audit gaps:

- The definitions-only loader now consumes the same collision audit as
  eager loading: shadowed servers' entries (wildcards included) are
  dropped before definitions are emitted, so the default execution
  path can never resolve a shadowed server's normalized function name
  to another server. The audit names thread from initializeAgent's
  heal; the loader self-fetches only when a configured name needs
  normalization.

- resolveCollisionAuditNames centralizes the audit-resolution policy
  (threaded set > self-fetch when needed > incomplete on failure), and
  BOTH loaders now fail closed under an incomplete audit: any
  normalization-sensitive reference (its own name needs normalizing,
  or it equals the normalized form of a configured special-character
  name) is skipped with a warning instead of being audited against
  operator names alone. isNormalizationSensitiveName lives in
  packages/api as a pure helper so test mocks use the real predicate.

- normalizeAgentToolKeys collapses duplicate ids after healing
  (order-preserving): a document carrying both spellings converges on
  one key, never two instances with the same function name.

* 🧯 fix: Thread the Audit Everywhere; Identity-Aware Alias Fallback

Round follow-ups on audit plumbing:

- The OpenAI-compatible and Responses tool loaders now forward the
  already-resolved accessibleMcpServerNames instead of discarding it,
  so the definitions loader neither repeats the registry lookup nor
  fails closed on a transient second lookup after the first succeeded.

- The skill-only path threads its audit: when the baseline agent has
  no MCP keys but a primed skill's allowed-tools fetched the complete
  set, that set (not the operator-only list) reaches the loader, so
  the collision remains visible and the shadowed reference stays
  rejected end to end.

- OAuth discovery iterates the collision-FILTERED tool list, so a
  request can no longer emit an OAuth prompt, wait out the connection
  timeout, and reconnect a server whose definitions were deliberately
  rejected.

- The definitions loader's alias fallback is identity-aware: when the
  parsed name IS a known accessible server, a null tool fetch means
  temporarily unavailable (OAuth pending, missing user variables,
  disconnected) and no longer reroutes to the raw alias - previously
  the aliased operator server's definitions could be emitted under the
  unavailable DB server's names.

* 🧯 fix: Legacy-Key Definition Lookup; Retain Audit for Deferred Execution

- createMCPTool resolves tool definitions by BOTH spellings: the key as
  persisted plus the canonical normalized key built from the resolved
  server name. Assistants and direct tool calls persisted before the
  rollout bypass the agent-boundary heal and arrive with raw keys, while
  availableTools is now indexed canonically - previously every such call
  missed the index, burned a reconnect, and returned the unavailable
  stub permanently via the negative cache.

- The initialized agent retains accessibleMcpServerNames (the COMPLETE
  collision audit this initialization resolved), buildAgentToolContext
  copies it into every per-agent tool context, and loadToolsForExecution
  threads it into the eager loader as bare options. Deferred/event-driven
  execution therefore reuses the snapshot instead of repeating the merged
  registry read - a transient failure there could fail-closed a tool the
  same turn already advertised from the successful first audit.

- MCP.spec.js keeps @librechat/api pure helpers REAL (requireActual
  spread) so normalization paths are exercised rather than mirrored.

* 🧯 fix: Parse Legacy Keys Against Both Server-Name Spellings

createMCPTool's boundary candidates were normalized-only, so a legacy
raw key whose server name contains the delimiter (foo_mcp_bar!) missed
the suffix match and fell to the generic last-delimiter split - the
canonical rebuild then produced a key that could never hit the index
and the persisted call stubbed out. The candidate list now carries the
RAW resolved name (and raw config names on the parse-only path) next
to the normalized spellings.

* 🧯 fix: Honest Audit Completeness; Shadowed-Server Form-Key Guard

- resolveAllMcpConfigs tolerates ensureConfigServers failures, so the
  merged registry read can silently omit config-only servers while the
  audit still reported complete: true - a foo/foo! collision would go
  unseen and a persisted key could route to the wrong server. Both
  audit consumers now union the snapshot-derived raw config names back
  in (resolveCollisionAuditNames unions the caller's rawServerNames;
  the initializeAgent heal unions configRawServerNames), keeping the
  completeness label honest without an extra read: operator names come
  from the registry-independent config snapshot, user-DB names from the
  merged read that fails loudly into the existing incomplete path.

- The client tool_options migration now mirrors the runtime heal's
  fail-closed rule for SHADOWED servers: when the dialog's server has
  lost its normalized slot to another catalog name, legacy raw keys
  stay raw instead of being rewritten onto the winning server's key,
  where a later save would apply the wrong server's per-tool settings.
  The dialog's own server joins the alias construction so a stale
  catalog map can't misread as a collision.

* 🧯 fix: Heal Legacy Assistant MCP Tool Names on Save

The assistants create/update controllers look tools up in the cached
definitions by exact key, and the cache is now normalized-keyed - an
assistant saved before the convention resubmits its raw-suffixed MCP
name on every edit, so any save silently removed the tool.

healMcpToolNames pre-heals the payload's tool list: a delimiter-bearing
string that misses the cache resolves through the configured raw names
(longest-suffix, boundary-exact) and rewrites to the normalized key
only when that key actually exists in the cache. SHADOWED raw names
stay raw and fail closed, mirroring the runtime heal; the config read
happens only when a delimiter-bearing name actually misses, and read
failures propagate (write path) rather than silently dropping tools.
v2's update loop also stops re-reading the tool cache per iteration.

* 🧯 fix: Full-Audit Shadow Set + Dedupe in the Assistant Key Heal

- The assistant-save heal built its shadow set from operator config
  names alone, so a cross-tier collision (user-DB `foo` owning the
  normalized slot of operator `foo!`) looked unshadowed and the legacy
  key healed into the shared normalized name - which direct-first
  execution then binds to the DB server. The shadow set now comes from
  resolveCollisionAuditNames' full accessible audit, and an incomplete
  audit skips healing outright (every rewrite candidate is
  normalization-sensitive by construction, so raw-and-fail-closed is
  the only safe answer).

- Healed string entries dedupe order-preserving: a payload carrying
  both spellings of the same tool collapses to one entry instead of
  expanding into duplicate function definitions the provider rejects.
2026-08-01 07:39:24 -04:00
Danny Avila
e7f1838515
⚡ feat: Reliable Interrupt & Steer Escalation and Recovery (#14558)
Some checks are pending
Docker Dev Branch Images Build / build (Dockerfile, lc-dev, node) (push) Waiting to run
Docker Dev Branch Images Build / build (Dockerfile.multi, lc-dev-api, api-build) (push) Waiting to run
GitNexus Index / index (push) Waiting to run
GitNexus Index / post-index (push) Blocked by required conditions
* feat: surface interrupt-steer escalation on waiting messages

The interrupt & steer feature shipped reachable only through the
composer chord, the send-button hovercard, and the composer button; a
message already waiting (queued for after the run, or steered and
parked at the next tool boundary) had no path to it. Both waiting
surfaces now carry one:

- Queued rows get an icon-only ZapOff escalation button beside the
  existing Steer primary. It routes through sendQueuedNow, which now
  takes a preempt option on its live-run path. The tooltip teaches the
  composer chord, derived through resolveComposerKeyDown so a rebound
  or yielded chord is never advertised.
- In-flight steer bubbles get an "Interrupt now" overflow entry with
  the same race rules as Edit: reclaim first, and only a `reclaimed`
  outcome resubmits (via retrySteer with preempt, swapping the chip
  for an interrupting one). `applied` and run-ended-mid-reclaim
  outcomes stop at the existing informational toasts, so the words can
  never land twice. Not offered on a steer already preempting.
- Every during-run overflow menu gains an "Always interrupt instead"
  toggle for steerInterruptsByDefault, next to the existing steer/queue
  default toggle. MenuEntry supports disabled for the new entries.

Only one interrupt can be unresolved at a time: while one preempt is
pending (or the run is paused on approval, where the server 409s),
every escalation control disables instead of racing the same seal.

Ten new tests across both surfaces; 381 green in the affected suites.

* fix: lock escalation across its reclaim window, keep the paused control visible, label as steer

Codex round 1, all three findings.

P2, escalation race. The single-interrupt invariant had a window between
clicking "Interrupt now" and the reclaim resolving, where no preempt
chip existed for the chip-derived gate to see: two bubbles escalated
back-to-back could both resubmit. A shared escalating flag (Jotai,
per-conversation) now covers the window and disables every escalation
control on both surfaces, and a fresh recheck before resubmitting
catches an interrupt armed elsewhere meanwhile (composer chord, queued
row); those words re-home to the queue with an informational toast
instead of breaking the invariant.

P2, unreachable paused state. canSteer is defined as
hasRealConvoId && !pausedOnApproval, so gating the button on canSteer
removed it exactly when it was meant to render disabled; the test only
passed on an impossible stub combination. The render gate is now
duringRunActive && (canSteer || pausedOnApproval), and the test uses the
real invariant.

P2, label semantics. "Interrupt & send now" borrowed the name of the
hard-abort action; this one preserves the partial answer and steers.
Renamed to "Interrupt & steer now" (com_ui_interrupt_steer_now).

Both behavior fixes counterfactually verified; 384 tests green across
the affected suites.

* fix: disable bubble escalation while the run cannot accept a steer

Codex round 2, one P2. Answer mode (ask_user_question) sets
duringRunActive false while pausedOnApproval stays false, since that
flag only detects approval-bearing tool calls. The bubble's escalation
entry stayed enabled there, so clicking it cancelled a healthy waiting
steer and the preempt resubmission bounced off RUN_PAUSED, degrading
the words to the queue. The entry now also disables on
!duringRunActive, matching the queued-row control's gate.

Counterfactually verified: reverting the gate fails the new
answer-mode test.

* fix: recheck live run state after the reclaim, not just at the click

Codex round 3, one P2, and it is the round-1 recheck principle applied
one level deeper: the entry-time disable cannot see a run that pauses
(tool approval, answer mode) while the reclaim round-trip is in flight,
and the .then closure held the render's stale steering controls, so the
resubmit would fire into a RUN_PAUSED rejection after the reclaim had
already surrendered the steer's boundary slot.

The escalation continuation now reads the LIVE controls through a
latest-ref: if the run can no longer accept a steer, the words re-home
to the queue with an informational toast instead of resubmitting, and
the resubmit itself also goes through the live controls.

Counterfactually verified: reading the stale closure instead of the ref
fails the new mid-reclaim pause test.

* refactor: make escalation one atomic server-side arm, in place

Codex round 4: four P2s, every one an interleaving of the same window —
escalation as reclaim-then-repost is a compound, non-atomic operation
whose continuation must revalidate the world (FIFO position lost, ref
assigned too late, no run fence, competing bubble actions). Rounds 1-3
patched that window with a lock and rechecks; round 4 shows the window
itself is the defect, so this removes it instead of guarding it again.

Escalation is now POST /chat/steer/arm: the server flips preempt on the
EXISTING queued item in one atomic store op (new IJobStore.armSteer; a
decode-patch-encode LSET Lua on Redis, an in-place mutation in memory),
fenced to the validated generation and refused once the queue closes.
The handler mirrors the steer POST's preempt contract exactly: durable
flag gated on the owner's recorded capability, volatile requestPreempt
fire-and-forget because the durable flag is the truth resume/handover
re-arm from.

By construction this resolves all four findings: FIFO survives (the
item never moves; the whole queue still drains in instruction order at
the seal), there is no continuation to hold stale controls, the store
op is fenced to the original run, and a competing Edit/Queue/Cancel
either beats the arm (armed:false, chip untouched) or operates on the
armed item, whose cancel already disarms.

The client escalation entry becomes one mutation: armed:true relabels
the chip in place (same steerId, same position), PREEMPT_UNSUPPORTED
and lost races toast honestly, and the round 1-3 machinery — the
escalating lock atom, the latest-ref, the post-reclaim rechecks and
their two toast strings — is deleted rather than extended.

Verified: 7 new handler tests on the real in-memory manager (including
FIFO preservation and the stale-generation fence), 2 Redis integration
tests against real Redis (in-place arm keeps order and every field;
missing/stale/closed all refuse), client suites 396 green.

* fix: decide capability inside the atomic arm, neutralize the lost-race toast

Codex round 5, both findings, both edges of the new arm design rather
than its mechanism.

P2, capability TOCTOU. A HITL resume on a rolling deploy rewrites
preemptCapable for the SAME generation, so the handler's read could go
stale between validation and the flag flip, arming a steer the live
owner cannot seal. armSteer now returns armed | missing | incapable,
with the owner's live capability part of the same atomic predicate as
the generation fence (HGET preemptCapable inside the Lua; the flat job
field, not a metadata blob — the in-memory store reads the same field).
The handler's pre-check is deleted rather than kept alongside; the
store predicate is the single source. New handler test rewrites the
capability after queueing and expects PREEMPT_UNSUPPORTED with the item
left unflagged; the Redis guards test now asserts the incapable refusal
against real Redis.

P2, ambiguous toast. armed:false covers injected, cancelled, re-homed,
and run-over alike, so telling the user the message "already reached
the agent" claimed one specific outcome. The lost-race branch now uses
a neutral message (com_ui_steer_arm_lost_race) and defers to the events
for what actually happened.

* fix: flip the escalation lock synchronously before the arm request

Codex round 6, one P2. Round 4 deleted the escalating flag along with
the reclaim continuation it guarded, but that left the one-interrupt
gate blind during the arm request's own round trip: the chip-derived
check cannot see an arm until its response relabels the chip, so on a
slow connection two bubbles could both arm before either response
landed. Double-arm is harmless server-side now (the run seals once and
drains the whole queue in order), but every escalation control
advertises "one interrupt at a time" by disabling, and the controls
must tell the truth.

The per-conversation escalating flag returns as a pure UX gate: set
synchronously at click, before the mutation, cleared on settlement, and
folded into interruptPending on both surfaces. Unlike its round 1-3
ancestor there is no continuation behind it to guard and no recheck to
pair with it.

Counterfactually verified: without the synchronous set, the two-bubble
race test arms twice. 207 tests green across the Chat Input suites.

* test(e2e): cover escalation of waiting messages through the real seal

Three mock-harness tests on E2E_SLOW_REPLY, a 160-chunk stream with no
tool boundary, so an in-thread steer part can ONLY come from a genuine
mid-stream seal — which makes each test a behavioral proof rather than
a UI check:

- Queued row escalation: the ZapOff button turns a waiting queued
  message into a preempt-armed steer (202 echoes preempt: true) that
  seals and injects, where the sibling steering.spec test proves the
  unescalated path waits for run end instead.
- Bubble in-place arm: an ordinary steer (202 with no preempt echo)
  waits as a bubble, POST /chat/steer/arm answers armed: true, the
  bubble relabels in place (same single bubble, same text, escalation
  no longer offered on reopen), and the armed steer seals mid-stream.
- Always-interrupt toggle: flipped from a waiting row's overflow menu,
  plain Enter now produces a preempt: true steer that seals in the SAME
  run, and the menu offers the way back. An afterEach clears the
  localStorage preference so a mid-test failure cannot leak
  preempt-by-default into the rest of the serial suite.

All three verified locally through the full harness (real backend, mock
LLM, seeded DB): 3 passed in 27s.

* feat: dedicated escalation arrow + shortcut, menu split into actions and preferences

The escalation was still half-hidden: the bubble only offered it inside
the overflow menu, and the tooltip taught the composer chord, which does
a different thing (interrupts with typed text, not this chip). Three
changes make it a first-class command:

- A shared EscalateNowButton (circular arrow, ghost-bordered like the
  composer's interrupt control) is always visible on BOTH surfaces:
  beside each queued row's Steer primary and on every waiting steer
  bubble next to its menu. It disappears once a steer is interrupting.
- A dedicated registry shortcut, escalateSteer (Cmd/Ctrl+Shift+.),
  editing-allowed and rebindable like every other action. Deliberately
  NOT an Enter chord: the composer owns every Enter chord, and the
  yield design rests on no default binding using Enter besides submit.
  Its handler clicks the newest enabled arrow control (bubbles beat
  queued rows), so the shortcut can never diverge from the button, and
  the arrow's tooltip teaches THIS command via the registry display.
- The overflow menus separate one-off actions from sticky behavior
  changes: Edit, Cancel, Queue, then a smaller "Preferences" section
  holding the queueing and always-interrupt toggles, each with the
  standard InfoHoverCard reusing the Settings panel's descriptions.
  "Interrupt & steer now" leaves the menu entirely.

386 client tests green, including a menu-structure test locking the
order and the absence of the escalation entry; bubble escalation tests
drive the visible arrow. The e2e spec's bubble test now clicks the
arrow, and a fourth test drives the dedicated shortcut end to end
through a real mid-stream seal.

* style: bind the escalation arrow to its message (variant A anatomy)

Two same-weight circles in a row read as one control group, leaving the
arrow's ownership ambiguous, and a floating arrow stops meaning anything
once several messages stack. The shared control now carries variant A's
anatomy: a thin divider binds a small SOLID arrow (filled, inverted) to
the message region on its left, and the menu ellipsis stays a bare
glyph, so the two affordances can no longer blur together — and the
divider+arrow pairing repeats cleanly per chip at N messages.

* chore: drop the unused within import CI lint caught

* fix: advertise the escalation shortcut only while the control is live

Codex on the e2e head, one P2: the tooltip appended the chord hint even
while the button was disabled, advertising a shortcut that does nothing
during an approval pause. The flagged control (InterruptNowButton) was
since replaced by the shared EscalateNowButton, which inherited the
pattern; the successor now omits the chord whenever the control is
disabled, matching the rule the during-run hovercard already follows.

* fix: harden steer escalation lifecycle and recovery

* test(e2e): disambiguate accessible steer preferences

* test: align abort persistence coverage with prerequisites

* chore(i18n): remove obsolete steer race message

* chore: normalize imports across steering changes

* test: exercise stream integration on Redis Cluster

* test: scope HITL checkpoints to generation

* test: fix cluster cleanup and locale policy

* fix: keep escalation visible during ask pauses

* fix: fence recovery downgrade and stale predecessors

* fix: require generation owner abort acknowledgement

* fix: validate delayed preempt arms

* test: align final escalation fixtures

* fix: preserve in-memory predecessor abort handoff

* fix: restore controls for recovered queued messages

* test: cover recovered queue controls

* fix: close final steering review gaps
2026-07-31 20:07:56 -04:00
Danny Avila
1e1de6eff9
🎯 fix: Exact Ask-Question Attribution via Interrupt tool_call_id (#14539)
The ask_user_question pause/answer stamps (server pause-time args stamp,
resume-time answer stamp, and the client mirror) targeted the newest
unanswered ask part by position. When a model emits several ask calls in
one turn, the interrupt's question and the user's answer land on the
wrong card.

@librechat/agents > 3.3.8 surfaces the interrupting call's tool_call_id
on the ask interrupt payload. All three stamps now target that id
exactly when present, keeping the positional fallback for older
payloads. The tool body passes config.toolCall.id through to
askUserQuestion via a typed alias that is a no-op on the pinned SDK and
lights up on the next dependency bump.

Companion to danny-avila/agents#366, which also fixes the underlying
dangling tool_use 400 (INVALID_TOOL_RESULTS) when one of the parallel
ask calls streams malformed args.
2026-07-31 10:11:08 -04:00
Danny Avila
f0d3bcb622
📄 fix: Filter Non-PDF Documents on the Anthropic Encode Path (#14535)
Anthropic's Messages API only accepts application/pdf for base64 document
sources, but encodeAndFormatDocuments sent every allowlisted file (docx,
xlsx, csv, html) through the base64 branch unfiltered. The provider 400
recurs on every retry because attachments are re-encoded each request,
permanently breaking the conversation.

- Add isAnthropicDocumentType / isAnthropicTextDocumentType to
  data-provider, mirroring isBedrockDocumentType
- Filter unsupported types before encoding (matching Bedrock semantics)
  and log the skipped attachments
- Send textual types as plain-text document sources (source.type 'text'),
  which Anthropic accepts and supports citations for, instead of invalid
  base64 blocks

Fixes #14485
2026-07-31 09:57:02 -04:00
Danny Avila
8e165eb451
🔒 fix: Remove Owner Email from Agent owner_contact Fallback (#14541)
Some checks are pending
Docker Dev Branch Images Build / build (Dockerfile, lc-dev, node) (push) Waiting to run
Docker Dev Branch Images Build / build (Dockerfile.multi, lc-dev-api, api-build) (push) Waiting to run
GitNexus Index / index (push) Waiting to run
GitNexus Index / post-index (push) Blocked by required conditions
* 🔒 fix: Remove Owner Email from Agent `owner_contact` Fallback

The owner-contact fallback for agents without an explicit support_contact
exposed the owner's private account email to any VIEW-level caller via
GET /agents/:id and GET /agents. The fallback now resolves a display name
only (name/username/authorName): the User query no longer projects email,
the resolver never returns one, and the shared AgentOwnerContact type drops
the field. Emails are only served when the owner opts in via support_contact.

* 🔒 fix: Reject Email-Shaped Owner Display Names in Contact Fallback

OpenID and SAML strategies fall back to the account email for the user's
name and username when no display-name claims exist, so the name-only
owner fallback could still surface the email through those fields. The
resolver now rejects email-shaped display-name candidates entirely.

* 🔒 fix: Treat Any @-Containing Display Name as Email-Derived

RFC-5321 quoted local parts may contain whitespace and the User schema
email validator is an unanchored substring match, so such addresses can
reach the name/username fields via SSO fallbacks. Rejecting on '@'
presence covers every legal email form without re-fetching the account
email.
2026-07-30 23:46:22 -04:00
Danny Avila
d6c2dc5d8e
🧵 feat: Background-Native Code Execution Tools (#14532)
The code-execution pair (execute_code/bash_tool) now defaults INTO
background dispatch whenever the run_in_background capability is enabled,
the same way the SDK's coding tools carry `intent` natively: enabling the
capability is enough, with no per-tool or per-spec flag required. An
explicit run_in_background: false opts the pair out (by definition name,
marker projection, or a narrowing spec selection's wildcard), and the
builder Code toggle flips to opt-out semantics: absent reads as on, and
turning it off persists an explicit false.

A spec's runInBackground: false now synthesizes an explicit wildcard
opt-out instead of staying a silent no-op. Pre-native, false and absent
were behaviorally identical, so a config that wrote false must not
silently flip to backgrounding code. The ephemeral toggle's false stays
no-policy: it is a badge default, not a decision.
2026-07-30 17:53:47 -04:00
Danny Avila
d5819becf2
🎯 feat: Per-Tool Intent Labels for Model Specs (#14526)
* 🎯 feat: Per-Tool Intent Labels for Model Specs

A model spec could only turn intent labels on for ALL of its eligible
tools. Saved agents have had per-tool control since the capability landed
(`tool_options[id].describe_intent`, with the builder toggle following in
the UI slice), but a model spec is admin YAML that produces an ephemeral
agent — there is no agent document to hold per-tool options, so
`describeIntent: true` synthesized an entry for every eligible tool.

`describeIntent` now accepts a string array alongside the boolean,
matching the `skills` field already in the same schema:

  describeIntent: true                                  # every eligible tool
  describeIntent: ['web_search', 'search_code_mcp_github']

This matters because the label costs schema tokens on every request, so
an admin may want it on a handful of illegible calls rather than the
whole toolset.

- Named tools still pass eligibility, so an excluded tool cannot be
  forced on by listing it.
- An empty array reads as disabled.
- Names that are not eligible or not equipped on the spec are logged
  rather than silently skipped — a typo in a spec would otherwise be
  undiagnosable.
- The ephemeral toggle stays boolean and stays global even when a spec
  list is present: it has no per-tool UI to drive it, so narrowing it
  would silently cover fewer tools than the user asked for.

`runInBackground` has the same all-or-nothing limitation and could take
the same shape; left alone here to keep this change reviewable.

* 🎯 feat: Per-Tool Background Dispatch for Model Specs

Gives `runInBackground` the same `boolean | string[]` shape as
`describeIntent`, so the two per-tool capabilities are configured
identically from a model spec:

  runInBackground: true                          # every eligible tool
  runInBackground: ['slow_report_mcp_analytics']  # only this one

Selectivity matters more here than for intent labels. An intent label is
inert — it costs tokens and nothing else. Backgrounding changes execution
semantics: the model gets a synthetic handle and must poll. Letting an
admin detach one slow MCP call without making every other tool in the
spec detachable is the difference between a usable setting and an
all-or-nothing one.

Same guarantees as the intent equivalent:
- Named tools still pass eligibility, so the exclusion list still holds —
  a list cannot force on web_search, file_search, image gen, the HITL
  tool, or anything whose attachments/artifact continuity would break.
- Empty array reads as disabled.
- Unmatched names are logged rather than silently skipped.
- The ephemeral toggle stays boolean and global; it has no per-tool UI.

Also fixes a latent no-op: this function did not skip the lazily-expanded
`mcp_all` placeholder, so a spec with an overlay MCP server recorded an
option under a name `applyBackgroundToolCalls` can never match. The
intent equivalent already skipped it; now both do.

* ♻️ refactor: One Definition of the mcp_all Placeholder Guard

Fixing the background no-op left the placeholder prefix declared twice —
once per capability synthesizer — which is the same duplicated-literal
shape that made the intent label marker fragile: two copies that must
agree, with drift producing a silent no-op rather than an error.

`MCP_ALL_PLACEHOLDER_PREFIX` and `isMCPAllPlaceholder` now live beside
`mcpToolPattern` in mcp/utils, so both synthesizers cannot disagree about
which tool entries to ignore, and anything added later that keys per-tool
config by exact name has an obvious guard to reach for.

Audited the rest of the capability family while here: only background and
intent synthesize per-tool options from a model spec. `defer_loading` and
`allowed_callers` have no model-spec path at all, so neither can carry
this bug. Both synthesizers now have an explicit regression test naming
the placeholder.

* 🧯 fix: Treat a describeIntent List as a Selection Policy, Not a Filter

Two build/behavior defects from the previous commits.

**Narrowing did not actually narrow.** Omitting a tool from the
synthesized options is not the same as opting it out, because intent has
two default-on paths background does not: `isIntentOptedIn` treats every
NATIVE_INTENT_TOOL_NAMES member as enabled when it finds no entry, and
`sanitizeIntentLabels` keeps an SDK-native label unless it sees an
explicit `describe_intent: false`. So `describeIntent: ['web_search']`
still labelled `set_memory`, and an empty list — the most explicit way to
say "none" — disabled nothing at all. A list is now a selection policy:
selected eligible tools get true, unselected get an explicit false.
Ineligible tools still get no entry at all.

Background needs no equivalent change: it opts in on
`run_in_background === true` only, with no default-on set, so omission
there genuinely means off.

**Fixed the CI build break.** `MCP_ALL_PLACEHOLDER_PREFIX` was exported
without a type annotation, which `tsc --noEmit` accepts but the package
build rejects under `--isolatedDeclarations` (TS9010) — the same reason
`mcpToolPattern` beside it is annotated `: RegExp`. Verified with an
actual `npm run build` this time, not just a typecheck.

* 🧯 fix: Propagate Intent Selection Through Capability Marker Expansion

A model spec's `tools` carries capability MARKERS, not the definition
names initialization actually registers, so an option recorded under a
marker never matches the tool it becomes — the same silent no-op as an
`mcp_all` placeholder entry.

Harmless for an opt-IN (the tool keeps its default) but not for the
opt-OUTs the previous commit introduced: `memory` becomes `set_memory` +
`delete_memory`, both default-on natives, and `execute_code` becomes
`bash_tool`, which carries an SDK-native label that survives unless
sanitize sees an explicit false. So `describeIntent: ['web_search']` on a
spec with `memory: true` still labelled both memory tools, and `[]`
disabled neither.

`expandIntentToolOptions` propagates a marker's value to the names it
expands into, mirroring `expandCodeToolOptions` in background.ts which
solves the same problem for the code marker. Applied at APPLY time rather
than synthesis, so hand-edited saved agents that key options by marker
benefit too, not just synthesized specs. An explicit per-tool entry always
wins — expansion only fills names the caller did not already decide — and
it runs in both `applyIntentLabels` and `sanitizeIntentLabels`, since the
SDK-native strip reads the same options.

Verified with an actual package build, not just a typecheck.

* ♻️ refactor: Resolve Spec Tool Selections Against Final Definitions

Three review rounds hit the same root cause from three directions: a
per-tool option synthesized at load time is keyed by names that may not
exist at injection time. Spec tools carry capability markers
(execute_code, memory), skills never reach the tools array at all, and
lazy MCP servers expand after synthesis - every mismatch was a silent
no-op, and each fix added another hand-maintained marker map that the
next case leaked past.

This removes the name-space gap instead of bridging it per case:

- Synthesis records the selection as a policy: a wildcard '*' entry
  carries the default (true for "every tool", false for "only the named
  ones") and listed names are recorded verbatim. No load-time
  enumeration, eligibility checks, or placeholder special-casing.
- Resolution happens at injection time, per final definition:
  explicit name -> capability marker projection -> wildcard. The
  projection maps a marker onto the names its registration actually
  produced this run - the registrars report their own tool names and
  initializeAgent accumulates them - so the mapping cannot drift from
  what gets registered.
- The unmatched-name diagnosis moves to the apply passes, the one place
  the real definitions are known, so a selection naming a marker whose
  runtime expansion is entirely ineligible (runInBackground: ['memory'])
  is now warned about instead of recorded as a dead success.

Covers all four round-3 findings: code opt-outs now reach
create_file/edit_file/read_file, skill definitions are governed by
narrowing selections, the memory marker is rejected and diagnosed for
backgrounding, and the mcp_all placeholder predicate is no longer
consulted for selections at all (its one remaining consumer is the
definitions loader that defines the convention).

* 🧯 fix: Reject Dead Ask-Tool Selections and the Reserved Wildcard

Two review findings on the selection policy, both fixed at the point
where they are knowably invalid:

- ask_user_question joins EXCLUDED_INTENT_TOOL_NAMES: createRun strips
  its provisional definition and rebuilds the graph tool from its own
  Zod schema, so definition-level injection never reaches the model. A
  describeIntent selection naming it now warns as ineligible instead of
  crediting a label that gets discarded. Real intent support for the
  ask tool lands with the HITL slice via the interrupt payload.

- A literal '*' in a describeIntent/runInBackground list is dropped
  with a warning at synthesis: it would overwrite the wildcard opt-out
  default and silently enable the capability for every eligible tool
  instead of selecting one named tool. The wildcard is reserved for the
  internal policy; boolean true is the supported way to cover
  everything.
2026-07-30 13:32:04 -04:00
Danny Avila
8af6414e13
🪟 fix: Surface MCP Initialization Errors (#14529) 2026-07-30 13:22:11 -04:00
Danny Avila
7bb6651883
🛑 feat: Preemptive Steer - Backend Interrupt & Steer (#14518)
* 🛑 feat: Preemptive Steer — server half (Interrupt & Steer, PR 2 of 3)

Lets the steer route ask the generating replica to seal its live model
stream at the next provider-safe boundary instead of waiting for a tool
step. The run is never aborted, job status never changes, the partial
answer is kept, and generation resumes in the same assistant message
after the injected steer. Consumes the SDK seam in @librechat/agents
(danny-avila/agents#335, #346).

Transport: IEventTransport gains a fenced emitPreempt/onPreempt pair
beside abort. RedisEventTransport fans PREEMPT out on the SAME events
channel and subscription (no new connection, key, or subscribe call);
onPreempt returns a registration-scoped unsubscribe with the same
replacement-safe state-identity guard onAbort uses. InMemory implements
neither — single-process preempt lives entirely in the runtime set.

Runtime state: RuntimeJobState carries the per-generation request set,
createdAt-fenced and capped at STEER_QUEUE_MAX_DEPTH, plus a bounded
`cleared` tombstone so a late cross-replica arm cannot resurrect a
request whose steer already drained. registerPreemptSubscription
mirrors the abort registration's double fence (runtime identity +
generation createdAt); releaseAbortSubscription retires BOTH listeners
and the armed set, so every terminal path drops preempt state for free.
Public surface: requestPreempt (arm + fenced publish, never a rejection
surface, never touches job status), isPreemptRequested (O(1)
level-triggered poll), noteSteersRemoved (drain/cancel bookkeeping +
fenced clear), clearPreemptRequests (empty-boundary disarm).

One drain body, two boundaries: createSteerDrainHook (PostToolBatch)
and createSteerPreemptBoundaryHook (PreemptBoundary) share
drainAndBuildInjections, so the two injection sites cannot drift — the
SDK's provider-safety argument rests on identical HumanMessage shapes.
The shared body builds injections incrementally under a swallow-all
catch (a mid-loop throw still injects what was applied — those parts
are already persisted), clears preempt requests in finally, and
disarms the generation when a boundary drains nothing.

Request path: POST /chat/steer accepts preempt: true. The guard ladder
is unchanged in order and in every status code. A preempt request is
NEVER a rejection reason — without the capability the steer still
enqueues and the 202 echoes preempt: false. Armed strictly after a
successful enqueue; cancel disarms. The capability is read from the
OWNING replica's recorded `preemptCapable` rather than the route
replica's own SDK probe, so a rolling deploy cannot label a steer
"interrupting" that the older owner will only inject at a tool step.

Durable label: SteerQueueItem.preempt → TPendingSteer.preempt, so a
parked/claimed/replayed chip keeps its wording.

Run wiring: createRun registers the PreemptBoundary hook and threads
RunConfig.preemption, both gated on isSteerPreemptSupported() — a
separate probe from isSteeringSupported(), so the client affordance can
never arm against an SDK that only injects at tool boundaries.
buildSteerWiring builds both hooks from one shared closures object, so
preemption survives HITL pause/resume for free.

Honest finalization: an empty preempt boundary persists and emits with
unfinished: true — the same contract an abort gets — re-marked
explicitly because BaseClient has already saved the row as
unfinished: false by that point.

Not changed: no new job status, store method, Lua, SSE event type,
endpoint, or authorization surface. abortJob, completeJob,
transitionStatus, closeAndDrainSteers, getResumeState, emitChunk,
applySteerPart and the whole abort path are untouched.

Tests: 120 packages/api steering specs (preempt lifecycle, tombstone,
fences, caps, terminal release, both-boundary drain parity,
level-triggered poll, request/cancel arming, owner-capability
degradation) plus 5 in api for buildSteerWiring gating, and 2
Redis-gated cross-replica transport specs.

* 🔒 fix: Codex round 2 — evict tombstones, scope the empty-boundary disarm, honest resumes

All four server findings were fresh consequences of the round-1 fixes,
which is the review doing exactly what it should.

- Tombstone cap refused new entries instead of evicting. Every drained
  or cancelled steer is tombstoned, not just preempting ones, so a
  generation that processed 20 steers exhausted the set and the
  late-arm race resurfaced silently. Now evicts oldest-first (Set
  iteration is insertion-ordered), with the budget named
  PREEMPT_TOMBSTONE_MAX rather than an inline expression.

- The empty-boundary disarm I added in round 1 wiped the generation's
  ENTIRE armed set. A second steer can enqueue and arm between the
  atomic drain returning empty and the disarm running — that arm is
  backed by a live, uninjected queue item and must survive. The drain
  now snapshots the armed ids BEFORE draining
  (getArmedPreemptIds) and clearPreemptRequests takes an explicit id
  list instead of clearing everything.

- HITL resume finalized with a hardcoded unfinished: false. The
  boundary hook is re-registered on resume via buildSteerWiring, so a
  resumed segment can end on an empty preempt boundary exactly like a
  fresh one; finalizeResumedTurn now reads getPreemptStats() and the
  halt reason, matching the normal request path.

- Ownership moves on resume, so the job's recorded preemptCapable must
  describe the replica that will actually generate. Refreshed before
  resumeCompletion; a job created on a capable replica that resumes on
  an older one during a rolling deploy no longer acknowledges steers as
  interrupting.

Tests: +3 (scoped disarm sparing a post-snapshot arm, oldest-first
tombstone eviction, id-list disarm). 122 packages/api steering specs
green.

* 🚨 fix: Codex round 3 — deserialize preemptCapable from Redis (feature was dead under Redis)

The P1 here is the most consequential defect in the whole feature, and
it was introduced by round 1's own capability fix.

- `RedisJobStore.serializeJob` writes booleans generically, so
  `preemptCapable` reached Redis — but `deserializeJob` is an EXPLICIT
  field map and had no line for it. Every `getJob()` therefore dropped
  the flag, `job.metadata.preemptCapable` was always undefined, and
  `handleSteerRequest` computed `preemptArmed: false` unconditionally.
  Interrupt & steer would have silently degraded to ordinary
  tool-boundary steering in EVERY Redis deployment — i.e. the feature
  shipping as a no-op in production while passing every in-memory test.
  Now deserialized, with a round-trip assertion in the metadata spec
  that fails (`Received: undefined`) against the unfixed store.

- The resume capability refresh moved from just-before
  `resumeCompletion` to immediately after `approvals.resolve` claims
  the run. That call already flips the job back to `running`, so the
  steer route accepts requests from that instant; leaving the refresh
  135 lines later (across the whole client reconstruction) left a real
  window where a steer read the PREVIOUS owner's capability. Not the
  fully atomic transition Codex suggested — that reaches into the
  approvals Lua — but it shrinks the window from seconds to one await,
  which is proportionate for a label-accuracy issue.

Refuted: "avoid triggering preemption inside subagents". The premise —
that the run-wide poll can seal a subagent stream — does not hold
against the shipped SDK. Child graphs are constructed with
`subagentScope: true` (SubagentExecutor) and `preemption` is NOT
propagated into child inputs, while `canClaimPreemptSeal()` requires
`!subagentScope && preemption != null`. Both conditions fail
independently, so a subagent can never claim a seal and the boundary
cannot fire with `agentId` set. The `input.agentId != null` guard in
the hook is defensive depth, not the thing standing between us and the
described failure.

140 packages/api specs green.

* 🔁 fix: Codex round 4 — re-arm durable interrupt steers when resume moves owners

- An arm lives only in the owning replica's runtime plus a transient
  pub/sub message, while the steer's `preempt` flag is durable on the
  queue item. A HITL resume landing on a different replica therefore
  started with an empty armed set and a poll stuck false, so an
  interrupt the user had already been ACKed for silently waited for an
  ordinary tool boundary. New `GenerationJobManager.rearmQueuedPreempts`
  rebuilds the armed set by peeking the durable queue (fenced on the
  generation) and re-arming every item flagged `preempt`; resume calls
  it right after claiming. Safe by construction: every item peeked is
  still queued, so no drained steer can be resurrected.

- Capability-refresh failure now logs at error rather than warn, but
  deliberately does NOT fail the resume — see the reply on that thread.

Tests: +2 (rebuild from queue arms only the flagged item and reports
the count; a stale generation arms nothing). 124 packages/api steering
specs green.

* 📡 fix: Codex round 5 — acknowledge only what was actually armed

- A cross-replica arm was fire-and-forget: `emitPreempt` logged its own
  publish failure and `requestPreempt` returned void, so the route
  answered `preempt: true` even when the owner never armed a poll. The
  steer still injected at the next tool boundary, but the chip claimed
  an interrupt that could not happen — and unlike HITL resume, an
  ordinary running generation had no durable reconciliation to recover
  it.

  `emitPreempt` now resolves to the subscriber count and rejects on
  failure; `requestPreempt` is async and returns whether the arm truly
  landed (owned locally, or delivered to at least one subscriber). The
  202 reports THAT rather than what was asked for, so the chip relabels
  to ordinary steering exactly as it does for a capability-degraded
  deployment. Errors are swallowed into `false` — an unarmed interrupt
  is a downgrade, never a failed steer.

- The owner capability is re-read immediately before enqueue rather
  than reused from the top of the guard ladder. `checkAgentAccess` and
  file resolution are awaits, so a request can span an entire HITL
  pause/resume that moves ownership to a replica with different
  capability and rewrites that very flag. Only paid for by requests
  that actually asked to interrupt.

Tests: +3 (not-armed when the publish reaches nobody; armed when this
replica owns the generation; a throwing publish downgrades instead of
propagating). 127 packages/api steering specs green.

* 🎯 fix: Codex round 6 — real ownership, confirmed disarms, and a CI regression of my own

Three review findings plus three CI failures the round-5 commit caused.

Review:
- Ownership came from `runtimeState`, which a cross-replica `getJob`
  populates with a FACADE runtime on any replica that merely read the
  job. Matching `createdAt` therefore proved only "we looked at this
  job", so a non-owner could arm nothing and report success. Ownership
  now comes from `ownedJobs`, the actual owner map.
- `armPreemptIds` returns how many ids it accepted, and a local arm is
  only reported as armed when one was. A tombstoned id (its steer
  drained at an ordinary boundary mid-request) no longer answers
  `preempt: true` for an interrupt that cannot happen.
- The cancel disarm is awaited. A dropped clear is worse than a dropped
  arm: the owner keeps a level-triggered request for a steer that no
  longer exists, seals its next chunk and truncates an unrelated
  answer. The boundary drain's own call stays non-blocking — there the
  owner is local, so the disarm is already effective and awaiting the
  informational publish would only delay injection.
- Subscriber count is NOT read as proof of owner receipt: the count
  includes this replica's own facade subscription. A successful publish
  reports armed, a rejected one does not. Documented rather than
  papered over — see the acknowledgement-semantics note on the PR.

CI regressions from round 5, all mine:
- `registerPreemptSubscription` was AWAITED at both runtime-init sites,
  so job creation blocked on a second Redis channel subscription and
  hung when that subscribe was slow. Abort is awaited because a missed
  abort strands a run; a missed preempt only degrades that steer to the
  next tool boundary, so it now registers without gating createJob.
- Two api specs mocked `@librechat/api` without the newly imported
  `isSteerPreemptSupported`, so the call threw before createJob; and one
  exact-match assertion needed the new `preemptCapable` metadata field.
- My own Redis integration spec asserted arm-before-clear ordering,
  which two publishes carry no guarantee of — the receiving tombstone
  exists precisely because of that. Now asserts delivery and payload
  fidelity, order-independent.

158 packages/api specs, 27 api specs green.

* 🧭 fix: Codex round 7 — settle the acknowledgement semantics (Option A)

Round 7's second finding is the incoherence I flagged on the PR: the
route persisted `preempt: true` on the durable queue item while
returning `preempt: false` when delivery could not be confirmed. Those
two then disagreed, and `rearmQueuedPreempts` trusts the DURABLE one —
so a resumed owner would honour an interrupt the client had explicitly
been told degraded to ordinary steering.

Rather than patch the disagreement, this settles the meaning:

`preempt` in the 202 means "queued as an interrupt request", NOT "a
seal is guaranteed". It mirrors `SteerQueueItem.preempt` exactly, so
the response, the durable record, and the resume-time re-arm can never
disagree. The gates that ARE knowable stay — the owner's recorded
capability and a successful enqueue. Everything past that degrades to
the documented fallback of injecting at the next tool boundary.

A route cannot synchronously know whether another replica will seal:
proving it needs a correlated request/response over pub-sub, and even
that only proves the owner heard, not that it is still streaming when
the arm lands. Four rounds of tightening this boolean each surfaced a
narrower case; the sequence does not converge, so the invariant is now
"the flag describes the durable decision" and an unconfirmed arm logs a
warning instead of rewriting the answer.

Also from this round: a failed disarm publish is retried once and its
outcome reported. `handleSteerCancel` keeps `removed: true` — the steer
really did leave the queue, and saying otherwise would make the client
re-show a chip for a steer that can never arrive — and adds
`disarmed: false` so the residual risk is visible rather than swallowed.
Damage stays bounded regardless: the empty-boundary self-clear disarms
the generation after a single seal.

Tests: +1 pinning the response/durable-flag invariant. 159
packages/api specs green.

* 🧹 fix: Codex round 8 — remove the unverifiable disarm signal

Round 8 found the same over-promise on the disarm side that round 7
corrected on the arm side, so this applies the same answer rather than
patching around it.

The `disarmed: false` field added in round 7 was both unreliable and
unused: a resolved publish is not proof the owner heard it (the
delivery count includes this replica's own facade subscription), and it
was never threaded into `CancelSteerResponse` or read by any client. A
signal that claims a certainty the transport cannot provide is worse
than no signal — it invites callers to trust it.

Removed from the response. The retry stays, because it genuinely
reduces the failure rate, and `noteSteersRemoved` still returns whether
the publish succeeded FOR LOGGING, now documented explicitly as
"published without error", not "the owner disarmed".

Disarm is best effort with a bounded, self-healing failure: if the
clear is lost the owner seals once, the empty-boundary self-clear
disarms the generation, and the turn is persisted `unfinished: true`
rather than silently truncated. Tightening that further needs a
correlated request/response over pub-sub with a timeout — noted on the
PR as the deliberate boundary of this design rather than an oversight.

130 packages/api steering specs green.

* 🧽 fix: Codex round 9 — spend snapshot arms on nonempty drains too

The round-6 scoping fix only cleared the pre-drain snapshot when the
drain came back EMPTY. On a nonempty drain the `finally` cleared just
the drained ids, so a stale arm — typically a cancel whose
cross-replica clear was lost — survived the boundary. It would then
immediately seal the continuation meant to answer the steer that had
just been injected, and land on an empty boundary as
`preempt_incomplete`: the interrupt appears to work, and the answer to
it is truncated.

A boundary that runs has spent its seal, so everything armed at
snapshot time is spent whether or not it came back from the drain. The
`finally` now clears the union of the snapshot and the drained ids.
Arms that land AFTER the snapshot are still spared — their queue items
are live and uninjected, which is the property round 6 added.

Also fixes an api-workspace CI failure of mine: `resume.spec.js` stubs
`GenerationJobManager` wholesale, and the round-3/4 resume work added
two calls (`updateMetadata`, `rearmQueuedPreempts`) the stub did not
define, so 34 specs threw. Stub extended.

Tests: +2 (a nonempty drain clears a stale snapshot arm; a nonempty
drain spares an arm that landed mid-drain). Counterfactually verified —
the stale-arm spec fails against the unfixed drain. 132 packages/api
specs, 60 resume specs green.

* fix: never let a failed preempt subscription reject into the void

registerPreemptSubscription is called detached at both sites, so a
rejected Redis SUBSCRIBE became an unhandled rejection — process-fatal
under Node's default --unhandled-rejections=throw. The comment already
promised this path merely degrades steering; it now does.

Swallowed and logged inside the registration rather than at each call
site, so a future third caller cannot reintroduce the trap. Losing the
channel costs this generation's cross-replica preempts, not the server:
same-replica arming is runtime state and still works, and remote arms
fall back to the next tool boundary.

Verified counterfactually — the new spec surfaces SUBSCRIBE failed as an
unhandled rejection against the unfixed registration.

* docs: state the real blast radius of a failed preempt subscription

LibreChat's own entrypoints install a global unhandledRejection handler
that logs and keeps serving, so the escaping rejection this guards was
never fatal to this server — only to another consumer of @librechat/api
that installs no handler. The fix stands either way; the comment just
should not overstate what it prevents.

* test: cover the cross-replica preempt hop with two manager instances

Every other preempt test runs against a single manager, so the hop that
actually carries an interrupt in production had no coverage: the steer POST
lands on whichever replica the balancer picks, which is usually not the one
generating. Non-owner publishes, owner arms, owner's level-triggered poll
flips — none of that was exercised end to end.

Two GenerationJobManagerClass instances are a faithful replica pair here.
runtimeState and ownedJobs are private instance fields, there is no
module-level mutable state between them, and createStreamServices duplicates
a dedicated subscriber connection per call, so separate OS processes would
exercise the same objects over the same Redis.

Both assertions verified counterfactually against real Redis:
- Deleting the preemptCapable deserialization in RedisJobStore fails this
  with 'Expected: true, Received: undefined' — the exact P1 that shipped past
  every in-memory test and would have made the feature a silent no-op on
  every Redis deployment.
- Dropping the non-owner arm publish fails it with 'Received: false'.

* test: remove the fixed sleeps and vacuity from the cross-replica preempt test

Codex round 11, both findings, both on the test I added last commit.

P2 — the 300ms waits were load-bearing. Redis pub/sub never replays and the
owner's SUBSCRIBE is detached, so on a slow CI worker the publish could land
before anyone was listening and the test would fail against correct code.
Now it republishes until the owner's state converges, which is safe because
arms and clears are idempotent set writes keyed by steerId. Side effect: the
tests got ~10x faster (85ms/57ms vs 929ms/606ms) since they finish on
delivery rather than on a timer.

P3 — afterEach destroyed only the transports, leaving each manager alive in
its own cleanup-interval closure, still working against a dead transport.
Now tracks the managers and awaits destroy(), which disposes the job store
and its timer too. Matches how the rest of this file cleans up.

Fixing the sleeps exposed a third problem codex did not flag: the stale-arm
test could pass vacuously, because an undelivered arm and a fenced one look
identical. It now brackets the stale publish between two control arms — the
first proves the owner is listening before the stale one is sent, the second
proves it has had its chance to arrive.

Verified counterfactually against real Redis, and stable over 5 runs:
- dropping the preemptCapable deserialization fails with 'Received: undefined'
- dropping the non-owner arm publish times out both tests
- removing the generation fence fails the stale test with
  ["control-before", "steer-stale", "control-after"] — which also confirms
  the bracketing orders as intended rather than by luck

* fix: gate interrupt on the OWNER's capability alone, not the route's

Codex round 12. The comment above this gate already said 'the OWNER's
recorded capability, not this replica's probe' — and then the code ANDed in
isSteerPreemptSupported(), which is exactly this replica's probe. The
contradiction dates to the original commit; round 6 made the gate
owner-scoped and wrote that comment without removing the local conjunct.

The route never seals. It enqueues and publishes an arm, neither of which
touches the SDK, so during a rolling deploy a steer landing on an
un-upgraded replica silently lost its interrupt even though the owner could
seal. When the route IS the owner the probe is redundant anyway: the flag it
would consult is the one this process wrote at createJob.

The real degradation path is unchanged and still tested — an owner that
recorded no capability relabels to an ordinary steer. The test that pinned
the local probe asserted an impossible same-replica state (capable metadata
plus an incapable local SDK, when the metadata is written from that probe);
it now pins the mixed-SDK direction instead, and fails with
'Expected: true, Received: false' if the probe is put back.

* fix: reconcile arms at handover, and stop holding the 202 on a publish

Codex round 13, two of three findings.

P2 — rearmQueuedPreempts only ever ADDED. A replica that merely read the job
still installs a facade runtime and subscribes, so it can accept an arm and
then miss the best-effort clear that follows the drain. HITL resume promotes
that facade to owner, the union keeps the orphan, and the first resumed
stream seals on a steer no longer in the queue, drains nothing, and
truncates the resumed answer as preempt_incomplete. acquireResumedJobOwnership
only sets ownedJobs, so nothing else was clearing it. The durable queue is
the sole authority at a handover: arms it does not back are now disarmed and
tombstoned, so an in-flight publish cannot revive them either.

Worth recording that my own independent review raised this and my verifier
refuted it. Codex found it separately; two reviewers converging should have
outweighed one refutation.

P2 — the route awaited the arm publish before answering. The 202 reports
capability, not delivery, so the await could not change the response; it only
exposed the caller to Redis latency after the queue item was already durable.
A client that times out and retries mints a second steer while the first
stays queued, injecting the same instruction twice, whereas a lost publish
merely takes the tool-boundary fallback. Detached, with both outcomes logged.

All three tests verified counterfactually: union-only rearm fails the two new
handover specs, and re-awaiting the publish hangs the stalled-publish spec
until jest kills it.

* fix: snapshot arms before reading the queue at handover

Codex round 14 — a regression from my own round-13 fix, and a worse failure
than the one it corrected.

Round 13 read the durable queue first, then tombstoned any armed id the
snapshot did not back. But approvals.resolve reopens steering before
reconciliation runs, so another replica can commit a preempt steer and
publish its arm while the peek is in flight. That arm is then present locally
but absent from a snapshot taken before the steer existed, so a LIVE
interrupt the route already acknowledged got dropped — and tombstoned, which
blocks the re-arm, making it unrecoverable rather than merely late.

Fixed by inverting the two reads rather than by locking or paying a second
round trip. A steer is durably enqueued BEFORE its arm is published, so any
id in an arms-first snapshot was already queued when it was armed, and the
later peek must observe it unless it has since drained — which is exactly the
orphan this reconciliation exists to drop. Arms landing after the snapshot
are simply not candidates.

Also re-checks runtime identity across the await, since the generation can be
replaced while the queue read is in flight.

New spec injects a steer + arm during the peek and verifies it survives;
against the round-13 ordering it fails with Received array: [].

* fix: bound the cancel disarm wait and fence enqueue to its generation

Codex round 15.

P2 — the cancel awaited its disarm publish unbounded. ioredis queues
commands during an outage rather than rejecting, so that await could hang for
the length of the outage with the steer ALREADY durably cancelled; a client
that gives up then treats the cancel as failed and restores a chip for a
steer that can never produce an applied event. Every successful cancel
publishes, so ordinary steers were exposed too, not only preemptive ones.
Now bounded at 1s, with the publish continuing behind it — its retry and
logging are unchanged, it is just no longer in front of the response. This is
the sibling of round 13's arm-publish finding; I fixed one path and left this
one.

P3 — enqueue was not fenced to the generation the capability decision was
made against. The access checks, file resolution and owner re-read are all
awaits, so the run can be replaced before the enqueue: the item then lands on
the REPLACEMENT queue while the durable preempt flag and the arm still name
the previous epoch, the arm is fenced out at the owner, and the 202 promises
an interrupt that cannot happen. enqueueSteer now takes an expected
generation, mirroring drain/peek, and the Redis path enforces it inside
STEER_ENQUEUE_LUA so the check is atomic with the push rather than racing it.

All three new specs verified counterfactually, including the Lua guard
against real Redis (removing it returns 1 where -1 is required).

* fix: fence the steer to its authorized generation, bound resume setup, keep preempt when Redis parks

Codex round 16, all three findings.

P2 — round 15 fenced the enqueue to owner.createdAt, the RE-READ job. Every
guard above it (ownership, tenant, paused-state, agent ACL) ran against the
job read at the top, so if the run was replaced during those awaits the fence
happily accepted the steer into a generation the request was never authorized
against, carrying the wrong agent's metadata. Now rejects on any mismatch
between the validated job and the re-read.

P2 — resume awaited its steering bookkeeping unbounded, after
approvals.resolve had consumed the action and flipped the job to running, and
outside the resume lifecycle's own try/finally. .catch does not fire on a
promise that never settles, which is what ioredis produces during an outage,
so the client times out, its retry gets a 409 for a spent action, and no
cleanup runs. Bounded at 1s with the writes finishing in the background.

P3 — Redis parks leftover steers inside its terminal-transition Lua, which
projects item fields one by one, so preempt was silently dropped and a steer
recovered from /chat/status lost its interrupting label. Added to both
projections.

All three verified counterfactually, two against real Redis. Worth recording
that my first version of the generation-mismatch test was VACUOUS — it faked
a createdAt matching no live job, so the round-15 enqueue fence rejected it
for the wrong reason and the test passed with the guard removed. Rewritten to
replace the run for real; it now fails with 'Expected 404, Received 202'.
2026-07-30 12:36:13 -04:00
Dustin Healy
af795be0c2
🪢 feat: Langfuse Fanout Connection Setting (#14108)
Some checks are pending
Docker Dev Branch Images Build / build (Dockerfile, lc-dev, node) (push) Waiting to run
Docker Dev Branch Images Build / build (Dockerfile.multi, lc-dev-api, api-build) (push) Waiting to run
GitNexus Index / index (push) Waiting to run
GitNexus Index / post-index (push) Blocked by required conditions
* feat: encrypt tenant Langfuse secret in admin config

Add generic per-field secret encryption to the admin config layer: registered
secret paths (langfuse.secretKey) are encrypted with encryptV3 on write and a
non-secret fingerprint companion is stored. Admin config reads (base + per
principal) redact registered secrets so they are never returned; the fingerprint
is kept so the UI can show which key is configured.

The Langfuse fanout read path decrypts the tenant secret before export. Adds
secretKeyFingerprint to langfuseConfigSchema and tests for the encrypt/redact
policy.

* fix(api): secure admin config secret handling

* fix(api): preserve encrypted langfuse config secrets

* fix(api): couple config secret fingerprint deletion

* fix(api): read langfuse fanout collector url from env

* fix(api): display langfuse secret key hint

* fix(api): remove langfuse secret fingerprint breadcrumbs

* fix(api): use langfuse destination keys for tenant config

* fix(api): remove langfuse config compatibility fallbacks

* refactor(api): simplify langfuse secret helpers

* refactor(api): simplify langfuse config secret handling

* feat: in-app Langfuse connection settings panel

Add a discoverable, admin-gated Langfuse connection panel inside LibreChat
Settings (Dify-style): enable toggle, host, public key, masked write-only secret,
configured-key fingerprint, and a test-connection action. Backed by a dedicated
/api/admin/langfuse/connection endpoint that encrypts the secret at rest, returns
metadata plus fingerprint on read, and validates credentials. Builds on the
per-field encryption and fanout decrypt from the langfuse-config-encryption branch.

* refactor: align Langfuse secret field to CustomUserVars pattern

Use the established SecretInput plus Set/Unset state pill (com_ui_set/com_ui_unset)
from the MCP CustomUserVars UI for the saved-secret state, instead of a bespoke
masked input.

* fix: drop em dash from saved-secret placeholder

* feat: show loading state on Langfuse test connection button

* feat: gate in-app Langfuse settings on fanout config and admin role

* test: align Langfuse connection spec with SecretInput refactor

* feat(langfuse): refine tenant connection controls

* fix(admin): refine Langfuse connection verification

* fix(langfuse): refine tenant connection settings

* fix(langfuse): simplify export enablement controls

* fix(langfuse): validate tenant export configuration

* fix(langfuse): align startup fanout gate

* fix(admin): time out Langfuse verification

* fix(ui): rename Langfuse connection setting

* fix(admin): enforce Langfuse config capability

* feat(langfuse): require explicit tenant export activation

* feat(langfuse): support single-tenant connection settings

* fix(i18n): remove obsolete integrations label

* fix(langfuse): authenticate ingestion verification

* fix(langfuse): validate public key independently

* fix(langfuse): localize connection errors

* perf(config): skip Langfuse checks for non-admins

* fix(langfuse): preserve trace sampling for feedback

* test(langfuse): fix feedback sampling fixture

* fix(langfuse): align secret preview field

* fix(langfuse): harden connection settings state

* fix(langfuse): preserve trace destination state

* fix(langfuse): enforce tenant-wide routing invariants

* fix(langfuse): preserve verified connection invariants

* fix(langfuse): preserve stable project identity

* fix(langfuse): warm project identity asynchronously

---------

Co-authored-by: Ravi Kumar L <ravi.lazar@clickhouse.com>
Co-authored-by: Danny Avila <danny@librechat.ai>
2026-07-29 18:33:10 -04:00
Danny Avila
cc813f430e
🎯 feat: Tool Intent Label Capability (tool_intents) (#14499)
* 🎯 feat: Tool Intent Label Capability (tool_intents)

Adds the fourth member of the per-tool capability family (defer_loading,
allowed_callers, run_in_background): an admin capability
AgentCapabilities.tool_intents plus a per-tool
tool_options[name].describe_intent flag. Opted-in tools get an optional
intent string injected as the FIRST property of their schema — one
model-authored sentence per call, streamed to the client as the call's
live status label (args already reach the client verbatim, so no new
event plumbing). Native host tools (web_search, create_file/edit_file,
set_memory/delete_memory, ask_user_question) default on while the
capability is enabled; explicit false opts out. SDK-native intent
schemas (@librechat/agents coding suite) are recognized and left alone.

- packages/api/src/agents/intent.ts: structural sibling of
  background.ts — first-key non-mutating injection with registry
  parity (covers deferred/tool_search discovery), eligibility and
  PTC-only skips, arg read/strip helpers, self-spawn strip for defs and
  registry, ephemeral/model-spec synthesis with a tool_options merge so
  the background and intent toggles compose.
- handlers.ts: intent runs BEFORE background injection so the label
  stays the first streamed key when a tool carries both (pinned by
  test); the arg is stripped before invocation unless the tool's own
  schema declares it, on both the foreground and background-dispatch
  paths; PTC target schemas are sanitized like background's.
- Capability plumbing through all four routes (endpoint initialize,
  openai + responses controllers, the exported OpenAI-compatible
  service) plus handoff discovery and added-convo agents, and the
  intentToolNames execution channel via configurable.
- describe_intent on toolOptionsSchema (all three written-out Zod
  annotations), ToolOptions, TEphemeralAgent, TModelSpec (+ zod), and
  data-schemas doc comments (tool_options is Mixed — no migration).
- intent.spec.ts: 28 tests cloned from background.spec.ts structure,
  including the intent+background key-order composition.

* 🧯 fix: Codex Review — Opt-Out Strips SDK-Native Intent, Skip mcp_all Placeholders

- An explicit describe_intent: false now REMOVES an SDK-native intent
  property from the definition and registry entry, so the per-tool
  opt-out actually disables the arg's token cost for tools like
  web_search that carry the schema natively (SDK bodies tolerate its
  absence). Previously the early return left the property in place.
- synthesizeIntentToolOptions skips lazily-expanded mcp_all
  placeholders instead of recording options under names that
  applyIntentLabels' exact-name matching can never match, and documents
  the limitation (parity with synthesizeBackgroundToolOptions).

The P1 about the client not rendering the label is the documented
slicing: the UI streaming-label PR follows once #14391's ToolCallGroup
changes merge — args already reach the client, so that slice is purely
rendering.

* 🧯 fix: Codex Re-Review — Label Marker Guard, Capability Kill Switch, Late Defs, Service Threading

- removeIntentParam is now marker-guarded (the label contract's opening
  instruction discriminates it), so an MCP/action tool's own business
  `intent` parameter is never stripped by an opt-out or the disabled
  path — previously an explicit false could remove a real, possibly
  required argument.
- New sanitizeIntentLabels pass runs AFTER every registration step
  (the skill catalog appends its SDK definition post-injection): with
  tool_intents disabled it strips SDK-native intent labels from all
  definitions and registry entries, making the capability a real kill
  switch over their token cost; with it enabled it enforces explicit
  per-tool opt-outs on late-registered definitions.
- ask_user_question removed from the native default-on set: its graph
  tool is rebuilt in run.ts from its own Zod schema (also the HITL
  card's wire shape), so definition-level injection never reached the
  model. Its intent support lands with the HITL slice, which threads
  the label into the interrupt payload deliberately.
- The exported OpenAI-compatible service now threads intentToolNames
  into the run configurable, so the executor's PTC path can strip
  host-injected intent schemas on that route like the in-repo
  controllers do.

* 🧯 fix: Codex Round 2 — Post-Skill Injection, PTC Native Strip, Service Boundary, Honest Docs

- Intent injection now runs LAST in initializeAgent, after the skill
  catalog — which both appends its own definition and REPLACES upgraded
  ones (skill-aware read_file), clobbering an earlier injection while
  intentToolNames still listed the tool. Injection PREPENDS while
  background APPENDS, so intent stays the first schema property under
  the new ordering (pinned by a reverse-order composition test).
- The PTC target-schema strip is now marker-guarded strip-ALL: SDK-
  native intent labels (which are deliberately never in intentToolNames)
  are removed from sandbox-advertised schemas alongside host-injected
  ones; business intent params survive.
- toolIntentsAvailable on the exported service documents the loader
  boundary: a custom LoadToolsFn returning only structured instances
  bypasses definition/registry injection and sanitize by construction.
- librechat.example.yaml describes tool_intents as backend groundwork
  with UI rendering in an upcoming release rather than promising a live
  label today.

* 📦 chore: bump `@librechat/agents` to v3.3.6

Brings in the SDK half of tool intent labels (danny-avila/agents#347,
#349): intent-first schemas on the coding suite across all three
engines, plus web_search / subagent / skill / tool_search, and the
outcome / outcome_patch result channel.

Activates three host paths that were inert while no SDK tool shipped an
`intent` property — verified against the real 3.3.6 schemas:
- capability OFF now strips SDK-native labels (a real admin kill switch)
- explicit `describe_intent: false` removes them per tool
- host injection stays idempotent against an SDK schema, keeping
  `intent` first and never double-injecting

* 🔬 test: Real-Provider Verification for Tool Intent Labels

Adds the live check the unit tests structurally cannot perform: whether a
real model actually authors the injected arg, places it FIRST, and gives
sibling calls to one tool distinct labels. Reuses the existing
real-provider harness (in-memory Mongo, seeded user, credential
neutralizer) and the existing stdio MCP fixture as a genuine tool, so no
external service is involved.

- e2e/config/librechat.real.yaml: adds the e2e-memory MCP server and the
  tool_intents capability, giving the real model something to call. The
  sibling spec asserts only relative token growth, so the extra schemas
  do not perturb it.
- e2e/playwright.config.real.ts: optional Langfuse passthrough. The
  LANGFUSE_* keys match the credential-neutralizer pattern and were being
  blanked before the server booted; they are preserved explicitly, read
  from the invoking environment only, and never written to the generated
  config.
- e2e/specs/real/tool-intents.spec.ts: two facts stored in one turn, both
  through the same tool, asserting intent is the first key of each call
  and that the two labels differ. Args are read from persistence rather
  than the DOM deliberately — no UI renders the label yet, and
  persistence is what a reloaded conversation and the trace both read.

First run against claude-haiku-4-5 produced 'Recording the location of
the OAuth callback router' and 'Recording the location of the MCP
connection pool configuration' — distinct, first-position, no tool name.

Also updates tool-intent-spec.md: records the 3.3.7 removal of the tense
verb map with the evidence that motivated it, the trimmed description and
the marker's role as an API, and a new mandatory requirement that
client-side label rendering be gated on a server-sent signal rather than
the presence of an intent key (a tool's own business 'intent' parameter
would otherwise render as a status label).

* 📦 chore: bump `@librechat/agents` to v3.3.7 and dedupe the intent contract

Picks up danny-avila/agents#353: the tense verb map is gone (a bare
intent now displays unchanged, with completion carried by UI state), the
model-facing description is trimmed 502 → 289 chars, and both the marker
and the description are exported.

Stops redeclaring the SDK contract here:
- INTENT_LABEL_MARKER is imported instead of duplicated as a string
  literal. Every removal path in this module keys on it, and a local copy
  that drifted from the SDK's would make them all stop recognizing
  SDK-native labels — failing OPEN, with labels left in schemas and
  per-tool opt-outs silently inert.
- INTENT_DESCRIPTION is imported too, so host-injected tools and
  SDK-native tools present the model with one identical instruction.
  Keeping the old local copy would also have meant host-injected tools
  still paying ~126 tokens per schema while SDK tools paid ~72.

Verified live against real Anthropic after the trim: two sibling calls to
one MCP tool produced 'Storing the OAuth callback router file location'
and 'Storing the MCP connection pool configuration file location' —
first-position and distinct, so the shorter description holds compliance.
2026-07-29 15:40:52 -04:00
Danny Avila
d70cab48fd
🎯 fix: Keep Run Steps and Labels in One Index Space After a Resume Sync (#14516)
Follow-up to #14391, which deliberately left this shared math untouched.

An edited resubmission offsets incoming indices past the prefix the client
retained, because the server indexes only NEW content. A resume sync
invalidates that arrangement twice, and run steps honored neither:

- It REPLACES `initialResponse.content` with the server's completion-local
  snapshot, so the live array stops measuring the retained prefix. Run
  steps derived their offset from that array, so a reconnect that produced
  an empty snapshot silently dropped the offset to zero and wrote over
  retained content.
- When it also replaces the RENDERED content, the prefix is gone entirely
  and server indices are already absolute. Run steps kept adding the
  snapshot's own length on top, writing past the end and leaving holes.

Activity labels already honored both facts (`editPrefixLength` +
`editPrefixClearedRef`), so a batch's tool cards and its header could
resolve in different index spaces: a label overwriting an unrelated part,
or a fill missing its own reservation and leaving the placeholder pending
forever.

Run steps now read the same two inputs. `useStepHandler` takes the
CAPTURED `editPrefixLength` rather than measuring the live array, gated on
a new `editPrefixCleared` flag that the resumable transport — which owns
the sync boundary — stamps onto dispatched submissions. The non-resumable
transport never sets it, so the plain edit path is unchanged.
`calculateContentIndex` now takes the offset directly instead of the
prefix array, so its ±1 trailing-text adjustment cannot diverge from the
offset every other path applies.

Tests (useStepHandler.spec): unedited applies no offset; a plain edit
still offsets; the captured length wins when sync replaced the live array;
a cleared prefix stops offsetting for both run steps and message deltas,
staying at absolute indices. Verified against the pre-fix code — the three
states this PR repairs fail there and pass here.
2026-07-29 15:39:10 -04:00
Danny Avila
becfc5a373
🗝️ feat: Custom Endpoint API Key Encryption via Array Secret Registry (#14510)
Extend the admin-config secret registry (#14509) with array-item secrets,
registering endpoints.custom[*].apiKey: encryptV3 at rest with apiKeyPreview
companions, redaction on admin reads (plaintext-legacy included, with
verbatim-name omit-to-keep preservation, duplicate-identity skip, and
legacy-plaintext self-healing), passthrough for user_provided/${ENV} refs,
strict-payload runtime decryption at getCustomEndpointConfig, the custom
model fetch, and the provider fallback, and rejection of named/indexed/
positional writes beneath the protected array path.
2026-07-29 15:37:46 -04:00
Danny Avila
7b6900d556
🏷️ feat: Activity Groups With Fast-Model Headers (#14391)
* ✨ feat: Activity Groups with Fast-Model Labels

Groups each contiguous block of reasoning + tool calls into a collapsible
unit headed by a fast-model label (claude.ai-style hierarchy), off the
critical path: a PostToolBatch hook claims a live content slot at the
batch boundary (steering index-offset pattern), renders a deterministic
counts phrase instantly, and swaps in the generated label ~1s later while
the next model call streams. Labels are UI-only — stripped before the SDK
formatter and skipped in the legacy formatter — and reach live clients
via a dedicated on_activity_label SSE event (live/replay/pending paths).

Grouping preserves legacy rendering byte-for-byte when no label part is
present. Generation bridges to Run.generateActivityLabel() when the SDK
ships it (session-grouped Langfuse tracing); falls back to a direct call
today. Env-gated: ACTIVITY_LABELS_POC=true, ACTIVITY_LABEL_MODEL.

* 🧷 fix: Address Codex and Copilot Review Findings for Activity Labels

- Settle in-flight label fills (bounded 3s) before finalization on both
  the main and resume paths, so a label resolving during the final batch
  still reaches the durable log and saved message.
- Overlay on_activity_label chunks in RedisJobStore content
  reconstruction (splice path last-wins per index; replay path
  chronological overwrite), matching steer handling.
- Wire activity labels into the HITL resume createRun so post-resume
  batches keep claiming slots.
- Guard against out-of-order publishes: fill() awaits the claim emit
  before emitting the resolved label, and the client applier ignores a
  stale pending placeholder once a resolved label is present.
- Stamp the batch's groupId onto label parts so parallel-column runs
  place them inside their group instead of filtering them out.
- Localize the counts fallback phrase (10 keys, singular/plural) through
  useLocalize across chat rendering and exports.
- Type the hook with Providers/ClientOptions instead of stringly types;
  drop the unknown cast in the spec; add a dedicated rAF retry ref for
  label events with effect cleanup.

* 🛡️ fix: Address Independent Review — Abort, Usage, Lane Context, Redis Test

- Propagate the run abort signal into label generation (both wiring call
  sites; runtime combines host + dispatch signals with the timeout) so a
  user abort cancels in-flight label calls instead of paying to timeout.
- Record label-call usage like titles: the SDK bridge aggregates via
  chainOptions callbacks, the fallback path via a per-generation callback
  factory; both feed recordCollectedUsage under context 'activity-label'.
- Scope block-context capture: reasoning collection stops at the previous
  block's label part and filters by executingAgentId, so consecutive or
  parallel batches can no longer bleed another block's thinking into the
  payload; intent text still scans past labels (persists across batches).
- Forward the effective charLimit to the SDK call so host and SDK prompts
  agree (SDK default aligned to 600 in agents#327).
- Add a Redis integration test proving last-write-wins reconstruction of
  on_activity_label chunks per claimed index.
- Rebased onto main: only the two activity commits replay (the nine
  steering commits belonged to the old base branch), zero conflicts,
  steering suites green.

* 📐 refactor: Move Activity-Label Wiring to TypeScript, Address Codex Round 2

- [P1] Slot claiming, lane stamping, emit ordering, context capture, and
  settle tracking now live in packages/api (createActivityLabelWiring +
  captureActivityBlockContext); client.js is a thin closure wrapper.
- Register the activity-label hook BEFORE the steer drain so a steer
  draining at the same batch boundary cannot flush the tool block and
  orphan the label outside its group.
- Resolve request-based header placeholders in resolveActivityLabelLLM
  (titleConvo parity) so metadata-keyed proxies work on label calls.
- Trim labels centrally before filling so whitespace-only output from
  either generation path keeps the deterministic counts fallback.

* 🧭 fix: Codex Round 3 — Capture Order, Shared Strip, Token Estimator, Hide Filter

- Capture block context BEFORE pushing the label part: the scan stops at
  ACTIVITY_LABEL parts, so post-push capture hit the just-inserted label
  and silently collected no reasoning excerpts (regression test added).
- Share stripActivityLabelParts from packages/api and apply it in the
  Responses and OpenAI-compatible controllers, closing the replay leak
  for entry points still running SDKs without the formatter skip.
- Exclude activity_label parts from the fallback response-token estimator
  (UI-only parts must not inflate no-usage provider billing).
- Keep label parts explicitly under hide_sequential_outputs — they
  summarize exactly the outputs that mode hides.

* 🔁 fix: Codex Round 4 — Resume Gap, Delta Flush, Agent-Scoped Intent, Token Counter

- Synthesize on_activity_label events for labels claimed or filled in the
  snapshot→subscribe window (the publish is fire-and-forget, so Redis-mode
  reconnects missed them). Feature-gated so the default path adds no
  content re-read; the client applier already ignores duplicates.
- Flush queued deltas before applying a label part, matching the pending-
  action and steer appliers — without it the handler read a stale message
  cache and syncStepMessage pushed a pre-delta copy back.
- Skip another agent's tail text when resolving intent, so parallel runs
  cannot seed a label prompt with a sibling agent's narration.
- Exclude activity_label parts from countFormattedMessageTokens (the
  agent-path counter), not just the legacy BaseClient one.

* 🏗️ refactor: Codex Round 5 — Extract Label Host Logic, Report Usage, Icon Strip

- Move provider/model resolution, usage-metadata mapping, and the settle
  loop into packages/api (activityLabels/host.ts); client.js keeps only
  thin delegations, per the repo's TypeScript-implementation convention.
- Fold label usage into the response rollup with an 'activity-label' tag
  (subagent precedent) so metadata.usage and the live cost gauge account
  for it; tagged, so it stays out of PRIMARY usage/context pairing.
- Narrow tool metadata once in ToolCallGroup so THINK parts in a labeled
  block no longer render phantom generic icons in the stacked strip.
- Import the activity-label helpers by deep path in GenerationJobManager:
  the package barrel now reaches provider-config/cache modules that
  import back into the stream layer, and the cycle broke suite loading.

Declined: resetting steerOffsetState before HITL resume — resume builds a
FRESH AgentClient via initializeClient (initialize.js:978), so the offset
is already zero; the seed wrapper alone accounts for pre-pause parts.

* 🚦 fix: Codex Round 6 — Stream Label Usage, Close Late Fills

- Emit an on_token_usage chunk for label calls (sink push alone left the
  live session gauge blind); retained in pendingSubagentEmits so job
  cleanup cannot race the persist, tagged 'activity-label' as before.
- Close the label scope when settle times out: the wiring gates fill() on
  isClosed and the client fires a label-scoped AbortController, so a
  straggling generation can neither mutate a saved response nor emit into
  a job whose runtime is gone. The controller also chains to the run
  signal, so a user abort still cancels label work.

* 🩹 fix: Repair CI — Package Typecheck and Module Mocks

Local runs covered the client tsconfig and jest, but never packages/api's
own tsconfig, so nine type errors in the extracted host module shipped.

- Type host.ts against the real contracts: ServerRequest, EndpointDbMethods,
  AppConfig from @librechat/data-schemas, IUser for createSafeUser, and a
  MaybeAzureConfig view for the azure instance-name probe and configuration.
- Widen resolveConfigHeaders' llmConfig to Partial<RunLLMConfig>: it only
  reads the three provider header carriers, so auxiliary generations with a
  bare ClientOptions can resolve headers without assembling a run config.
  Type-only widening; every existing caller still satisfies it.
- Add stripActivityLabelParts to the @librechat/api mock in the OpenAI and
  Responses controller specs — those mocks enumerate exports, so a new
  import read as undefined and threw before the assertions ran.
- Use the real activity-label helpers in the ToolCallGroup spec's ~/utils
  mock; stubbing them out would hide the header logic under test.

* ⚙️ feat: Configure Activity Labels via librechat.yaml, Drop Env Vars

Replaces the ACTIVITY_LABELS_POC / ACTIVITY_LABEL_MODEL env gate with
per-endpoint settings, following the title options convention rather than
a top-level block — each endpoint picks its own cheap label model.

- Add activity, activityModel, activityEndpoint, activityPrompt,
  activityMaxPerRun, and activityCharLimit to the endpoint schema, and to
  the endpoints.all pick list (enumerated, so 'all:' would otherwise drop
  them silently).
- resolveActivityConfig reads them with title-style precedence:
  endpoints.all > named endpoint > custom endpoint config.
- Model precedence is now activityModel > titleModel > the agent's model.
  activityEndpoint runs labels on another endpoint's credentials, with
  titleConvo's fallback-on-unknown-name behavior.
- Thread activityPrompt/MaxPerRun/CharLimit through the wiring into the
  hook and the SDK bridge; they were hardcoded defaults.
- The resume gap-repair gate keyed on the env var; it now keys on the
  snapshot actually containing label parts, so deployments without the
  feature still perform no extra content read.
- Document the fields in librechat.example.yaml; add host.spec.ts
  covering precedence, custom-endpoint fallback, and opt-out.

* 📝 refactor: Rename Enable Flag to activityLabel, Document Schema Inheritance

- Rename the boolean from `activity` to `activityLabel`, matching the
  titleConvo/titleModel shape: a verb-object toggle whose prefix matches
  its modifiers (activityModel, activityPrompt, ...). `activity: true`
  alone read ambiguously — it could mean tracking or logging activity.
- Document the two endpoint-schema inheritance paths, which behave
  oppositely and are ~900 lines apart:
  * `endpoints.all` omits from baseEndpointSchema, so new options are
    inherited automatically — nothing to maintain.
  * `azureEndpointSchema` enumerates via .pick(), so a new option is
    silently unavailable on Azure endpoints until listed there.
  The activity block now carries a pointer to the Azure caveat.

* 🔍 fix: Address Codex Findings on the Config Rework

- Pass the matched custom-endpoint config into the label gate. Custom
  endpoints live in the `endpoints.custom` ARRAY, so without it every
  custom endpoint resolved as disabled — including the example this PR
  added to librechat.example.yaml.
- Give label usage a unique `runId:seq`. Label usage is billed but never
  appended to `collectedUsage`, so its length was static: every label
  event reused the last primary usage's pair and collided with itself,
  and the client dedupes on exactly that.
- Attach `cost` to label usage when `interface.contextCost` is on;
  aggregateEmittedUsage treats coverage as all-or-nothing, so an event
  without it suppressed the whole response's cost.
- Honor `activityPrompt` on the direct fallback path, not just the SDK
  bridge — it previously always used the built-in instruction.
- Seed the per-response label cap from labels already on the response so
  a HITL resume cannot mint a fresh quota after every approval.
- Reconcile label gaps on resume via a durable per-job `activityLabels`
  flag instead of probing the snapshot: the FIRST label of a run can be
  claimed inside the snapshot->subscribe window, which the old signal
  missed. The flag is read from a job record already fetched there, so
  runs without the feature still add no content read.
- Auto-collapse labeled single-tool groups; one-call batches are common
  in agent runs and rendering them expanded defeats the grouping.

* 🎯 fix: Correct Label Usage Seq, Cross-Endpoint Pricing, Close Scopes

- Give label usage a NEGATIVE seq namespace. The previous fix was wrong:
  seq is a position in `collectedUsage` (push, then emit with the new
  length), so sink-length + array-length still lands on a real position —
  primary emits 1, the label computes 2, the next primary also emits 2.
  Labels have no position at all (billed separately, never appended), so
  they now occupy a namespace positional sequences cannot reach. The
  client key is a string used for Set membership, so the sign is inert.
- Price cross-endpoint labels with the LABEL endpoint's token config:
  resolveActivityLabelModel now returns the resolved endpointTokenConfig,
  and both the streamed cost and recordCollectedUsage use it instead of
  the agent endpoint's rates.
- Make close state per-wiring rather than per-client. A HITL resume
  rebuilds the wiring, and resetting a shared flag re-opened closures from
  the pre-pause segment whose provider call ignored the abort; settle now
  closes every retained scope, past generations included.

* 🎯 fix: Make the Activity Header Say Something the Cards Cannot

The header read "ran 1 command" next to a card already labeled "Code" —
it restated the UI beneath it instead of adding to it. Two causes, both
about content rather than timing:

- A deterministic tool-type tally was the primary display and also fed
  the prompt, so the best case was a tally and the worst case was a
  tally dressed as prose. Removed from the metadata, the prompt, the
  part type, and the client.
- The instruction only ever reached the fallback path. The wiring
  passed a prompt only when  was configured, so the
  preferred SDK path silently used the published package default. The
  wiring now always supplies one and the hook forwards it on both
  paths.

The register is rewritten around what the cards cannot show:
past-tense git-commit-subject, leading with the distinctive noun,
outcome over attempt, tool names and counts and arguments explicitly
forbidden. The batch entries are labeled as reference material so the
model stops transcribing them.

Claiming a slot no longer emits. The slot still reserves its index so
streamed parts never collide, but with nothing to say there is nothing
to render: until a description exists the block looks exactly as it
does without the feature.

* 🧹 fix: Drop the Localize Hook Left Unused by the Counts Removal

* ✅ test: Add Activity-Label e2e Coverage with a Recording Label Server

Activity labels are the one model call a mock run does not already fake:
fake-model.js swaps the GRAPH model via overrideTestModel, while
run.generateActivityLabel() calls the endpoint resolved client options
over HTTP. The custom endpoints already point baseURL at 127.0.0.1:8889,
so serving that port exercises the real path with no production seam.

fake-label-server.js answers it in both JSON and SSE form, records each
prompt, and can inject blank/error responses. Recording is what lets the
spec assert the CONTRACT rather than the rendering: that this repo
register and the tool OUTPUTS actually reach the model. That is the bug
class that produced unusable labels before, and rendered text looks
identical whether or not the instruction arrived.

Labels get a dedicated endpoint (Mock Provider E). A labeled block
auto-collapses even at one tool call, which hides the tool cards other
specs assert on -- enabling this on a shared endpoint broke
steering.spec.ts. Provider D is the unlabeled control.

Request-count assertions are scoped to a per-test token: a 5xx label
response is retried by the provider client, and a retry can land after
the next test has reset the server.

* 🩹 fix: Address Review Findings on Activity-Label Indexing and Pricing

Replay index (P1). Reserving the slot only in server memory left no event
for it, so a cross-instance replay rebuilt content as [tool, hole, later],
compacted the hole away, and the fill for the reserved index landed on the
following part and overwrote it. The claim now publishes the empty,
pending part so the index is real for every consumer, and fill publishes
even when generation returned nothing so the client cannot stay pending.

It stays invisible: an empty label still DELIMITS its batch in
groupSequentialToolCalls but is not attached as the header, so grouping
does not re-shuffle when the text lands and the block renders exactly as
it does with the feature off.

Edited-response index (P1). Edit-and-resubmit replays the kept prefix and
the server indexes only new content, so run steps offset by that prefix.
Labels are claimed in the same space and now take the identical shift;
without it a label could land inside the prefix and overwrite it.

Redis flag. deserializeJob never read activityLabels back, so every Redis
reload left it undefined and resume skipped label gap reconciliation.

Executing agent. RunActivityLabelOptions.agentId selects the executing
agent tracing metadata AND its tool-output redaction policy; omitting it
let a handoff be redacted under the default agent configuration.

Label pricing. An undefined endpointTokenConfig is meaningful for a
built-in label endpoint (priced from the shared table), so the nullish
fallback billed those labels at a custom primary rates. Inherit only when
the label runs on the agent own endpoint.

HITL usage sequence. runId is the response message id and the counter was
instance-local, so a resume restarted at -1 and the client runId:seq
deduper discarded the post-approval label usage. Seeded past the labels
already on the response.

Also distinguishes "cannot serve" (undefined) from "no label" (null) in
the SDK bridge, so a missing run falls back to the direct call instead of
filling the slot empty. Version gating already happens at wiring time via
the sdkCapable prototype probe.

* 🩹 fix: Keep Unfilled Activity Labels Invisible and Unmask Endpoint Settings

Follow-up review round. Publishing the reservation on every batch made two
latent rendering paths reachable on every run, and both are fixed here.

Empty labels no longer change grouping. The previous pass still formed a
tool-group for a textless label, which wrapped even a single tool call and
pulled THINK parts inside it — and since a reservation is published the
moment each batch ends, that applied during every generation and
permanently after a blank or failed fill. An empty label now flushes the
legacy way instead: it still delimits its batch, but the block re-splits
exactly as it renders with the feature off.

Parallel lanes no longer show a blank line. Lanes render raw parts, so an
unfilled label had nothing to draw; empty ones are dropped. Making labels
act as collapsible headers inside lanes is still a separate gap.

Edited responses no longer offset on resume. The sync replaces
initialResponse.content with the server's aggregatedContent, which already
contains the kept prefix AND everything generated since — so its length is
not the prefix length, and indices reconciled from that snapshot are
already absolute. Offsetting again pushed the label past its slot onto a
later part. The shift now applies only to a fresh edited submission.

Activity settings resolve field by field. Selecting one config object
whole meant any endpoints.all block — even one carrying nothing but
headers — shadowed the named or custom endpoint and silently disabled
activity labels everywhere. Global still wins per field.

Adds groupToolCalls coverage for the invisible-while-empty contract, which
is the part most likely to regress: it is normal state on every run, not
an edge case.

* 🔒 fix: Scope Detached Label Writes to Their Generation Epoch

Epoch scoping (P1). Label generation is detached and can outlive the
generation that started it. emitChunk only proves that SOME runtime is
current, not that the caller belongs to it, so an aborted generation's
fill(null) -- and its usage event -- could be attributed to whichever
generation replaced it, landing an index from the abandoned response on
top of the new one. Because an empty label renders nothing, that
overwrote content silently. emitChunk now takes an optional jobCreatedAt
and drops the event when the runtime epoch differs, mirroring the
existing setGraph/setContentParts convention, and both label emitters
pass it.

An abort now CLOSES the label scope instead of only cancelling the call:
the rejected generation still runs its catch and calls fill(null), which
would otherwise emit into a stream the next generation may already own.

Edited-response indexing (P1). The previous pass skipped the prefix
offset on resume, which was the wrong half of the problem: a sync
replaces initialResponse.content with the server's aggregatedContent,
which is completion-local, so after a reconnect its length is not the
kept-prefix length and the offset is wrong -- but it is wrong for run
steps in exactly the same way. Tool cards and the label that heads them
must share one index space; a label shifting differently from its tools
lands on another part. The label path now uses the identical expression
as useStepHandler, with no resume special-case. Correcting the
post-resume prefix length belongs in calculateContentIndex, where it
fixes both at once.

titleModel masking. The activity settings were made per-field last pass,
but the titleModel fallback a few lines below still selected an entire
config object, so a partial endpoints.all (for example one carrying only
headers) hid a named endpoint's titleModel and quietly fell the label
back to the main agent model. Both now read through one shared per-field
helper.

Resume reconciliation no longer depends solely on markActivityLabels,
which is best-effort yet had come to gate correctness: a lost flag write
silently dropped a label. The snapshot is consulted as a fallback.

The exported host type for generateLabel now admits undefined, which is
the documented "cannot serve, fall back to the direct call" signal the
hook keys on -- distinct from null, meaning it ran and produced nothing.

* 🧷 fix: Keep Group Identity Stable and Memoize Label Endpoint Resolution

Group remount. Tool-group identity was keyed on the first part in the
block. An activity label absorbs the block's leading THINK part the moment
its text lands, so the key flipped from tool:<id> to fallback:<scope>:<idx>
mid-run, remounting the group and discarding whatever the user had
expanded. The key now scans for the first tool call, which does not move
when the block re-forms.

Label endpoint resolution is memoized per response. It reads provider
config and can hit the database for user keys, yet nothing it depends on
changes between batches of one run — and it ran twice per batch, once for
generation and once for usage accounting. The promise is cached rather
than the value so concurrent batches share a single in-flight resolution,
and a rejection is evicted so one transient credential failure cannot
disable labels for the rest of the response.

* 🎯 fix: Offset Edited Resubmissions by a Prefix Length That Survives Resume

The server indexes only NEW content for an edited resubmission, so the
client offsets incoming indices by the prefix it retained. That prefix was
read as initialResponse.content.length, which is correct only until a
resume: the sync replaces that array with the server's completion-local
snapshot, whose length is unrelated to the prefix. After a reconnect every
offset was therefore wrong -- run steps and activity labels alike -- and
could write over content the edit kept. For a label the symptom is worse
than a bad position: the fill misses its own reservation, so the pending
placeholder is never resolved.

The prefix length is now captured when the submission is built, while
initialResponse.content still IS the retained prefix, and carried on the
submission as editPrefixLength. calculateContentIndex takes that length
instead of deriving it from an array that a resume may have replaced, so
run steps and labels share one index space by construction rather than by
both happening to read the same field.

Note the prefix is the FULL original content with the edited part
substituted in place (useChatFunctions clones latestMessage.content and
mutates one entry) -- it is not a slice, so the length cannot be inferred
from editedContent.index.

Group identity no longer changes when a label fills. Tool-group keys were
derived from the first part in the block; an activity label absorbs the
leading THINK part when its text lands, flipping the key mid-run and
remounting the group, which discarded the user's expansion state. The key
now scans for the first tool call, which does not move.

Label endpoint resolution is memoized per response. It reads provider
config and can hit the database for user keys, yet ran twice per batch --
once to generate, once for usage accounting -- while nothing it depends on
changes within a run. The promise is cached so concurrent batches share one
in-flight resolution, and rejections are evicted so a transient credential
failure cannot disable labels for the rest of the response.

The resume gap passes for steers and activity labels now share a single
lazy content read instead of each issuing its own. The label pass stays
gated on the run flag with a snapshot fallback: reconciling
unconditionally would also close the residual first-label window, but it
would bill a read to every resume of every run, including deployments with
the feature off -- which the steer pass deliberately avoids. That residual
requires a lost flag write, which shares fate with the content writes the
labels live in.

* 💵 fix: Bill Cross-Endpoint Labels at Their Own Rates

recordCollectedUsage never accepted an endpointTokenConfig, so the value
the activity-label caller passed was dropped and the balance transaction
was written at the primary agent's rates. Only the UI cost honored the
label endpoint, so a custom primary pointing activityEndpoint at another
endpoint showed one price and charged another. The parameter is now
accepted, and an explicit config wins outright over per-agent resolution:
that map is keyed by AGENT, so it cannot describe usage that ran on a
different endpoint.

Group identity is stable for id-less tool calls too. The previous pass
anchored the key to the first tool call ID; where a supported tool call
carries no id the fallback still used the block's first part index, which
shifts when a filled label absorbs the leading THINK part. The fallback now
anchors to the first TOOL entry's index, so only a block containing no
tool call at all keys off parts[0].

markActivityLabels is retried rather than fire-and-forget. It gates resume
gap reconciliation and is a SEPARATE write from the durable label append,
so a single lost write silently drops a label the content itself recorded.
The earlier "shared fate with content writes" reasoning was wrong. One
retry at run setup costs nothing and removes the only realistic way the
gate goes stale, without billing a content read to every resume.

* 🧮 fix: Stop Offsetting Once SYNC Drops the Edited Prefix

The edit offset was applied unconditionally, but whether it is correct
depends on which branch SYNC took. SYNC either preserves the content
already loaded for the response -- which still contains the retained
prefix, so the offset is required -- or replaces it with the server's
aggregatedContent, which is completion-local and indexed from zero, after
which any offset writes past the end of a now shorter array.

That is why the two previous attempts each fixed half of it: skipping the
offset on resume was right for the replace branch, applying it
unconditionally was right for the preserve branch, and neither holds on its
own. The offset now tracks the actual state of the rendered content.

For an activity label the replace branch was worse than a bad position:
the fill landed past its own reservation, so the pending placeholder was
never resolved and the block kept its generic header for the rest of the
run.

Applied to run steps as well, not just labels. useStepHandler reads the
prefix from the same submission and had the same unconditional offset, so
after a mid-session resume of an edited response tool cards were misplaced
too. Normalizing at the dispatch boundary keeps both in ONE index space by
construction: a label that shifted differently from the tools it heads
would land on another part.

Note the reload path was already coherent -- useResumeOnLoad rebuilds the
submission without editedContent or editPrefixLength, giving no offset
against server-supplied content -- so only the mid-session SYNC path was
inconsistent.

* 🧾 fix: Keep Label Accounting Out of the Primary Usage Slot

Label usage no longer owns getStreamUsage(). recordCollectedUsage assigned
its result to this.usage unconditionally, so when the primary provider
reported no usage metadata but the label provider did, BaseClient took the
label's output tokens as the assistant response's authoritative count. The
later primary call returns early on an empty collectedUsage and never
replaced it, so the wrong value stood, the text-based fallback was skipped,
and the real generation went unbilled. Secondary usage is still billed but
no longer writes that slot.

Cross-endpoint pricing keys off an explicit discriminator rather than the
presence of a value. A built-in label endpoint prices from the shared
table, so an undefined endpointTokenConfig is its MEANINGFUL value --
reading that as "no override" fell back to the primary's custom rates and
restored the exact mismatch the previous pass set out to fix. The caller
already knows whether the label ran elsewhere and now says so.

markActivityLabels rejects on failure instead of swallowing it. The flag
gates resume gap reconciliation and the caller retries it, but the internal
catch resolved successfully and made that retry unreachable -- so the two
changes cancelled out and a transient write failure still left the flag
absent.

Late label accounting is suppressed with the same gate as the late fill. A
straggler that outlived the settle timeout still ran its finally block, so
it charged the balance and appended to usageEmitSink after the response had
passed its usage flush and metadata snapshot: a cost the user pays but is
never shown.

The cleared-prefix state is scoped to one generation. It was set on a
resume SYNC that replaced the response and then never reset, so a later
edited resubmission in the same mounted hook dispatched run steps and
labels with no offset against content that still held its retained prefix.
Reconnects pass isResume and keep the state; a new generation clears it.

* 🔑 fix: Key Prefix State to the Stream and Honor current_model for Labels

The cleared-prefix reset keyed on isResume, which skips exactly the case it
was added for: a submission whose POST succeeded server-side but lost its
response is retried, comes back resumed: true, and subscribes in resume
mode even though it is a NEW generation. A previous generation's cleared
state then survived into it, and incoming run steps and labels applied no
offset against content that still held its retained prefix. The state is
now keyed to the stream id, which changes with the generation and stays put
across reconnects of one.

activityModel now honors current_model. The options are documented as
title-shaped and the titleModel fallback already excludes the sentinel, but
the higher-precedence activity override passed the literal string through to
getOptions and the provider, so an endpoint following that convention failed
every label instead of using the agent model.

* 🎯 fix: Key Prefix State to the Generation and Resolve the Run Model

The cleared-prefix state was keyed to the stream id, which never changes
within a conversation: request.js sets streamId = conversationId, so once a
reconnect cleared the state every later edited resubmission in that
conversation dispatched run steps and labels with no offset and could
overwrite the prefix it retained. It is now keyed to the response message
id, the only per-generation identity available here -- minted per
submission and carried through a resume unchanged.

That is the third identity tried for this state. isResume missed the
deduplicated-retry path (a lost response returns resumed: true for a new
generation); the stream id is conversation-scoped. The response id is the
boundary that actually matches a generation.

current_model labels now resolve the model the run is really using.
initializeAgent merges the request's endpointOption override into
model_parameters and the run gives it precedence, so preferring the saved
agent.model could send labels to a different, potentially unavailable or
more expensive model than the conversation is on.

* 🆔 fix: Key Prefix State to the Submission and Keep the Origin Title Model

Editing an assistant response reuses that response's messageId as
editedMessageId, and useChatFunctions carries it onto
initialResponse.messageId -- so re-editing the same response produced two
generations with the same key and the cleared-prefix state survived between
them, leaving run steps and labels with no offset against content the edit
retained. Keyed now to clientRequestId, the per-submission uuid, which is
minted fresh per edit attempt and forwarded unchanged on retries.

That is the fourth key this state has had, and each earlier one failed at a
real boundary: isResume missed the deduplicated-retry path, the stream id is
the conversation id, and the response message id is reused across edits of
one response. clientRequestId is the identity that actually means "this
submission".

The titleModel fallback is read from the ORIGINATING endpoint again, matching
how titleConvo captures its config before switching credentials. Reading it
after an activityEndpoint switch meant an OpenAI endpoint configured with
titleModel claude-haiku and activityEndpoint anthropic fell through to the
OpenAI run model and sent that name to Anthropic, failing every label. The
destination endpoint supplies credentials, not the model choice.

* 🧷 fix: Close the Remaining Edit, Epoch, and Scope Gaps for Labels

SYNC clears the edit prefix on the new-row branch too. When a resumed
edited submission cannot match an existing assistant row, that branch
builds the response straight from the server's completion-local
aggregatedContent, so it holds no retained prefix -- but the reset lived
only in the matched branch, leaving later steps and labels adding an
offset to indices that were already absolute.

Label usage is keyed per GENERATION. Editing one assistant response reuses
its responseMessageId while each fresh generation restarts
activityLabelUsageSeq, so a second edit re-emitted the same runId:seq and
the client discarded the newer usage while its balance transaction was
still written. The key now carries jobCreatedAt, the run's own epoch:
stable across reconnects and HITL resumes, distinct between generations.

The scope is revalidated at commit time. Checking once before the await let
a scope that closed mid-flight still charge the balance after finalization,
while the matching fill saw the closed scope and dropped the label --
billed but never surfaced, the exact outcome the guard exists to prevent.

The titleModel fallback no longer reaches the destination endpoint. With
activityEndpoint set and no titleModel on the originating endpoint, it
picked up the destination's, so changing only the credential target
silently changed the model and its cost. Precedence is activityModel, then
the originating endpoint's titleModel, then the run model; the destination
supplies credentials only.

* ✂️ refactor: Confine the Edit-Prefix Offset to Activity Labels

useStepHandler is now byte-identical to dev again. The resume-aware prefix
offset was applied there too, which was more correct in principle -- the
post-resume prefix length is genuinely wrong for run steps as well -- but it
changed index math that EVERY run step flows through, for every user,
including everyone who never enables activityLabel.

That shared correction needed five revisions in two days (isResume, the
stream id, the response message id, clientRequestId, and the SYNC new-row
branch), each passing the full suite and each failing at a boundary only
review found. Carrying it inside an opt-in feature put every user behind
logic with that track record. It belongs in its own change, with tests that
construct the edit-plus-resume states none of the current suites reach.

The offset now applies only where the label handler places its part, so
this PR cannot alter rendering for anyone with the feature off. The known
consequence is recorded in the description: with activity labels ENABLED,
an edited response that reconnects mid-generation can place its label and
its tool cards in different index spaces. That is a bug for opt-in users
rather than a regression for everyone, and it disappears once the shared
fix lands.

submission.editPrefixLength stays: the label path still needs a prefix
length that survives a SYNC replacing initialResponse.content.

* 🧾 fix: Commit Labels Before Billing and Keep Blank Slots Invisible

Round-nine review (all P2, feature-scoped):

- Billing ordering (client.js:409, runtime.ts): usage accounting ran
  BEFORE the slot commit on both generation paths, so the settlement
  deadline could expire during the balance write — charged, then the
  fill dropped as out-of-scope: billed, never shown. `slot.fill` now
  resolves a commit flag, generators register their accounting via
  `deferUsage`, and the hook runs it only after a committed fill.
- Scope gates (client.js:757): the direct-fallback `collect` omitted
  `scopeOpen`; both paths now gate on the OWNING wiring's scope, so a
  pre-pause straggler cannot bill because the resumed generation's
  scope is still open.
- Blank-label grouping (groupToolCalls.ts:81): a blank slot forced a
  flush, splitting adjacent single-call batches into standalone cards
  where the feature-off path merges them. Blank labels now only mark
  the claim boundary — structurally invisible, while a later filled
  label still cannot claim an earlier batch.
- Stale fill indices (wiring.ts:301): the skill-card unshift and the
  hide-sequential filter reshape contentParts before the finalization
  settle, so an in-flight fill emitted its claim-time index against a
  shifted array. Both completion paths now settle label fills before
  any post-run content reshaping (the finally settle stays as the
  error-path net; the second call sees an empty pending list).
- Bounded serialization (runtime.ts:238): `JSON.stringify` fully
  materialized unbounded tool results to keep 200/600 chars per entry.
  A budget-bounded serializer stops at the limit (which also bounds
  cyclic values) and preserves the exact truncate-with-ellipsis output.

Tests: fill/bill ordering + suppression on dropped fills (runtime.spec),
blank-slot merging and claim boundaries (groupToolCalls.test), bounded
serialization equivalence and giant-output truncation (runtime.spec).

* 🧮 fix: Keep Deferred Label Billing Inside the Settle Window

Self-review follow-up to the billing reorder: deferring usage until
after the commit moved it PAST the fill's resolution, so a settle keyed
on fills alone could let finalization flush the usage sink and snapshot
metadata while the label's billing was still in flight — the usage row
would silently miss the message rollup even on the happy path.

The hook now reports its whole detached task (generate → fill →
deferred usage) via a `trackTask` option, wired to the same settle
tracker as the fills, so finalization waits for billing exactly as it
did when accounting preceded the fill. The task never rejects. Pinned
in runtime.spec: the tracked task resolves only after usage collection.

* 🧰 fix: Harden Label Resolution, Output Bounds, and Cache Billing

Round-ten review (all P2, feature-scoped); the sixth finding is the
documented edited+reconnect index-space limitation, answered on-thread
as deliberately out of scope for this PR.

- Rejected-LLM memoization (runtime.ts): the hook cached a rejected
  `resolveLLM()` promise permanently, failing every later batch and
  silently defeating the host resolver's own rejected-cache eviction.
  The memo now evicts on rejection so the next batch retries.
- `current_model` precedence (host.ts): an explicit
  `activityModel: current_model` resolved to `undefined` and then lost
  to a configured `titleModel`. The sentinel now resolves straight to
  the run model; the title fallback applies only when `activityModel`
  is absent.
- Output bounds (runtime.ts): label text was persisted verbatim; a
  model ignoring the 4–9-word instruction (or steered by injection in
  untrusted tool output) could emit thousands of tokens duplicated
  through SSE, the chunk log, persistence, and the UI.
  `normalizeLabelOutput` keeps the first non-empty line, collapses
  whitespace, and hard-caps at 200 chars on both generation paths.
- Cache-token billing (host.ts, client.js): the usage mapper dropped
  cache fields, vanishing Anthropic cache tokens from billing and
  charging OpenAI cache reads at the full input rate. The mapper now
  normalizes Anthropic/OpenAI/LangChain cache shapes into
  `input_token_details`, and the emit + cost path carries them with the
  label endpoint's `provider` (additive-provider adjustment).
- Usage-type union (runs.ts): `TTokenUsageEvent.usage_type` now
  includes the emitted `activity-label` literal; the lone consumer
  keys on `usage_type != null`, so this is type-level completion.

Tests: sentinel/title/explicit model precedence and all three cache
shapes (host.spec), transient-resolution retry and output normalization
with truncation (runtime.spec), the new usage literal (runs.spec).

* 🪗 fix: Let Settled Labels Collapse Void Tools and Keep the Tail Cursor

Round-eleven review (all P2, client-side). Two fixed; the other two
findings restate documented Known limitations (edited+reconnect run-step
index space; parallel-lane collapsible headers), answered on-thread.

- Void-tool auto-collapse (ToolCallGroup.tsx): `allCompleted` keyed
  solely on output truthiness, so a tool that legitimately returns an
  empty string kept its labeled group expanded forever. A settled,
  filled label is itself a completion proof — the PostToolBatch claim
  only happens after every output in the batch returned — so it now
  satisfies `allCompleted`; pending labels keep the group live.
- Trailing-reservation cursor (ContentParts.tsx): a blank label
  reservation at the content tail renders nothing but still counted as
  the last part, stripping the streaming cursor and last-item
  affordances from the last VISIBLE part until the next delta.
  `lastContentIdx` now walks back past empty label slots.

Tests: labeled void-tool group auto-collapses, pending-label group
stays expanded (ToolCallGroup.test).

* 💳 fix: Price Label Cache Correctly, Honor endpoints.agents, Cancel Every Retry

Round-twelve review: four fixed here; the remaining P1 (move the
client.js bridge into packages/api) is an architecture call answered
on-thread for the maintainer.

- Provider on billed entries (client.js, P1): round ten added cache
  details to label usage entries but not `provider`, and `splitUsage`
  treats an unknown provider as additive — re-adding cache_read and
  cache_creation on top of an input count that already contains them,
  double-charging Anthropic/OpenAI cached label calls while the
  streamed cost (which carried the provider) disagreed. Every mapped
  entry now carries the label endpoint's provider.
- endpoints.agents honored (host.ts, client.js): `initializeAgent`
  rewrites `agent.endpoint` to the backing provider, so activity
  settings under the PUBLIC `agents` endpoint — valid config, inherited
  by `agentsEndpointSchema` — were silently ignored. Field resolution
  is now `all` > public endpoint > backing provider/custom, applied to
  both the enable gate and the model/titleModel resolution.
- E2E_LABEL_PORT reaches the YAML (playwright.config.mock.ts): an
  overridden port moved the fake label server and its health check but
  not the generated config's hard-coded 8889 baseURLs, so readiness
  passed while every label request targeted the wrong port. The
  override is now substituted into the generated copy.
- Every retry frame cancelled (useResumableSSE.ts): concurrent label
  retry chains (reservation + fill per slot) overwrote one rAF handle,
  so cleanup cancelled only the newest chain; the rest ran up to 120
  frames past unmount and could apply a stale label to a replacement
  generation reusing the same response id. Outstanding frame ids now
  live in a Set that cleanup drains.

Tests: public-endpoint gate/precedence/all-above-public (host.spec).

* 🖱️ fix: Keep the Last-Part Cursor in Parallel Lanes Too

Round-thirteen review (single P2): `ParallelContentRenderer` computed
`lastContentIdx` from the unfiltered array, so a trailing blank label
reservation — filtered out of every lane — left NO rendered part
carrying the last-part cursor and running-subagent affordances until
the label filled.

The sequential renderer's walk-back is extracted into a shared
`lastVisibleContentIdx` helper (utils/activityLabels) used by both
`ContentParts` and `ParallelContentRenderer`, so the two index spaces
cannot drift again. Behavior pinned in activityLabels.spec: trailing
blank skipped, consecutive blanks skipped, filled label counts,
label-free content unchanged.

* 🧹 chore: Alias the Retry-Frame Set for the Effect Cleanup Lint Rule

* 📏 fix: Let activityCharLimit Reach Tool Inputs

Round-fifteen review: `activityCharLimit` is documented as the
per-entry truncation for tool input AND output, but `buildPrompt`
hard-coded inputs at 200 characters — so raising the setting could
never surface a distinguishing path, query, or operation that appears
past the first 200 characters of a long argument. Inputs now truncate
at the configured limit alongside outputs; the 200-char constant
remains only for the intent line (renamed INTENT_CHAR_LIMIT to match).
Config fidelity pinned in runtime.spec: a 400-char argument survives a
450 limit and truncates under a 50 limit.

The round's other finding is the fifth restatement of the documented
edited+reconnect index-space limitation, answered on-thread with the
prior four cross-references.

* 🤝 fix: No Labels for Pure Handoff Batches

Round-sixteen review: a PostToolBatch containing only `transfer_to_*`
calls claimed a label slot, but transfer parts are never groupable —
the client flushed the handoff card standalone and the label orphaned
into a stray line after it, restating what the card already says.

Two-sided fix:
- Hook (runtime.ts): a batch whose every entry is a transfer call
  claims nothing — no slot, no model call, no `maxPerRun` consumption.
  Mixed batches still label (the header describes the real work).
- Renderer (groupToolCalls.ts): an orphan label whose `tool_call_ids`
  are all transfer calls is dropped instead of rendered standalone,
  covering content persisted before the hook-side skip.

The round's two P1s are repeats answered on-thread: the packages/api
extraction (maintainer-decided follow-up, recorded in the description)
and the sixth restatement of the edited+reconnect index limitation.

Tests: transfer-only batch claims nothing, mixed batch still claims
(runtime.spec); transfer-only orphan label dropped, real-batch orphan
label still renders (groupToolCalls.test).

* 🎛️ fix: Sanitize Label Client Options and Bound the Batch Prompt

Round-seventeen review: two fixed; the other two findings repeat the
maintainer-decided packages/api extraction (follow-up) and the
edited+reconnect index limitation (seventh instance), answered
on-thread.

- Primary-option strip (host.ts): the label client copied the resolved
  `llmConfig` wholesale, so an endpoint whose defaults enable extended
  thinking or carry model-specific output caps forwarded them to the
  (often cheaper) label model — unsupported options failed every label,
  and supported thinking spent real tokens and the settlement window on
  a 4–9 word header. The copy now strips `omitTitleOptions` keys and
  the `modelKwargs` output caps exactly like the title path, restoring
  the Anthropic `clientOptions` carrier by reference so proxy
  `defaultHeaders` still reach label requests.
- Batch prompt budget (runtime.ts): per-entry truncation left the batch
  dimension unbounded — hundreds of parallel calls could build a prompt
  past the fast model's window. The entries section now has a total
  budget (8k chars, scaling with `activityCharLimit` so a raised limit
  still fits several entries); entries past it are skipped without
  paying their serialization cost, and the list notes how many were
  omitted. The first entry always renders in full.

Tests: option strip with header-carrier survival (host.spec); giant
batch bounded with omission marker, small batch untouched
(runtime.spec).

* 🛡️ fix: Keep SSRF Guards on Label Calls, Skip Mixed Handoff Batches

Round-eighteen review: four fixed; the fifth repeats the
maintainer-decided packages/api extraction (eighth instance), answered
on-thread.

- SSRF-safe carrier (host.ts, P1): the sanitize step restored the
  Anthropic `clientOptions` carrier only when `defaultHeaders` existed,
  but for user-provided base URLs `getLLMConfig` stores the guarded
  Undici dispatcher and `redirect: 'error'` there — dropping it
  reopened DNS-rebinding/redirect paths on label calls to
  user-controlled URLs. The carrier (client CONSTRUCTION options, not
  generation params) is now restored whenever present, same reference.
- Primary maxTokens (host.ts): top-level `maxTokens` is not in
  `omitTitleOptions` and survived the strip; the title path deletes it
  explicitly, and a cap sized for the primary model can be rejected by
  the substitute. Deleted on the copy.
- Bounded keys (runtime.ts): the object branch materialized every key
  via `Object.keys` and quoted oversized keys in full before the budget
  check. Enumeration is now lazy (`for..in` + own-property guard) and
  keys slice to the budget before quoting, like string values.
- Mixed handoff batches (runtime.ts, groupToolCalls.ts): the client
  flushes the block at the transfer card, so a mixed batch's label
  orphaned exactly like a pure one. The hook now skips ANY batch
  containing a transfer call, and the renderer drops orphan labels
  covering one (legacy content).

Tests: carrier survival without headers by same reference, maxTokens
strip (host.spec); mixed batch claims nothing (runtime.spec); mixed
orphan dropped, real-batch orphan kept (groupToolCalls.test).

* 🧢 fix: Cap Label Generation, Order the Flag Persist, Detach Settled Listeners

Round-nineteen review: three fixed; the fourth is the ninth instance of
the edited+reconnect index limitation, answered on-thread.

- Generation cap (host.ts): stripping the primary output caps left
  label calls with NO cap at all — `normalizeLabelOutput` bounds what
  persists, not what the provider generates and bills, so a model
  ignoring the 4–9-word instruction (or steered by injected tool
  output) could emit its provider-default output per batch. The
  sanitize step now installs a 256-token label cap (per provider
  family: `maxOutputTokens` for Google-style wrappers, `maxTokens`
  otherwise), after the filter so the omit set cannot remove it.
- Flag-persist ordering (client.js): the `markActivityLabels` write was
  fire-and-forget, so an immediate cross-replica reconnect could read
  the job between the write and the first claim, see neither flag nor
  snapshot label, and skip gap reconciliation. Label emission now
  awaits the (settled-on-failure) persist chain, making "a label event
  exists" imply "the flag is durable" — the race window is gone; only
  the documented double-write-failure residual remains.
- Listener detach (client.js): each HITL approval cycle's wiring adds a
  `once` abort listener to the shared job signal that only an actual
  abort removes; settled segments now detach theirs in
  `settleActivityLabels`, so long multi-approval runs cannot accumulate
  dead closures toward the listener-limit warning.

Tests: the primary cap is REPLACED by the 256-token label cap
(host.spec).

* 🎯 fix: Route the Label Cap Per Model Family

Round-twenty review: the 256-token label cap set maxTokens
unconditionally, but GPT-5+ rejects max_tokens (the OpenAI builder
routes its cap into modelKwargs.max_completion_tokens /
max_output_tokens) and o-series models reject it with no stable kwargs
alternative — every label on those models would have failed. The cap
now mirrors the builder: modelKwargs for GPT-5+ (responses-API aware),
no cap for o-series (title parity; the 200-char persistence bound
still applies), maxOutputTokens for Google, maxTokens otherwise.
Pinned in host.spec for both reasoning families.

The round's other finding is the tenth instance of the documented
edited+reconnect index limitation, answered on-thread.

* ⏱️ fix: Persist the Label Flag at Run Start, Not on the Emit Path

Round-twenty-one review: two fixed; the other three repeat the
maintainer-decided packages/api extraction, the edited+reconnect index
limitation, and the parallel-lane header limitation — all answered
on-thread with their standing decisions.

- Flag ordering, corrected (client.js): sequencing label emission
  behind the flag persist (previous round) delayed the claim-time
  reservation while the shared index offset had ALREADY shifted
  subsequent SDK chunks — reopening the cross-instance
  hole-compaction overwrite the reservation emit exists to prevent.
  The reservation emits immediately again; instead, run start
  (processStream and resume alike) awaits the settled-on-failure
  persist chain, so the flag is durable before any batch can claim a
  label. Same guarantee, zero latency on the emit path.
- Tail-label cursor (ContentParts.tsx): a filled label at the content
  tail is consumed into the group header rather than listed in
  `group.parts`, so the `isLast` check missed it and nothing held the
  streaming cursor until the next delta. The check now includes
  `labelPart.idx`.

* 🔌 fix: Detach Label Abort Listeners Even Without Claims

A segment with labels enabled can end without a single claim (text-only, or handoff batches, which skip labels); the early return in settleActivityLabels skipped the detach added for HITL listener accumulation. The detach now runs on both paths.

* ⚖️ fix: Make the Commit Flag the Sole Billing Authority

Round-twenty-three review: a committed fill racing a late scope close
(user abort or settle timeout during the durable emit) stayed visible
— the part is mutated and persisted before the close — yet the
deferred accounting's scope gates then skipped the charge: a completed
provider call escaping both the label charge and the primary abort
accounting.

The scope gates on the deferred-usage path are removed; the hook's
commit flag is now the single billing authority in BOTH directions. A
dropped fill never reaches the accounting callback (billed-never-shown
stays impossible), and a committed fill bills regardless of when its
scope closed (shown-never-billed now impossible too). The dead
`scopeOpen` payload threading is removed with it; the
`recordActivityLabelUsage` parameter survives, defaulting open, for
callers that own no commit signal.

The round's other finding is the twelfth instance of the documented
edited+reconnect index limitation, answered on-thread.

* 🧮 feat: Bill Labels by Estimate When Providers Omit Usage

Maintainer decision: follow the title convention rather than leaving
label calls unbilled when a provider returns no usage metadata.

The hook now passes a LAZY estimate thunk with the deferred accounting
on the success path — the EXACT prompt the direct path sent (or the
locally built equivalent for the SDK path: same entries, context,
instruction, truncation contract, and continuity headers) plus the
final normalized label. `recordActivityLabelUsage` invokes it only
when no collected entry carries a real token count, counts both texts
with the shared o200k_base tokenizer, and feeds the synthesized entry
through the SAME pipeline (provider-tagged, streamed event, cost,
balance transaction). Real provider usage always wins when present.
The failure path passes NO estimate: a throw before a response bills
only real collected metadata, never a full phantom prompt.

Tests: the estimate thunk carries the exact invoked prompt and final
label; the failure path defers with no estimate (runtime.spec).

* 💵 fix: Estimate From the Raw Completion, Not the Normalized Label

The fallback estimate counted the normalized label (first line, 200-char cap) while the provider generated and would bill the raw output up to the 256-token generation cap — under-recording verbose replies. The estimate thunk now carries the raw pre-normalization text; the persisted label is unchanged. Pinned with a multi-line reply test.

* 🧾 fix: Commit Label Text Only After the Durable Emit, Estimate the Real SDK Prompt

Round review on the billing work: two fixed; the third is the
fourteenth instance of the edited+reconnect index limitation, answered
on-thread.

- Copy-first fill (wiring.ts): the fill mutated the shared content part
  BEFORE its durable emit, so a failed emit left the label text on
  `contentParts` anyway — persistence could save and display a label no
  client ever received and billing (keyed on the commit flag) never
  charged. The new state is staged on a copy; the shared part mutates
  only after the emit succeeds, so content, delivery, and billing move
  together.
- Real SDK prompt for estimates (client.js): the estimate thunk carried
  this module's locally built prompt, but the SDK path frames entries
  differently — the estimated input count was for a prompt never sent.
  Chain-start callbacks (handleLLMStart/handleChatModelStart) now
  capture the prompt the SDK actually rendered, and the deferred
  accounting substitutes it into the estimate when capture succeeded,
  falling back to the local approximation otherwise.
2026-07-29 14:05:47 -04:00
Dustin Healy
f4e0888f14
🕶 feat: Generalize Admin Config Secret Redaction (#14509)
* feat: generalize admin config secret redaction to a field registry

Replace the single hardcoded langfuse.secretKey handling in the admin
config secrets module with a registry (CONFIG_SECRET_FIELDS) so every
credential-shaped config field is encrypted at rest, redacted on read,
and preserved when omitted on a subsequent write.

Registry covers langfuse.secretKey plus speech tts/stt provider apiKeys,
ocr.apiKey, the webSearch provider apiKeys, and the assistants /
azureAssistants endpoint apiKeys. Fields that conventionally hold
${ENV_VAR} references keep those references as plain, visible values;
literal secrets are always encrypted. langfuse.secretKey behavior
(display companion, always-encrypt, array-section handling) is unchanged.

Wire runtime decryption for the consumers that read these values from
the merged app config: resolveConfigSecret in the speech STT/TTS
services and decryptConfigSecret in the Mistral OCR auth loader. Legacy
plaintext literals and ${ENV_VAR} references continue to resolve.

* feat: add masked display companions for every registered config secret

Every non-langfuse field added to the secret registry was missing the
displayPath that langfuse.secretKey already had, so redacted admin reads
returned nothing for those fields instead of a masked value like
sk-mis...Z789. The registry-driven encrypt/redact/preserve/mutation-path
logic in secrets.ts was already field-agnostic; the only backend fix is
setting displayPath on the other 15 registry entries.

Add the matching optional display<Field> companion to each zod schema
in librechat-data-provider (ocr, speech tts/stt providers, webSearch
providers, the shared assistants/azureAssistants endpoint schema) so the
field is typed for consumers, mirroring langfuse.displaySecretKey. DB
overrides are Mixed-typed, so nothing breaks without this at the storage
layer, but the type is needed for any typed consumer of TCustomConfig.

A display path can never be written as a secret: direct writes to it are
rejected, and an ancestor-object write that includes a spoofed display
value alongside or instead of the real secret is overwritten or dropped,
never encrypted or persisted.

* fix: harden the config secret write path against masking and mixing bugs

getDisplaySecretKey disclosed the entire value for any secret of 10
characters or fewer, since the first-6/last-4 mask overlaps or covers
the whole string at that length (e.g. self-hosted LocalAI tokens).
Short secrets are now fully masked instead.

writeSecretIntoSection/writeDottedSecret encrypted a literal secret's
raw string verbatim, including leading/trailing whitespace, so a
padded paste round-tripped with the whitespace intact and a
whitespace-only value was not treated as empty. Literals are now
trimmed before encrypting and masking.

Both functions also returned early on an env-placeholder value without
clearing the display companion, so replacing a literal secret with
${ENV_VAR} left the previous masked value stale in the stored config,
and a client-supplied display value submitted alongside a placeholder
secret was never overwritten. The placeholder branch now clears the
display companion in both the dotted-patch and object-valued write
paths.

* fix: fail closed in Mistral OCR auth when a stored ciphertext can't decrypt

loadAuthConfig fell back to the raw v3: ciphertext string whenever
decryptConfigSecret returned undefined, so a corrupted or otherwise
undecryptable stored secret was sent to the Mistral API verbatim as
the apiKey instead of triggering the existing env-var fallback.

isEncryptedConfigSecret is now exported so the OCR auth loader can
distinguish "this looks like ciphertext and failed to decrypt" from
"this was never encrypted" and treat only the former as empty,
preserving literal and ${ENV_VAR} values exactly as before.

* fix: omit undecryptable TTS provider headers instead of sending "undefined"

openAIProvider, elevenLabsProvider, and localAIProvider built their
Authorization/xi-api-key headers directly from resolveConfigSecret's
return value, which is undefined on a decrypt failure. That produced a
literal "Bearer undefined" header (or an undefined-valued xi-api-key
header) sent to the provider instead of failing gracefully.

Each provider now resolves the key once and only includes the header
when it's non-empty, matching the pattern already used by
azureOpenAIProvider and STTService's providers.

* fix: strip secret-ancestor arrays at any depth, not just the top level

encryptConfigSecrets/redactConfigSecrets only stripped an array-valued
registered-secret ancestor when it appeared as a literal top-level key
(e.g. a dotted "speech.tts.openai" key). A true nested array at any
depth, e.g. { speech: { tts: { openai: [{ apiKey: "sk-secret" }] } } },
made walkToParent return null and silently skip that field entirely,
so the literal secret was stored unencrypted and returned verbatim to
any reader with section-level read access.

pruneSecretAncestorArrays now walks every registered field's ancestor
chain and deletes any array found at any depth before encryption or
redaction runs, closing the gap for both write and read paths.

* fix: migrate legacy plaintext secrets instead of dropping them on preserve

preserveConfigSecrets only restored an omitted secret when the existing
stored value was already v3-encrypted. Every field this PR newly
registers was previously stored as plaintext with no protection at
all, so any deployment upgrading into this registry has real
credentials sitting in Mongo as plaintext today. The first time an
admin edited an unrelated field in the same section (e.g. mistralModel
next to ocr.apiKey), the omitted plaintext secret failed the
"already encrypted" check and was silently dropped instead of
preserved, breaking the integration.

The existing value is now encrypted in place when it isn't already
ciphertext or an allowed env placeholder, so the credential survives
the edit and gets a computed display companion instead of being lost.

* refactor: derive masked-preview companions as <field>Preview

Replace the display*-prefixed companion names (displaySecretKey,
displayApiKey, displaySerperApiKey, ...) with a uniform <field>Preview
suffix (secretKeyPreview, apiKeyPreview, serperApiKeyPreview, ...) derived
automatically from the registered secret path — registry entries no longer
declare a displayPath, and the name never collides with display-label
config fields like modelDisplayLabel.

Legacy langfuse.displaySecretKey companions (the only shipped instance,
with no released reader) are stripped from writes and reads and migrated
to secretKeyPreview on preserve, so stored documents self-clean.

Also reject MongoDB operator segments ($, $[], $[id]) in admin config
field paths: isValidFieldPath previously accepted them, letting a patch
like webSearch.$[].serperApiKey reach patchConfigFields as a positional
update that bypassed secret-path validation.

* fix: translate legacy displaySecretKey to secretKeyPreview on reads

Redaction previously deleted the legacy companion outright, so the first
admin read of a not-yet-migrated document showed no configured-secret
indication until a later write migrated it. Reads now surface the legacy
value under secretKeyPreview (when no new-name preview exists) while still
stripping the legacy key from the response; stored documents migrate for
real on their next write.

* fix: detect runtime ciphertext by full encryptV3 payload shape

Runtime resolution (resolveConfigSecret, mistral OCR auth) now identifies
decryptable values by the exact v3:<32-hex-iv>:<hex> shape encryptV3
produces instead of the bare v3: prefix, so a legitimate literal credential
that merely starts with v3: (e.g. from YAML, which the admin write path
never encrypts) resolves as a literal instead of failing decryption.
Write-side prefix rejection stays broad as spoof/echo defense.

---------

Co-authored-by: Danny Avila <danny@librechat.ai>
2026-07-29 13:55:45 -04:00
Ravi Kumar L
f4723220cc
🛡️ fix: validate message feedback payloads (#14500)
* fix: validate message feedback payloads

* fix: enforce feedback rating tag consistency
2026-07-28 22:17:30 -04:00
Danny Avila
9c95bf445f
🍂 chore: Prune Deprecated Packages From the Dependency Tree (#14483)
Some checks are pending
Docker Dev Branch Images Build / build (Dockerfile, lc-dev, node) (push) Waiting to run
Docker Dev Branch Images Build / build (Dockerfile.multi, lc-dev-api, api-build) (push) Waiting to run
GitNexus Index / index (push) Waiting to run
GitNexus Index / post-index (push) Blocked by required conditions
Removes three of the eight deprecation warnings emitted on `npm install`.

- Drop `@types/winston` from `packages/api` and `packages/data-provider`.
  The published tarball ships no type declarations at all, so `winston`'s own
  types were already being used. Declare `winston` as a devDependency instead,
  since both packages `import type { Logger } from 'winston'` and were relying
  on hoisting to resolve it.
- Drop `rollup-plugin-postcss` from `packages/client`. It is unreferenced since
  the package moved to tsdown, and pulled in `cssnano -> postcss-svgo -> svgo@2`,
  which is the only consumer of the deprecated `stable`.
- Override `test-exclude` to ^8 so `babel-plugin-istanbul` stops resolving
  `test-exclude@6`, which pins the deprecated `glob@7`.

The remaining five warnings (`ldapjs`, `whatwg-encoding`, `node-domexception`,
and workbox-build's `glob`/`source-map`) are transitive with no non-deprecated
version available upstream.
2026-07-28 09:28:06 -04:00
Danny Avila
728fc1276e
🔒 fix: Bound /files/usage TTL Hold Instead of Clearing It (#14470)
* 🔒 fix: Bound `/files/usage` TTL Hold Instead of Clearing It

`POST /files/usage` marks queued attachments so the 1-hour upload-window
TTL cannot reap them before the client queue drains. It did this by
calling `updateFilesUsage`, which unsets `expiresAt` outright, turning
every touched upload into a permanently retained file.

The client queue is ephemeral browser state, so this also leaks in normal
use: a closed tab or cleared queue leaves nothing referencing the files,
but their TTL is already gone. The same mechanism let an authenticated
user pin arbitrary owned uploads indefinitely, and the route was excluded
from the file limiters, so the touch was entirely unmetered.

Make the operation match its intent, a renewable hold rather than a
release:

- Add `extendFilesTTL`, which pushes `expiresAt` forward by a bounded
  window in a single owner-scoped `updateMany`. Two filter guards keep it
  safe under client-supplied ids: `$exists: true` so an already-released
  file never has a TTL re-added (that would schedule a live file for
  deletion), and `$lt` so a hold only ever moves the deadline later.
  The owner scope is a required argument, so an unscoped call is a no-op
  rather than a cross-user update.
- `handleFilesUsageRequest` now holds for 24h instead of clearing, and no
  longer increments `usage`, since a queue touch is not a send. The real
  release still happens at drain, where `updateFilesUsage` marks the
  files used against an actual message.
- Give `/usage` its own per-user limiter. Keeping it off the upload quota
  was intentional, leaving it unmetered was not.

Abandoned queues are now reaped on schedule, and a replayed touch can only
ever re-assert the same bounded window.

* 🔒 fix: Anchor the `/files/usage` hold to upload time

Codex review on b687922.

The hold derived each new deadline from `Date.now()`, so a caller touching
once a day advanced it by another 24h every time, far below the rate limit.
That left indefinite preservation reachable and made the PR's replay claim
wrong: the window was bounded per call but not in aggregate.

Anchor the deadline to the file's immutable `createdAt` instead of the
request clock. `extendFilesTTL` now takes a lifetime and sets
`expiresAt = max(expiresAt, createdAt + holdMs)` in an aggregation
pipeline, so the target is a fixed point per file and replay is inert
rather than merely bounded. `$max` keeps the widen-only property and the
`expiresAt: {$exists: true}` filter still refuses to resurrect a released
TTL; `createdAt: {$exists: true}` fail-closes when the anchor is absent.

The update runs with `timestamps: false`: a hold is TTL bookkeeping, not a
content write, and bumping `updatedAt` also made every re-touch count as a
modification, hiding whether the deadline actually moved.

Also drop four `.node_modules-*` symlinks that `git add -A` swept in from
an npm install. They pointed at absolute paths on one machine, so every
other checkout got dangling entries. Added the pattern to .gitignore so a
workspace install cannot reintroduce them.

* 🔒 fix: Track the configured approval window in the `/files/usage` hold

Codex review on 9277620.

`endpoints.agents.checkpointer.ttl` is a positive int with no upper bound,
and its docs invite raising it for longer review windows. It drives the
pending-action expiry, so a run can legitimately stay paused past 24h. The
fixed 24h lifetime would then let Mongo reap an attachment while its
approval was still live, and the later queue drain would send a file that
no longer exists.

Replace the fixed constant with `resolveFilesUsageHoldMs`, which adds the
configured approval window to a 24h baseline covering upload, enqueue, and
the run reaching its pause. The route reads the window from the same
`getApprovalTtlMs(checkpointerCfg)` the pending action uses, so the two
stay in lockstep.

The replay bound is unaffected: the window is a per-deployment constant and
the deadline is still `createdAt + holdMs`, so a replayed touch re-asserts
the same instant and `$max` skips the write. Only an operator config change
moves it, never a client.

* 🔒 fix: Renew the `/files/usage` hold across queued runs, under a ceiling

Codex review on 2bd3c52.

The drain sends one queued item per run completion, and each item starts a
run that may itself pause for the full approval window. Since the hold was
taken once at enqueue and pinned to the upload time, an item several places
back could sit through multiple approval windows and lose its attachment
while its chip and the live approval were still there. Another regression
from this PR: the old `$unset` made retention permanent, so deep queues
happened to work.

The queue is unbounded, so no fixed lifetime covers it. Split the hold into
a renewable window and a ceiling:

  expiresAt = max(expiresAt, min(now + renewMs, createdAt + maxLifetimeMs))

`renewMs` covers one run's wait and is granted from now, so a queue that is
still draining re-asserts it at each transition; `useQueueDrain` now marks
the remaining items' files whenever it pops one. `maxLifetimeMs` is
measured from the immutable upload time and clamps every renewal, so
repeated touches converge on a ceiling instead of advancing per call, which
keeps the replay bound from the previous round intact.

This also tightens abandonment: a queue nobody drains now lapses one
`renewMs` after its last touch instead of surviving to the ceiling.

`useQueueDrain`'s spec gained a QueryClientProvider, since the renewal goes
through react-query.

* 🔒 fix: Renew queued holds on a heartbeat, and stop dropping batches

Codex review on f616bed.

Three gaps in the renewal added last commit:

- `collectQueuedFileIds` returned early at the server's 10-id cap, so a
  remainder holding more than one batch renewed only its first message and
  left the rest on their enqueue-time hold. Collect everything and split
  into capped requests instead of truncating.
- A refused `ask()` restores the popped item, but renewal ran before the
  send and covered only the pre-existing remainder. Since the run-end signal
  is already consumed, nothing would touch that item again. Renewal now runs
  after `ask` and includes the restored item.
- A single run can interrupt for approval more than once, each pause running
  to the configured window, so renewing only at drain transitions leaves a
  gap longer than `renewMs` with no renewal in it. The ceiling cannot help
  when nothing renews.

The third is the same structural gap as the previous round along a new axis:
renewal tied to discrete events loses the file whenever two events are
further apart than the hold. Rather than hook each transition, renew on a
30 minute heartbeat while anything is queued, which is far below the
smallest hold (24h) and so covers any single gap regardless of cause.

Still bounded: every renewal is clamped against the file's upload time, so
the ceiling is unchanged. A queue nobody has open emits no heartbeat and
lapses one `renewMs` after its last touch, preserving the abandonment
behaviour.

* 🔒 fix: Cover the pre-migration queue, first tick, and `/usage/`

Codex review on 892a27d.

- The heartbeat watched only the active conversation id, but `drainNext`
  merges in the `NEW_CONVO` queue, which outlives the URL update: items
  queued during the first turn stay keyed there until that run ends. It now
  renews the union of both, deduped since they are the same atom before
  migration.
- The interval installed without firing, so returning to a conversation
  whose hold was nearly up waited out a full period before the first
  renewal. It now renews immediately, then on each tick.
- Express's non-strict routing sends `POST /files/usage/` to the same
  handler with `req.path === '/usage/'`, so the exact comparison pushed it
  onto both upload limiters. A trailing-slash client would have spent its
  upload quota, and collected file-upload violations, on metadata
  heartbeats. Matching now tolerates the trailing slash.

Firing on effect start also made the drain-time renewal redundant: popping
an item changes the held set, so the renewal effect re-runs on its own. The
one case it cannot see is a refused send, where restoring the item leaves
the set identical, so that branch keeps an explicit renewal and the rest is
removed. Net one request per transition instead of two.
2026-07-28 07:37:26 -04:00
Danny Avila
250aca375a
🔗 fix: Resolve MCP Tool-Key Boundary Against Configured Server Names (#14448)
* fix: resolve MCP tool-name delimiter collision at invocation time

MCP tool keys are identified internally as `${rawToolName}${mcp_delimiter}${serverName}`
(delimiter `_mcp_`). Several call sites parsed this back apart with a naive
`toolKey.split(Constants.mcp_delimiter)`, assuming the delimiter occurs exactly once.

When the raw upstream tool name itself contains the delimiter substring - which
happens whenever it's exposed through a gateway that prefixes aggregated tool names by
server (e.g. a gateway's own "gitlab-get_mcp_server_version" for GitLab's
"get_mcp_server_version" tool) - the combined key has the delimiter more than once.
`.split()` then produces more than two segments, and destructuring
`[toolName, serverName]` silently keeps only the first two, yielding a bogus server
name that matches no configured server. Tool listing still worked (a different code
path builds keys directly without re-splitting), but invocation failed with
`Tool {name} not found`, and `filterAuthorizedTools` rejected such keys outright as
malformed.

Add `splitMCPToolKey`, which splits on the *last* occurrence of the delimiter instead:
the server-name half is always LibreChat's own normalized suffix (guaranteed not to
contain the delimiter), while the raw tool-name half is untrusted and may legitimately
contain it. This matches `.split()`'s result whenever the delimiter occurs once, and
correctly resolves the collision case. Update the four call sites that parsed this
manually (`handleTools.js`, `MCP.js`, `mcp.js` controller, `filterAuthorizedTools` in
`v1.js`) plus one in the client (`useVisibleTools.ts`) to use it.

Fixes #14440

* fix: resolve MCP tool-key boundary against configured server names

splitMCPToolKey moves to librechat-data-provider so the client and backend
share one parser, and takes the configured server names when the caller has
them: the longest name the key actually ends with wins, which is exact.

Position alone cannot identify the boundary because both halves may contain
the delimiter. lastIndexOf alone fixes gateway-prefixed tool names but
regresses servers whose own name contains it, which ToolService.spec.js
already covered; the last-delimiter path now only serves as the fallback for
callers with no configured set.

Also converts the remaining first-occurrence parsers that the delimiter fix
missed - mcp/auth.ts (custom user vars silently unresolved), mcp/oauth/events.ts,
agents/initialize.ts, and the three client parsers that labelled tool calls
with the wrong server.

* fix: keep client tool-call labels on first-delimiter parsing

The three client parsers had deliberate, tested first-delimiter semantics
(ToolCall.test.tsx asserts the full server name for 'foo_mcp_bar' and the
synthetic 'oauth_mcp_server' call), and the client has no configured server
list in scope to resolve the boundary exactly, so they are left as they were.

Threads the configured names into the event-driven definition loader so it
resolves the same boundary as the authorization filter that admits the key,
and documents the one case that stays undecidable without provenance.

* fix: resolve tool-key boundary against all configured servers

resolveConfigServers only returns lazily-initialized config overrides -
ensureConfigServers skips unmodified YAML servers - so on a stock deployment
the known-name list was empty and suffix resolution never engaged. Adds
resolveMcpServerNames, which keeps every configured server in the normalized
form tool keys carry, and uses it at the loading, auth-map and definition
sites.

Background-tool eligibility now resolves against all configured names before
testing ephemeral membership, so a non-ephemeral server whose name ends in an
ephemeral one is no longer misclassified, and useVisibleTools resolves against
the server map it already receives.

* fix: use resolved server provenance and one app-config read

createMCPTool now uses the serverName loadTools already resolved for the key
and only parses as a fallback, so an unmodified YAML server whose name
contains the delimiter no longer resolves to the wrong server for auth,
reconnection and callTool.

resolveMcpServerContext derives config servers and all configured names from
a single getAppConfigForRequest, replacing two independent lookups on the
chat startup path, and degrades to empty like resolveConfigServers instead of
aborting tool loading when the config lookup fails.

* chore: drop unused resolveConfigServers import

* fix: forward server provenance on the all-tools path and read config once

createMCPTools builds each toolKey from the server name it already has but did
not forward it, so the sys__all__sys path re-derived it by parsing and bound
an unmodified YAML server whose name contains the delimiter to the wrong auth
and invocation context.

loadAgentTools now resolves the MCP server context once and threads it into
loadTools, replacing the second app-config read it had introduced on the
non-event-driven chat startup path.

* fix: carry resolved MCP server name through tool classification

definitions.ts resolves the server for each key and then dropped it when
building loadedTools, so buildToolClassification re-derived it with a
last-segment split and recorded 'Workspace' for a server configured as
'Google_mcp_Workspace'. The resolved name now rides along on the tool
instance and classification prefers it over re-parsing.

* fix: consume carried server name when extracting MCP servers

extractMCPServers re-derived the name with a last-segment split, so a server
configured as Google_mcp_Workspace resolved to Workspace and its instructions
were silently omitted. Prefers the name carried on the tool definition
instance, falling back to the split.

* fix: fail closed on ambiguous MCP keys when persisting server names

Persisted mcpServerNames grant agent-scoped access to a DB server by name
(ServerConfigsDB.getAccessibleServers), so a wrong guess exposes an unrelated
server to everyone who can view the agent. The last-segment split turned
search_mcp_Google_mcp_workspace into 'workspace'; such keys were previously
rejected outright at agent save, so admitting them opened this path.

Derives a name only from unambiguous single-delimiter keys. This is #12250's
guard moved to the boundary it was actually protecting, instead of blocking
tool admission.

* fix: keep DB server access for multi-delimiter tool keys

The fail-closed guard was wrong for the case this PR exists to fix. This index
only grants DB-backed servers, and DB names are slugs that cannot contain the
delimiter (generateServerNameFromTitle strips underscores), so the trailing
segment is always the real server for them - dropping it cost every consumer
of a gateway-prefixed tool their shared-agent access.

Also gates the MCP server-context lookup on the filtered MCP set, so an agent
with no MCP tools no longer pays an app-config read on startup.

* fix: resolve tool-call display names without breaking OAuth calls

The display parsers could not use the shared boundary parser because their
tested behavior depends on first-delimiter semantics. That constraint only
applies to synthetic MCP OAuth calls, whose tool half is always exactly
'oauth', so everything after the first delimiter is the server even when the
server name carries one.

splitToolCallName special-cases that form and defers to splitMCPToolKey for
real tool keys, so a gateway-prefixed tool now renders its own name and
server while oauth_mcp_foo_mcp_bar still resolves to foo_mcp_bar.

* fix: persist resolved MCP server provenance on agents

Deriving mcpServerNames from the tool key cannot tell a config server's
trailing segment from a real DB server name, so a config server named
a_mcp_b indexed an unrelated DB server b and shared the agent's viewers into
it. Neither string rule works: the suffix guess exposes, and failing closed
drops legitimate DB access for gateway-prefixed tools.

filterAuthorizedTools already resolves each tool's server against the merged
registry config, so it now collects those names and create, update and
duplicate persist them. No extra registry queries: the update path unions the
newly resolved names with what the agent already had, and duplicate replaces
the copied list rather than inheriting the source's servers.

Display parsing also takes the configured names, so a real tool call on a
delimiter-bearing server renders the right server and icon.

* test: teach MCP hook mocks about useMCPServerNames

Three specs mock ~/hooks/MCP with a hand-listed factory, so adding the hook
to ToolCall made useMCPServerNames undefined under test and every render
threw. Returns a stable array so the mock cannot perturb render counts.

* fix: rebuild agent MCP server index from surviving tools

Unioning the prior names kept a server indexed after its last tool was
detached, so viewers of a shared agent retained agent-scoped access to it.
The index is now rebuilt from the tools that survive the edit: a prior name
carries forward only while some retained tool still resolves to it, using the
agent's own persisted names as the candidate set, and the rebuild runs on any
tool change rather than only when a new MCP tool is added.

* fix: keep duplicate indexes on registry fallback and harden the oauth split

Duplication blanked mcpServerNames when the registry was unavailable, because
filterAuthorizedTools grandfathers the source's tools without resolving them -
the copy kept tools it could no longer resolve. Source names now carry forward
for the tools that still point at them.

splitToolCallName also treated any oauth_mcp_ prefix as a synthetic OAuth
call, so a genuine upstream tool by that name resolved to the wrong server. A
configured server name now decides when one matches, since a real key always
ends in its server, and the prefix only breaks ties for unconfigured servers.

* fix: thread configured server names through display parsing

parseToolName and getMCPServerName resolved context-free, so a configured
server whose name contains the delimiter showed the wrong server in grouped
tool summaries and subagent tool labels, and stacked icons missed its entry in
the icon map. Both take the configured names now, supplied by the components
that render them.

Adds the hook to SubagentCall's mock factory: the spec renders the real
component, so an unmocked useMCPServerNames would reach the query with no
provider.

* test: cover the auth-map boundary, server provenance and context fallback

Adds regression coverage for three behaviors this PR changed that no test
exercised: customUserVars resolving under the right plugin key for a
gateway-prefixed tool name (the failure that made these tools loadable but
unusable), the resolved server name reaching createMCPTool instead of being
re-parsed, and resolveMcpServerContext degrading to empty rather than
aborting tool loading when the config lookup fails.

Each was checked against a mutated source to confirm it fails when the
behavior is broken.

* fix: normalize server-name candidates and cover the boundary guard

Tool keys embed normalizeServerName's output while the config is keyed by the
raw name, so callers passing raw keys never matched a server whose name needs
normalizing and silently fell back to the last delimiter. filterAuthorizedTools
now maps normalized names back to their config key, and createMCPTool
normalizes its candidates.

Adds the cases an audit found surviving mutation: a configured name that is a
bare but not delimiter-aligned suffix must not match, an empty candidate list
behaves as no list, and splitToolCallName still falls back to the oauth prefix
when a list is supplied but nothing in it matches.

* fix: keep resolved server names when a non-owner retains MCP tools

The shared-agent path keeps an agent's existing MCP tools verbatim but supplied
no mcpServerNames, so persistence re-derived them and reduced a configured
server like Google_mcp_Workspace to Workspace - which ServerConfigsDB then
treats as a DB server, granting the agent's viewers access to an unrelated one.
Carries the existing resolved names across instead, and clears the index on the
owner path where every MCP tool is removed.

* fix: preserve resolved MCP names for every tools update

extractMCPServerNames was reachable from any caller that writes tools without
mcpServerNames - the Action edit path does exactly that - so a configured
Google_mcp_Workspace was reindexed as Workspace and ServerConfigsDB granted
shared-agent viewers an unrelated DB server by that name.

updateAgent now rebuilds the index from the agent's own resolved names: one
carries forward while a retained tool still resolves to it, and only keys
matching none of them fall back to derivation. Callers are safe by default
rather than by remembering to pass the set.

normalizeServerName moves to librechat-data-provider so the client can match
its candidates against tool keys, which embed the normalized form; the icon map
is keyed the same way since it is looked up with a parsed server name.

* refactor: move MCP context resolution into packages/api

New backend logic belongs in the TypeScript workspace per CLAUDE.md, with /api
kept to a thin wrapper. resolveMCPServerContext now lives in
packages/api/src/mcp/context.ts and takes ensureConfigServers by injection,
since the registry accessor is still legacy-only; the /api function is reduced
to loading the request app config and translating failures into the empty
degrade it already promised.

* test: teach the MCP service mock about resolveMCPServerContext

The spec mocks @librechat/api with a hand-listed factory, so moving the
resolver into that package left it undefined and the wrapper degraded into its
own catch, returning empty config servers. The stub mirrors the real resolver
so these tests still cover what the wrapper owns - loading the request config
and degrading on failure - while the resolution logic is unit-tested in
packages/api.

* fix: only persist an authoritative MCP server index on update

Assigning the resolved set unconditionally pinned the index to [] whenever
nothing authoritative was available - a legacy agent holding MCP tools with no
stored mcpServerNames - which suppressed updateAgent's derivation and stripped
agent-scoped access to its DB-backed server.

The field is now supplied only when the result is authoritative: names were
resolved, or no MCP tool survives so the index genuinely is empty. The
retained-tools branch likewise leaves it unset when the agent has none stored.

---------

Co-authored-by: Jens Schumann <schumajs@gmail.com>
2026-07-27 14:45:38 -04:00
Danny Avila
cd215150cc
✳️ feat: Claude Opus 5 Support (#14422)
Some checks are pending
Docker Dev Branch Images Build / build (Dockerfile, lc-dev, node) (push) Waiting to run
Docker Dev Branch Images Build / build (Dockerfile.multi, lc-dev-api, api-build) (push) Waiting to run
GitNexus Index / index (push) Waiting to run
GitNexus Index / post-index (push) Blocked by required conditions
* ✳️ feat: Claude Opus 5 Support

- Add claude-opus-5 to Anthropic/Bedrock model lists, token maps, and pricing
- Extend requiresExplicitThinkingDisabled to Opus 5 so thinking-off sticks
- Clamp xhigh/max effort to high when thinking is disabled (Opus 5 400)

* 🪣 fix: Use Bedrock Inference Profiles and Add Vertex Opus Models

Bare `anthropic.` Claude 4+ IDs are not invocable on-demand via Converse:
Bedrock rejects them with "Retry your request with the ID or ARN of an
inference profile that contains this model." Verified live against
us-west-2 for Fable 5, Opus 5, Opus 4.8, Sonnet 5, Sonnet 4.6, Opus 4.6,
Sonnet 4.5, Haiku 4.5, and Opus 4.1. Switch those defaults to the
`global.` profile (no regional pricing premium); Opus 4.1 has no global
profile, so it uses `us.`.

Also add the modern Opus family to the Vertex defaults. `loadEndpoints`
swaps the shared Anthropic list for the Vertex model names, so Opus was
invisible to every Vertex deployment that did not enumerate models by hand.

* 📋 chore: Cover Opus 5 Gaps From PR #14420

Picks up items from the parallel community PR by @jona7o:

- Add claude-opus-5 to the librechat.example.yaml Vertex example (both the
  legacy array and the deploymentName map), which already lists Fable 5
  and Opus 4.8
- Mention Opus 5 in the configureReasoning doc comment, and note that its
  early return is why the effort cap is enforced by the caller
- Assert Opus 5 carries no long-context premium pricing
- Cover the Sonnet 5 negative case for the effort cap, and the persisted
  disabled-object round-trip carrying an effort

* 🌍 docs: Warn That Vertex Regional Endpoints Reject Modern Models

Anthropic serves Sonnet 4.6 and earlier on specific Vertex regional
endpoints; newer models (Opus 4.7+, Opus 5, Sonnet 5, Fable 5) require
`global` or a multi-region location and 404 on a specific region. The
`us-east5` default therefore cannot serve the Opus models added here, nor
the Sonnet 5 entry that predates this branch.

Documents the constraint at all three places an operator sets the region,
and at the fallback itself. Leaves the default unchanged: switching it to
`global` would silently alter data routing and residency for existing
deployments, which is a separate call.

* 🩹 fix: Restore PDF Exemption for Undated IDs and Gate Vertex Defaults

Two issues raised in review:

- BEDROCK_CLAUDE_4_PLUS_RE required a `-` after the major version, so it
  matched `claude-opus-4-8` but not undated IDs like `claude-opus-5`.
  Those models silently lost the Claude 4+ PDF exemption and fell back to
  the 4.5 MB limit. Sonnet 5 and Fable 5 were already affected before this
  branch; Fable/Mythos were also missing from the family alternation.

- The Vertex defaults advertised models that only `global` and the
  multi-region locations serve, so a default `us-east5` deployment listed
  Opus choices that 404 on first request. Filter the built-in defaults by
  configured region instead of changing the region default, which would
  alter data routing for existing deployments. An explicit `vertex.models`
  list is the operator's choice and is never pruned.

* 🧩 fix: Match Bare Claude IDs in the Bedrock PDF Exemption

An application inference profile maps a LibreChat model ID with no
`anthropic.` segment, so `claude-opus-5` failed the Claude 4+ check and
fell back to the 4.5 MB PDF limit. Make the prefix optional and accept
both segment orders, mirroring BEDROCK_CLAUDE_4PLUS_THINKING in
librechat-data-provider, which matches on the family token for exactly
this reason.

Only reached for the Bedrock provider, so the looser prefix cannot leak
into other endpoints. Verified Claude 3.x, Nova, Llama, Cohere, and
Mistral IDs still fall through to the default limit.

* 🧹 fix: Drop Retired Claude 3.5 Models From Bedrock Defaults

The three Claude 3.5 entries reached end of life at AWS and return
ResourceNotFoundException in every prefix form (bare, `us.`, `global.` —
verified live against us-west-2), so selecting one was a hard error.
Their modern equivalents are already in the list: Sonnet 5 / Sonnet 4.6
supersede the 3.5 Sonnets, and Haiku 4.5 supersedes 3.5 Haiku.

Every remaining Anthropic default is now live-verified invocable.
`.env.example` swaps its retired example ID for Haiku 4.5.

* 🔒 refactor: Narrow Effort-Clamp Types Instead of Asserting

Both clamp sites reached into loosely-typed containers with assertions:
llm.ts used an `as unknown as { type?: string }` double assertion to read
the thinking type, and the Bedrock parser cast `output_config` to
`{ effort?: unknown }` before confirming it was an object. CLAUDE.md's
type-safety rules call for narrowing over both.

Adds `isThinkingDisabled` and `clampOutputConfigEffort` to
librechat-data-provider, using `in`-operator narrowing and a type
predicate so no assertion is needed at all. Both call sites now share one
implementation rather than duplicating the clamp.

Behavior is unchanged; existing clamp tests cover it.
2026-07-24 21:45:32 -04:00
Danny Avila
ad5bb477af
🎞️ fix: Surface Clear Error for Unprocessable Gemini YouTube Videos (#14396)
Google rejects a YouTube video it cannot ingest with a generic
`400 INVALID_ARGUMENT` that names no cause, which LibreChat relayed
verbatim. Attribute the failure using request context instead: when a
Google/Vertex turn carried an injected YouTube video part and the
provider returns that generic rejection, map it to a typed error the
client localizes.

Verified against the live API: a public 9h15m video is refused this way
on gemini-2.5-flash, 3.5-flash, 3.5-flash-lite and 3.6-flash, including
at MEDIA_RESOLUTION_LOW, while a short video with an identical payload
succeeds. Duration is the dominant trigger; region and access
restrictions return the same response, so the copy leads with length
without overclaiming.

A duration preflight was evaluated and skipped: oEmbed does not expose
duration, leaving only watch-page scraping — a blocking call against
undocumented markup from rate-limited datacenter IPs that would fail
open and still need this mapping underneath.
2026-07-22 12:11:06 -04:00
Danny Avila
cbaa2fe2e3
⚡ feat: Add Gemini 3.6 Flash and Gemini 3.5 Flash-Lite Support (#14369)
* ⚡ feat: Add Gemini 3.6 Flash and Gemini 3.5 Flash-Lite Support

Adds first-class support for Google's Gemini 3.6 Flash (`gemini-3.6-flash`)
and Gemini 3.5 Flash-Lite (`gemini-3.5-flash-lite`) for both the Gemini API
(AI Studio) and Google Cloud/Vertex integrations.

- Context window (1M) in googleModels; API + cache pricing in tx.ts.
- Model dropdown (config.ts) and GOOGLE_MODELS examples for both integrations.
- Generalize the Gemini 3.5 Flash overrides into a flash-family handler that
  strips deprecated temperature/topP/topK and applies each model's default
  thinking level (3.6 Flash: medium, 3.5 Flash-Lite: minimal), with
  longest-prefix resolution so flash-lite does not collide with flash.

Ref: https://ai.google.dev/gemini-api/docs/latest-model#api-changes-and-parameter-updates

* 🩹 fix: Strip unsupported penalty params for Gemini Flash family

Gemini 3.6 Flash, 3.5 Flash-Lite, and 3.5 Flash reject presencePenalty/
frequencyPenalty with HTTP 400 ("Penalty is not enabled for this model",
verified live). These pass through llmConfig via knownGoogleParams, so add
them to the flash-family strip list alongside the deprecated sampling params.

* 🩹 fix: Strip Flash-blocked params on custom Google endpoint path

For custom OpenAI-compatible endpoints with defaultParamsEndpoint=google,
getOpenAIConfig strips Flash-blocked params via getGoogleConfig but then
transformToOpenAIConfig re-applies raw addParams, undoing the strip. Filter
addParams through stripGeminiFlashBlockedParams before the transform so the
deprecated sampling / rejected penalty params cannot reach the provider.

* 🔧 chore: Update sharp package to version 0.35.3 in package-lock.json, api/package.json, and packages/api/package.json

* 🔧 chore: Update dependencies in package-lock.json to latest versions for @google/genai (2.13.0), @hono/node-server (1.19.14), fast-uri (3.1.4), hono (4.12.31), and svgo (2.8.3)

* 🔧 chore: Update dependencies in package.json and package-lock.json for @librechat/agents (3.2.67), @opentelemetry/sdk-node (0.221.0), and add new dependencies for @opentelemetry/propagator-jaeger (2.10.0) and protobufjs (7.6.5). Update monaco-editor version in client package.json to 0.56.0.

* 🔧 chore: Upgrade turbo package to version 2.10.5 in package.json and package-lock.json, and update schema reference in turbo.json

* 🩹 fix: Resolve CI breakage from bundled dependency bumps

Not related to the Gemini models — both are fallout from the dep bumps on
this branch:
- monaco-editor 0.56 changed IEditorHoverOptions.enabled from boolean to
  'on' | 'off' | 'onKeyboardModifier'; update ArtifactCodeEditor to match
  (mirrors the sibling occurrencesHighlight/matchBrackets pattern).
- sharp 0.35.3 fails resize+encode on a degenerate 1x1 PNG (vipspng: libpng
  read error); the provider-file e2e fixture was 1x1, so use a 16x16 PNG.
  Normal images are unaffected (verified 64x64 resize/encode/jpeg all OK).

* 📝 docs: Correct e2e image-fixture comment (bad IDAT CRC, not a sharp bug)

Root cause was the old 1x1 fixture's corrupt IDAT CRC (verified: IHDR/IEND
CRC OK, IDAT CRC BAD), which sharp 0.35.3's stricter libpng correctly rejects.
Not a dimension/resize edge case and not a sharp bug; comment now reflects that.
2026-07-21 21:14:11 -04:00
MarcAmick
ade02054c8
🛟 fix: Keep File Uploads Alive With SSE Heartbeats (#14295)
Some checks are pending
Docker Dev Images Build / build (Dockerfile, librechat-dev, node) (push) Waiting to run
Docker Dev Images Build / build (Dockerfile.multi, librechat-dev-api, api-build) (push) Waiting to run
GitNexus Index / index (push) Waiting to run
GitNexus Index / post-index (push) Blocked by required conditions
Sync Locize Translations & Create Translation PR / Sync Translation Keys with Locize (push) Waiting to run
Sync Locize Translations & Create Translation PR / Create Translation PR on Version Published (push) Blocked by required conditions
Sync Helm Chart Tags / Ignore non-main push (push) Waiting to run
Sync Helm Chart Tags / Sync chart tags (push) Waiting to run
* fix: Use SSE to upload files in order to avoid idle timeouts.  Idle timeouts can occur for example from gateways and other services like cloudfare when uploading large files.  For example during rag processing the file is uploaded to librechat which then sends it to rag.  While librechat is waiting for the embeddings to come back from rag the file upload is sitting idle.  Gateways tend to want to cancel the upload with an http 408 , 504, or 524.  This change uses SSE to perform the upload so that while librechat is sending the file to rag, it consistently sends back a heartbeat event to the client to keep the connection alive.  This is especially useful when utilizing  EMBEDDING_BATCH_SIZE in librechat rag which will allow rag to process signifigantly larger files without running out of memory.

* added tests to packages\api\src\files\sse.spec.ts in order to test the new sse.ts

* fix: Harden SSE file upload lifecycle

* style: Sort data provider imports

---------

Co-authored-by: Marc Amick <MarcAmick@jhu.edu>
Co-authored-by: Danny Avila <danny@librechat.ai>
2026-07-21 19:27:09 -04:00
Danny Avila
9e245aced4
🎟️ fix: Claim Idempotency Keys to Dedup Retried Generation Requests and Prevent Double Billing (#14344)
* 🐛 fix: Dedup retried start-generation requests to prevent duplicate billing

A lost or reset start-generation response makes the client re-POST the
identical payload (up to 3x on network errors). The resumable-stream
controller had no idempotency: createJob unconditionally overwrote the
running job without aborting the prior one, so both requests ran full
LLM completions and both billed while the UI showed only one (#14339).

Add a stable per-submission clientRequestId (uuid, fresh per ask() so a
regenerate differs, reused across the start-generation retries) and an
atomic claim on the job store keyed by userId:clientRequestId. The first
request wins and generates; a retried POST loses the claim and receives
the original stream, which the client subscribes to and replays - no
second billed generation.

- IJobStore.claimIdempotencyKey/releaseIdempotencyKey (in-memory Map+TTL,
  Redis single-key SET NX PX + GET Lua, cluster-safe)
- GenerationJobManager.claimGeneration/releaseGeneration (20m TTL)
- Controller claims before the concurrency check, dedups with a resumed
  response, releases on start-failure/429
- clientRequestId threaded through TSubmission/TPayload/createPayload

* 🐛 fix: Harden start-generation dedup (Codex review)

Address three P2 findings on the idempotency path:

- Resume replay: a deduped retry now subscribes with resume=true so the
  client replays prior content and any pending-action from the running
  stream instead of only live events (cross-replica / HITL correctness).
  startGeneration returns { streamId, resumed } and the response's
  status:'resumed' drives the subscribe mode.
- Wait for the job record: a duplicate that loses the claim now waits
  briefly for the winner to create the job before returning the stream
  (a stream with no job 404s terminally). If the winner has not
  materialized, return 503 SERVER_NOT_READY so the client retries via the
  existing readiness path instead of attaching to a dead stream.
- Release only owned claims: track whether the request actually won the
  claim; the 429 and init-error paths no longer release a claim owned by
  another in-flight generation (fail-open path could erase it and
  re-enable double billing).

Adds controller tests covering dedup, the 503 race fallback, win-then-
create, and claim-release ownership on 429 / fail-open.

* 🐛 fix: Don't trap deduped retries on missing job records (Codex review)

The previous round returned 503 SERVER_NOT_READY when a deduped retry's
job record was absent. But a missing job usually means the original
generation already completed and was cleaned up (cleanupOnComplete) — the
correct recovery is to return the stream and let the client's subscribe
404 handler refetch the persisted messages. The 503 instead trapped the
send in a readiness-retry loop until the client's window expired.

Keep the bounded wait (it still covers the job-about-to-be-created race)
but always return the resumed stream afterward; a gone/never-created job
recovers via the client's existing 404 path instead of being treated as
indefinitely starting. Updated the controller test accordingly.

* 🐛 fix: Gate deduped resume on claim age, not just job presence (Codex review)

Removing the 503 entirely (previous round) reintroduced the inverse race:
if the winning request stalls between claimGeneration and createJob, a
losing duplicate saw no job, returned status:'resumed' anyway, and the
client subscribed to a stream that did not exist yet — the 404 handler
tore the turn down while the winner went on to generate and bill with no
UI attached.

Distinguish the two missing-job cases by claim age (claimedAt now travels
on the claim value):
- fresh claim, no job yet → winner is still starting → 503 SERVER_NOT_READY
  so the client retries via the readiness path (bounded, not indefinite).
- old claim, no job → the original already completed and was cleaned up
  (or the winner died) → attach; the client's 404 handler refetches.

Tests cover both age branches.

* 🐛 fix: Scope dedup fail-open + keep resumed convos on 404 (Codex review)

- Fail-open only on claim acquisition: a store error while checking an
  already-confirmed existing claim no longer falls through to createJob
  (which would start a second billed generation during a Redis hiccup).
  Once claim.existing is known, a job-lookup error returns 503 retry.
- Don't drop a resumed convo on 404: the optimistic-conversation cleanup
  in useResumableSSE now runs only for fresh (non-resume) subscribes. A
  deduped resume whose original completed and was cleaned up 404s, but its
  conversation is persisted and must stay in the sidebar.

Adds a controller test for the job-lookup-error path (503, no createJob).

* 🐛 fix: Reconcile resumed convos on 404 instead of guessing (Codex review)

Round-4's !isResume guard fixed the completed-and-cleaned case (don't drop
a persisted convo) but left the inverse: a new-conversation retry deduped
to a claim whose original worker died before persisting still resumes,
404s, and — with removal skipped — leaves a phantom /c/<streamId> sidebar
entry.

Stop guessing keep-vs-remove on a resume 404. Reconcile against the
server: invalidate the conversations list so a real (persisted) convo
stays and a phantom is dropped. Fresh (non-resume) optimistic streams
still prune immediately. Adds a client test for the resume path.

* 🐛 fix: Finalize failed job before releasing its claim (Codex review)

In the initialization-error catch, the idempotency claim was released
before completeJob(streamId). A racing retry could win the released key
and createJob() the same streamId while this catch was still running, and
completeJob() (not guarded by the original createdAt) would then abort the
replacement. Finalize the failed job first, then release the claim.

Adds a controller test asserting completeJob precedes releaseGeneration.

* 🐛 fix: Clear claims on destroy + survive completeJob failure (Codex review)

- InMemoryJobStore.destroy() now clears the idempotencyClaims map, so a
  reused/reconfigured store instance doesn't dedup a fresh start against a
  torn-down job's stale claim.
- Init-error cleanup: completeJob() is swallowed so a store-hiccup
  rejection can no longer skip the idempotency-key release and the
  pending-request decrement (which would wedge the retry behind the claim
  and leak the concurrency slot). A failed completeJob finalized nothing,
  so releasing afterward still can't abort a later replacement.

Tests: claims cleared on destroy; release + pending decrement still run
when completeJob rejects.
2026-07-21 08:16:31 -04:00
Danny Avila
7447fddfb2
🙊 refactor: Clarify Ask Question Schema Errors and Retry Guidance (#14279)
* fix(agents): clarify ask question validation errors

* fix(agents): narrow question failure detection

* fix(agents): persist question validation failures

* fix(agents): track question validation failures
2026-07-15 11:06:29 -04:00
Danny Avila
91658339ec
🎫 fix: Strip Reserved Fields From Bedrock additionalModelRequestFields (#14246)
* 🐛 fix: Strip duplicate `system` from Bedrock additionalModelRequestFields

Bedrock Anthropic presets bind the system prompt to the `system` model
param. bedrockInputParser routes `system` into additionalModelRequestFields,
then bedrockOutputParser promotes it back to the root as a known key without
removing the copy. Bedrock Converse then sees `system` in both places and
rejects the request ("The additional field system conflicts with an existing
field"), which surfaces once context compression/summarization runs.

Delete `system` from additionalModelRequestFields after promoting it to the
root. `system` is the only leaked field that collides with a reserved
top-level Converse field, so the fix is scoped to it and leaves other
passthrough fields untouched. Clones before mutating to avoid touching the
caller's input.

Closes #14029

* 🛡️ fix: Guard scalar additionalModelRequestFields before `in` check

DocumentType permits scalar values (boolean/number/string), so a saved
Bedrock preset/agent can carry a non-object additionalModelRequestFields.
The new `system` cleanup used `'system' in amrf`, which throws
TypeError on a truthy scalar. Guard with a typeof-object check to keep the
prior tolerant behavior; the empty-check is left unchanged.

* 🛡️ fix: Strip all reserved Converse fields from additionalModelRequestFields
2026-07-14 15:29:39 -04:00
Danny Avila
5771bf6e06
♨️ feat: Prewarm Stateful Code Sandboxes with Cold-Boot UX Feedback (#14239)
* ♨️ feat: Prewarm Stateful Code Sandboxes with Cold-Boot UX Feedback

* 🧹 fix: Drain Prewarm Response + Reset Sandbox Atoms on Stream Cleanup

* 🚿 fix: Propagate Prewarm Drain Failures + Warm Marker for Host File Tools

* 🌡️ fix: Decouple Prewarm In-Flight State from Warm Refreshes + Precise Ready Gates

* ☁️ refactor: Redis-Backed Sandbox Prewarm State via standardCache

* 🧪 chore: Hermetic Prewarm Spec + Accurate Signal JSDoc (Copilot review)
2026-07-14 10:25:37 -04:00
Danny Avila
9bb351ad9c
🧭 feat: Mid-Run Steering and Queued Messages for Agent Runs (#14220)
* 🧭 feat: Mid-Run Steering and Queued Messages for Agent Runs

Steering: submit a message while a run is generating; the server queues
it in the job store (cross-instance) and a run-scoped PostToolBatch hook
injects it into graph state at the next tool-batch boundary, records an
inline 'steer' content part on the response (replayed as a user message
on later turns), and streams on_steer_applied to the client.

Queuing: messages composed during a run auto-send as normal follow-up
turns after clean completion (one per final event, FIFO); user aborts
leave them as chips unless armed by interrupt-and-send.

Requires hook injectedMessages support in @librechat/agents
(danny-avila/agents#299); hard-gated via a capability probe so older
SDKs 501 the steer route instead of draining and dropping messages.

* 🧵 fix: Harden Steering Against Finalization Races and Route Guard Gaps

Addresses local Codex review findings on the steering feature:

- Close-and-drain the steer queue atomically at finalization (final event,
  abort) so a steer POST racing teardown is rejected instead of 202-ACKed
  and then silently cleared; the closed flag lives on the job hash and is
  reset when a replacement job reuses the stream id.
- Clear inherited steer queues on createJob — a job replacement must not
  drain the replaced run's messages.
- Keep steers queued across a HITL pause instead of draining them into
  ephemeral client state: resumeState re-seeds chips on reload and the
  resumed run injects them at its first tool boundary (steers key TTL now
  extends to the approval window; on_steers_pending event removed).
- Queue the NO_ACTIVE_RUN steer fallback while the final SSE is still
  settling — a direct send would be dropped by ask()'s in-flight guard.
- Reconcile the 202 ACK against on_steer_applied events that beat it over
  the SSE, so a chip can't be re-minted after its removal event passed.
- Allow the per-send Steer override when the default action is queue.
- Apply the configured message rate limiters and the PII filter to
  POST /chat/steer — a steer is model-bound user text.

* ✅ ci: Assert Steering Capability Probe Against the Installed SDK

CI installs the published @librechat/agents pin (pre-injectedMessages),
where isSteeringSupported() is legitimately false — the probe test now
asserts it mirrors the installed SDK's capability flag instead of
hardcoding the capability-bearing build's value. Verified against both
the published 3.2.61 dist and the agents#299 build.

* 🛟 fix: Preserve Steer Text Across Run-End, Error, and Abort Races

Codex round 2 (4 P2s):
- Applied-steer-id set survives run end (capped at 100) and converted
  ids join it, so a 202 ACK that lands after final/abort drops its chip
  instead of re-minting a stranded pending one.
- Failed runs no longer strand acknowledged chips: both error paths
  convert local pending chips to queued follow-ups (chip text is
  client-local), and the server closes the steer queue before emitting
  the error so a racing steer POST gets 404 fallback instead of a 202
  whose payload dies with the job.
- sendQueuedNow keys on steer availability, not the default action —
  send-now on a queued chip is an explicit override for queue-preferring
  users.
- Stop path consumes pendingSteers from the abort HTTP response as a
  fallback for the SSE final event it may close before processing;
  conversion is deduped so double delivery is a no-op (shared
  useSteerConvert hook).

* 📎 feat: Carry Attachments Through During-Run Queued Messages

Steering stays text-only (SDK injection, inline STEER part, and replay
are all text), so a during-run submit with media now queues the whole
message as one unit instead of silently stranding the files:

- QueuedMessage gains `files`; composer attachments are consumed into
  the queued item at queue time (steerFromComposer / queueFromComposer /
  interruptAndSend), fixing the latent hazard where lingering composer
  files glued onto whatever `ask` vacuumed up next.
- Enter-steer with attachments degrades to queue with an explanatory
  toast; the per-send menu routes through the same composer-aware
  wrappers.
- The drain and sendQueuedNow pass the item's files as `overrideFiles`;
  media items never steer (send as a normal turn when idle, re-front
  otherwise). ask() no longer clears composer state for caller-supplied
  overrideFiles — only regenerate keeps that behavior.
- During-run submits hold while uploads are in flight, mirroring the
  send button's filesLoading gate; queued chips show a paperclip count.

* 🎛️ feat: Rework During-Run Chips into Action Rows

Full-width rows above the composer (reference-UI parity): each queued
message shows a primary Steer/Send-now action, delete, and a "…" menu
with Edit message (restores text + attachments into the composer) and a
Turn on queueing/steering toggle that flips the Enter default. Steer
rows share the layout with status text; failed steers keep retry /
edit / queue-convert. The per-send menu gains the same default toggle.
Queued file refs now retain filename + bytes so edit-restore rebuilds
real composer entries (draft-recovery shape).

* 🖇️ feat: Steer With Attachments (Multimodal Mid-Run Injection)

Steering now carries media end-to-end instead of degrading to queue:

- The steer POST accepts sanitized attachment refs (cap 10; only
  file_id is trusted — the drain re-fetches owner-scoped and re-derives
  everything else). SteerQueueItem/TPendingSteer/SteerContentPart carry
  `files` refs; encoded data is never persisted or queued.
- New api/server/services/Files/steering.js decouples attachment
  building from the request path: encodeSteerContent reuses the exact
  per-turn pipeline (addFileContextToMessage + processAttachments'
  single-pass categorize/encode, SDK formatMessage assembly,
  prependFileContext for extracted text) with zero new encoding code.
  buildSteerMedia feeds the drain hook's new buildMedia seam (any
  failure degrades that steer to text-only — words always land);
  stampSteerPartMedia re-encodes past steer parts per turn with ONE
  batched owner-scoped fetch and stamps a transient `media` array,
  replaced immutably so it can never leak into a save. Replay honors
  resendFiles like regular message media.
- The SDK's formatAgentMessages (the formatter agents actually use)
  gained the steer replay branch on the PR branch; the local
  formatMessages.js branch now mirrors the media preference.
- Client: steerFromComposer consumes composer files into the POST,
  chips/seeding/conversions carry files everywhere (retry, queue
  convert, abort/error recovery), queued media items steer for real,
  and SteerBubble renders the steered attachments inline.

* 🧵 fix: Harden Steer Recovery Races and Drain Isolation

Codex round 3 (7 fixes):
- A 202 ACK landing after the run ended converts straight to a queued
  follow-up (server queue is gone; no event will ever resolve a pending
  chip for a finished run). Covers stream errors with in-flight POSTs.
- A Stop that lands pre-completion can arrive as a final with
  unfinished:true and no aborted flag — runEnd now treats it as aborted
  so queued messages are not auto-sent against the user's Stop.
- Leftover-steer conversion merges chronologically by createdAt instead
  of appending, preserving the order the user composed.
- Auto-drained queued messages pass explicit (possibly empty)
  overrideFiles/overrideQuotes/overrideManualSkills: a drain can no
  longer vacuum up files, quotes, or skill picks staged in the composer
  for the user's NEXT message (ask() treats overrideFiles != null as
  authoritative).
- Failed-steer Retry and resume-on-load chip restoration keep the
  steer's attachments.
- The job-replacement guard moved INSIDE the store's atomic
  drain/close-and-drain (Lua createdAt compare; in-memory equivalent):
  a stale run's hook or finalization can neither consume, close, nor
  steal a replacement job's steer queue, and the drain hook drops its
  separate check-then-drain round trip.

* 🧰 refactor: Typed Steer Controller, Single-Query Media Pass, Round-4 Fixes

Codex round 4 + efficiency tightening in one pass:

- Moved the steer guard ladder (validation, file sanitization via a
  shared toSteerFileRef picker, ownership/tenant checks, status-guarded
  enqueue) into packages/api as handleSteerRequest; api/steer.js is now
  a thin wrapper. Ladder covered against the REAL in-memory job manager
  in request.spec.ts; the api spec pins only the wrapper contract.
- Folded the steer replay stamp into the turn's ONE historical-files
  query: collectHistoricalFileRefs also gathers steer-part refs, the
  owner-scoped doc map rides client state, and stampSteerPartMedia
  consumes it (no second round trip) while encoding parts in parallel.
- Stamped steer media now counts against the run budget (existing
  multimodal counter over the non-text parts, folded into
  indexTokenCountMap/promptTokens after the stamp).
- Steer route runs the PII filter BEFORE moderateText, matching chat.js
  so blocked sensitive text never reaches the external moderation API.
- Interrupt & send survives the abort-response-beats-SSE-final race:
  stopGenerating writes the run-end signal itself when the one-shot
  interrupt flag is armed and no signal landed (double-fire safe).
- Resume reconciles chips against the server's still-queued list even
  when EMPTY, clearing chips for steers applied while disconnected.
- The local formatter's steer flush preserves non-text assistant parts
  (array-content AIMessage) instead of folding to text.

* 🔒 fix: Replay-Aware Capability Gate and Round-5 Race Closures

- isSteeringSupported now requires BOTH halves of the SDK contract:
  injection (HOOK_INJECTED_MESSAGES_CAPABLE) AND replay
  (ContentTypes.STEER, shipped in the same SDK commit as the
  formatAgentMessages steer branch). An SDK that can inject but not
  replay 501s the steer route — no release window can create steer
  parts that would leak into provider-facing assistant content.
- The local formatter mirrors the SDK's anchor reset: a post-steer
  tool_call mints a fresh AIMessage instead of attaching to the
  pre-steer anchor (invalid provider ordering).
- Queued-chip send-now and the NO_ACTIVE_RUN fallback pass explicit
  (possibly empty) overrideFiles so an idle send can't vacuum composer
  files staged for a different draft.
- Redis createJob deletes the stale steer list BEFORE the replacement
  hash is written as running — a steer 202-accepted against the new job
  can never be wiped by the reset.
- Resumed-turn finalization mirrors the normal path's terminal drain:
  createdAt-guarded close-and-drain, leftovers ride the resumed final
  event as pendingSteers instead of being cleared by completeJob.
- buildSteerMedia restores composer order over the $in result so
  multi-attachment steers reach the model in the order the user saw.

* ⚛️ fix: Atomic Job Replacement and Boundary-Clean Steering Module

Codex round 6 (5 fixed, 1 standing deferral):
- createJob resets the steer queue and writes the job hash in ONE
  same-slot Lua script (JOB_CREATE_LUA): a steer POST can no longer
  interleave between them on cluster, so a steer accepted against one
  run can never be drained into another. Redis-validated.
- The steering media pipeline moved to packages/api
  (agents/steering/media.ts) with injected getFiles and a structural
  client interface — /api keeps zero steering logic; specs ported to
  the DI seam.
- handleSteerRequest checks the job BEFORE the capability gate: a steer
  racing completion on an unsupported SDK gets 404 (send-now) instead
  of a 501 queue with no run-end signal left to drain it.
- useQueueDrain binds to the active conversation: navigating away
  between the final SSE and the drain effect leaves the signal
  unconsumed instead of submitting A's follow-up into B; the drain
  fires on return.
- abortJob closes and drains the steer queue BEFORE the content
  snapshot, so a drain-hook apply that lands pre-drain is captured
  inline rather than lost between the snapshot and the terminal drain.

* 🚦 fix: Parked Run-End Signals, Interrupt Priority, Settled-Run Fallbacks

Codex round 7 (5 fixes):
- Run-end signals for a non-active conversation are PARKED per
  conversation instead of squatting the shared index slot: a later run
  finishing on the same pane can no longer overwrite them, and the
  parked drain fires when the user returns.
- "Interrupt & send" front-inserts carry a priority flag that outranks
  createdAt when abort leftovers merge back chronologically — the
  urgent redirect drains first, not the oldest steer.
- STEER_UNSUPPORTED/RUN_PAUSED/QUEUE_FULL rejections landing after the
  run settled mirror the NO_ACTIVE_RUN fallback and send immediately
  (queueing would strand the text with no run-end signal left); on the
  pinned SDK this is the common Enter-near-run-end path.
- A failed abort (e.g. 404 when the run completed first) still signals
  the interrupt drain, so the queued interrupt message can't strand and
  the armed flag can't leak onto a later run.
- Steered-image fallback alt text is localized (com_ui_attached_image).

* 📌 chore: Adopt Published @librechat/agents Types Post-Bump

dev's pin bump to ^3.2.62 (the release carrying injection + steer
replay) landed via merge; the steering runtime now uses the SDK's real
InjectedMessage/hook-output types instead of the local structural
mirrors that bridged the pre-publish window. The two-half capability
probe stays as the defensive gate for mismatched deployments — and the
capability spec now exercises its TRUE path against the published
package in CI.

* 🛅 feat: Park-and-Claim Steer Recovery + Host-View Content Reads

Codex round 8 (6 fixed incl. both P1s, 1 push-back):
- The long-deferred no-subscriber gap is closed: every terminal drain
  (final, aborted-final, error, abortJob, resumed finalize) PARKS
  acknowledged leftovers on the job hash (unrecoveredSteers), and the
  status route claims them exactly once for inactive jobs — a client
  that closed/reloaded past the transient final event restores its
  steers as queued chips within the post-terminal TTL. A replacement
  run clears the parked copy (a live client started it).
- Same-instance content reads are steer-complete: RedisJobStore now
  caches the HOST content array (WeakRef) via setContentParts and
  prefers it over the SDK graph cache, whose view never contains
  host-authored steer parts; the graph fallback splice-INSERTS steer
  chunks at their recorded host-view indices (the graph array is
  unshifted, so assignment would overwrite SDK parts).
- Replay token accounting now counts prepended file-context text: full
  stamped content minus the steer body (already counted), so large
  steered documents hit the budget instead of bypassing pruning.
- The queue drain restores an item when ask() refuses without sending
  (history not yet in cache after navigating back) — text is never
  silently dropped.
- The armed interrupt flag travels WITH a parked run-end signal, so
  another run on the same pane can neither consume nor clear it.
- parseTextParts extracts steer text (search indexing / audio).

* 🎛️ refactor: Single Send Slot + In-Thread Steer Messages

- Merge the during-run send affordance into the send/stop button slot:
  with composer text the send button replaces Stop (Enter = default
  action), hover reveals Steer/Queue/Interrupt rows with shortcuts;
  drop the separate DuringRunActionsMenu chevron
- Add during-run keyboard chords: Cmd/Ctrl+Enter = non-default action,
  Alt+Enter = interrupt & send (plain-Enter submitters only)
- Render steers as standard user messages in the thread: SteerPart
  (icon + author header + user text presentation) replaces the
  SteerBubble, and submitted steers appear immediately at the projected
  injection point via the PendingSteers slot on the streaming message
- Keep composer rows only for recoverable states: failed steers
  (retry/edit/queue) and queued follow-ups

* 🩹 fix: Keep the Replacement Submission Alive Across Abort Settlement

The aborted run's final SSE event fires before the abort HTTP response
resolves, so an armed interrupt & send drains and starts the NEXT
submission while the abort POST is still in flight. The response
handler's unconditional clearAllSubmissions() then reset the new
submission, aborting its stream attach before the subscribe — the
follow-up ran and persisted server-side but the live placeholder
finalized empty (content appeared only after reload).

useAbortCleanup captures the submission before the abort round-trip
and both settlement paths (success and 404-catch) clear only when the
captured submission is still current; a replacement stays untouched.
Plain Stop behavior is unchanged.

* 🧭 test: Playwright E2E for Mid-Run Steering and Queuing

- Add e2e/specs/mock/steering.spec.ts: steer mid-run (202 + immediate
  in-thread pending part + real MCP tool boundary + words survive run
  end), Cmd/Ctrl+Enter queue with auto-send after clean completion,
  and Alt+Enter interrupt & send with the follow-up streaming into the
  live view
- Add the E2E_STEER_TOOL_REPLY fake-model marker: slow preamble, a
  real remember_fact MCP tool call (PostToolBatch boundary), then a
  final turn
- Test 1 pins the run-end degradation contract while the SDK's
  top-level agentId stamping bug blocks live injection; its header
  documents the assertions to flip once the fixed SDK is pinned

* 🧷 fix: Job-Independent Steer Recovery + Expiry and Resume-Gap Parking

Codex round 10: the park-and-claim recovery had lifecycle holes.

- Move parked steers off the job hash onto their own bounded-TTL store
  key (JOB_CREATE_LUA resets it; deleteJob leaves it alone): the default
  completeJob path deletes the job record immediately, and the Redis
  read path never deserialized the old hash field — recovery previously
  worked only with STREAM_KEEP_COMPLETED_JOBS on the in-memory store
- Carry the owner identity inside the parked payload and authorize the
  claim against it, so the status route recovers steers on its jobless
  branch too (the common reload-after-terminal case); a non-owner claim
  returns nothing and re-parks the payload
- Park queued steers on approval expiry: snapshot the frozen queue
  before the requires_action→aborted CAS (whose terminal cleanup drops
  the steers key) and park only when the CAS wins
- Mirror the terminal drain/park block in resume.js's failure path,
  which previously let completeJob's backstop clear 202-accepted steers
- Close the Redis snapshot→subscribe resume gap: re-peek the queue
  after attaching and re-surface missed on_steer_applied events from
  the durable content view (synthesizeAppliedSteerEvents), updating
  resumeState.pendingSteers to the live queue

* 📌 chore: Require @librechat/agents 3.2.63 + Applied-Steer E2E Contract

- Bump the @librechat/agents pin to ^3.2.63 in api/ and packages/api/:
  it scopes the hook agentId marker to subagent child graphs, so the
  steering drain hook fires at top-level tool-batch boundaries and
  mid-run injection is active (danny-avila/agents PR 307)
- Flip e2e steering test 1 from the documented degradation contract to
  the applied-steer contract: the optimistic in-thread part transitions
  to the persisted part at the tool boundary and survives inside the
  response after run end, with no queued follow-up turn

* 🎗️ feat: Steered Messages Join the Message-Nav Ribs

Steers are user messages, so they get their own clickable rib on the
navigation rail, interleaved at their in-thread position inside the
response that absorbed them (one DOM query in document order). SteerPart
anchors itself as #steer-<id> with a steer-render marker — both the
optimistic pending entry and the persisted part — and the rib carries
the user role label with a preview drawn from the steer's text body,
skipping the author header.

* ❎ feat: Cancel a Queued Steer Before Injection + True User-Message Alignment

- Add POST /chat/steer/cancel: removes ONE still-queued steer by id via
  an atomic list rebuild (Redis Lua preserves order and TTL), authorized
  against the job owner; removed:false is advisory — the cancel lost its
  race to the drain or the run end, never an error
- Surface an × on the in-thread pending steer (server-acknowledged
  entries only): optimistic removal, restored if the POST fails since
  the server would still inject the words
- Outdent SteerPart past the response's icon column so steers sit flush
  with top-level message rows, reading as regular user messages

* 🧯 fix: Round-11 Recovery Hardening + Provider-Free Pending Slot

- Reconcile the resume steer gap by steerId SETS, not queue length — a
  steer added in the gap (or an equal-length drain+enqueue swap) now
  refreshes resumeState.pendingSteers and still synthesizes the missed
  on_steer_applied events
- Make completeJob's terminal backstop park: direct error-path callers
  without the controllers' close-and-park no longer silently clear
  202-accepted steers (createdAt-guarded closeAndDrain + owner park
  before the terminal write)
- Persist the steer part BEFORE media encoding in the drain hook: an
  abort inside the encode window can no longer lose a file-steer (the
  part refs come from the enqueue-sanitized item; replay re-encodes
  per turn unchanged)
- Move the parked-claim owner check INSIDE the atomic store claim
  (substring gate in the Lua / in-memory equivalent): a non-owner probe
  can no longer transiently delete the recovery payload; the app-side
  parse stays authoritative
- Park queued steers in BOTH stores' own requires_action expiry
  cleanup, which bypassed the manager-level sweep
- Sweep expired parked steers from the in-memory store's periodic
  cleanup; restore a queued chip when send-now's submit is refused;
  upsert steer ACKs so an SSE reconnect reseed cannot duplicate chips
- Mount the cancel mutation per steer item so the pending slot needs no
  QueryClient on ordinary streaming renders (fixes the CI failure in
  ContentParts.integration.test)
- Skipped delivery-gated parking (finding 8): transport receiver counts
  cannot prove browser delivery, and gating the only durable copy on
  them trades cosmetic chip resurrection for real text loss; the window
  is already bounded by claim-on-read, createJob reset, and the TTL

* 🩺 fix: Annotate PARKED_STEERS_TTL_MS for isolatedDeclarations

tsdown's d.ts generation requires explicit types on exported consts
with computed initializers; tsc --noEmit does not run that check, so
the round-11 export slipped past local verification and broke Build
packages (and every downstream CI job that consumes the built dist).

* 🛟 fix: Round-12 Terminal-Path Recovery + Durable Steer Events

- Park queued steers before the stale-running reap deletes a crashed or
  hung job in BOTH stores — the one terminal path with no controller
  finalization; requires_action expiry parking refactored onto the same
  snapshot/park helpers
- Enqueue instead of dropping when a steer fallback send is refused:
  both the NO_ACTIVE_RUN branch and the settled-run rejection branch
  now observe sendNow's false return
- Recover on the SSE reconnect-404 terminal path: convert local pending
  steers to queued, claim parked steers via /chat/status, and write a
  non-completed run-end signal so interrupt flags release without
  auto-sending an unknown outcome
- Fall back to a positive parked-recovery TTL when completedTtl is 0
  (SET EX 0 is invalid and silently killed recovery)
- Make on_steer_applied durable before publish: emitChunk gains a
  durable option that awaits the chunk-log append (best-effort) ahead
  of the transport publish; the default delta path stays fire-and-forget

* 🔐 fix: Round-13 Steer Authorization + Trusted File Refs

- Resolve client-supplied steer file refs against the DB owner-scoped
  at enqueue and queue only DB-derived shapes (same filter as the
  injection fetch, shared via refs.ts); any unresolved id fails loud
  with 400 — spoofed type/filepath metadata can no longer be persisted
  into assistant content or rendered in chat/share views
- Enforce agent authorization on /chat/steer against the ORIGINATING
  run's job identity: the chat path's role gate (AGENTS:USE, with the
  same non-agents-endpoint skip) plus the per-agent ACL check with the
  capability bypass — revoked access mid-run can no longer inject;
  cancel stays ownership-only (nothing model-bound)
- Mark steered uploads used after a successful enqueue (owner-scoped,
  best-effort) so the upload-window TTL cannot reap a file the
  persisted steer part references
- Consume the parked recovery copy after live delivery: converting
  final/abort/error pendingSteers fires one owner-gated claim-on-read,
  so dismissed chips can no longer resurrect on a later reload

* 🎙️ fix: Round-14 Composer-Context Fidelity + TTS and Queue-State Gaps

- Keep steer text out of generic assistant text extraction:
  parseTextParts excludes STEER parts by default with an includeSteer
  opt-in for the full-record surfaces (Meili indexing, aborted-response
  persistence) — TTS callers no longer speak the user's own mid-run
  words
- Mark queued uploads used at enqueue time via a minimal owner-scoped
  POST /files/usage (fail-closed without a user; upload limiters do not
  apply to a metadata touch), fired once wherever composer files enter
  the queued state — the upload-window TTL can no longer reap a file
  waiting out a long run or approval pause
- Carry quote chips and manual skill picks on queued items: captured
  and consumed from the composer at queue/interrupt time exactly like
  files, threaded through the drain and send-now overrides, and
  restored by the queued row's Edit message
- Key an early-aborted FIRST turn's run-end signal to NEW_CONVO
  (resolveRunEndTarget) so queued follow-ups stay visible on the
  restored new-chat composer instead of parking under an optimistic
  stream id the user never sees again

* 🧿 fix: Round-15 Gap Coverage + Consolidated Sweep (Share Leak, Abort Ids, Chip Hygiene)

- Run the resume steer-gap check for every still-active job: an empty
  snapshot no longer skips the re-peek, and synthesis now keys on the
  FRESH content view so an applied-in-gap steer that was never
  snapshotted still re-surfaces (over-emission is benign — applied-id
  dedupe, index-stable parts)
- Thread queued context through steer degradation: sendQueuedNow passes
  the item's quotes/skills into submitSteer, and every fallback
  (requeue or settled send) restores them instead of dropping to
  text+files
- Stop shared links from leaking steer attachment refs: the share
  snapshot now walks content — files-excluded shares strip steer-part
  files entirely; files-included shares sanitize and share-route them
  like top-level files (copy-on-write, non-steer content by reference)
- Seed pending-steer chips unconditionally on load/return so a steer
  applied while away cannot linger as a stale chip beside its part
- Use the abort response's resolved job id: chips/drain-signal land
  where the user actually is (NEW_CONVO for a new-held first turn,
  consistent with resolveRunEndTarget) while the parked-copy claim hits
  the resolved id instead of a no-op /chat/status/new
- Open steered documents like normal message files (FilePreviewDialog)
- Cap the applied-steer id set on the live path via a shared helper;
  kept surviving run end deliberately (late-ACK race depends on it) and
  fixed the atom comment that claimed otherwise

* 💡 fix: Un-light Steer Ribs When Their Node Is Replaced

Two stacked gaps kept a steer rib lit after scrolling away: the
pending→applied swap replaces the DOM node under the same id, which
produces no IntersectionObserver exit and — because the entry list
dedupes on (id, preview) — no entries change either, so the observer
kept watching a detached node; and the rail's mutation filter only
reacted to .message-render nodes, so steer-node swaps and removals
never triggered a refresh at all.

- reconcileObservedElements re-points the observer at replaced nodes
  from the mutation-driven refresh regardless of entries identity,
  dropping stale visibility until the fresh node reports (the observer
  fires its initial intersection immediately, so a truly visible part
  re-lights within a frame)
- The mutation filter now recognizes steer-render nodes alongside
  message rows

* 🪪 fix: Round-16 Recovery Owner Fields + Context Stickiness + Share Labels

- Park resumed-run leftovers with the manager facade's metadata owner
  fields: a bare job.userId is undefined on that shape, which made
  every parked payload from a resumed HITL run unclaimable
- Keep a queued item's quotes/skills sticky through a successful steer
  ACK: the pending chip carries them (client-only), reseeds preserve
  them across reconnects, and every terminal conversion — local or
  server-list, merged by steerId — restores them onto the queued item
- Convert resumeState.pendingSteers on the inactive status branch
  (deduped against unrecoveredSteers) so steers observed in the
  expired-pause-before-sweeper window convert instead of vanishing
  until a later reload
- Label shared steer parts share-safely via the existing ShareContext:
  a viewer's own name no longer appears on the sharer's steered
  messages

* ✂️ fix: Carry Steer Context Through the Failed-Chip Edit Action

Retry and convert-to-queue already preserve a failed steer's carried
quotes/skills; Edit message dropped them on the way back to the
composer. It now restores them through the same context path.
2026-07-14 10:11:10 -04:00
Dustin Healy
b0d46b0518
🗝️ feat: Encrypted Langfuse Fanout Config (#14107)
* feat: encrypt tenant Langfuse secret in admin config

Add generic per-field secret encryption to the admin config layer: registered
secret paths (langfuse.secretKey) are encrypted with encryptV3 on write and a
non-secret fingerprint companion is stored. Admin config reads (base + per
principal) redact registered secrets so they are never returned; the fingerprint
is kept so the UI can show which key is configured.

The Langfuse fanout read path decrypts the tenant secret before export. Adds
secretKeyFingerprint to langfuseConfigSchema and tests for the encrypt/redact
policy.

* fix(api): secure admin config secret handling

* fix(api): preserve encrypted langfuse config secrets

* fix(api): couple config secret fingerprint deletion

* fix(api): read langfuse fanout collector url from env

* fix(api): display langfuse secret key hint

* fix(api): remove langfuse secret fingerprint breadcrumbs

* fix(api): use langfuse destination keys for tenant config

* fix(api): remove langfuse config compatibility fallbacks

* refactor(api): simplify langfuse secret helpers

* refactor(api): simplify langfuse config secret handling

---------

Co-authored-by: Ravi Kumar L <ravi.lazar@clickhouse.com>
2026-07-14 08:19:55 -04:00
Danny Avila
520af663bc
🧵 feat: Background Tool Calls for Agents & Model Specs (#14197)
Some checks are pending
Docker Dev Branch Images Build / build (Dockerfile, lc-dev, node) (push) Waiting to run
Docker Dev Branch Images Build / build (Dockerfile.multi, lc-dev-api, api-build) (push) Waiting to run
GitNexus Index / index (push) Waiting to run
GitNexus Index / post-index (push) Blocked by required conditions
* 🧵 feat: Background Tool Calls for Agents & Model Specs

Opt-in, poll-based background tool execution. The model marks an eligible tool
call with `run_in_background: true`; the host executor registers a task, returns
a handle immediately (so the graph turn resolves), runs the tool as a detached
promise, and the model retrieves the result via a new `check_background_task`
poll tool. Host-side only — no `@librechat/agents` change.

- Opt-in mirrors `deferred_tools`: admin capability `run_in_background`
  (off by default) + per-tool `tool_options.run_in_background`.
- Model specs / ephemeral agents: `TModelSpec.runInBackground` /
  `TEphemeralAgent.run_in_background` synthesize per-tool options; both paths
  converge at `initializeAgent`.
- In-process task registry: scoped per user+conversation, idempotent by
  toolCallId (safe across resume/replay), capped, TTL-swept.
- Excludes direct-path / host-special / code-session tools. Subagents and push
  notifications are deferred follow-ups.

* 🩹 fix: Harden background tool calls (Codex review)

- Reliable per-agent execution gate: thread the injected `run_in_background`
  tool names from `initializeAgent` through `configurable.backgroundToolNames`
  (`toolRegistry` only reaches the executor for PTC/tool_search), fixing the
  silent no-op + unstripped-arg leak for ordinary event-driven tools.
- Enforce the per-tool opt-in at execution (`backgroundToolSet.has(name)`) so a
  non-opted-in tool can't be backgrounded via an extra arg.
- Gate the `check_background_task` interception on the run actually enabling
  background, so a user tool sharing that name still executes.
- Forward `backgroundToolsAvailable` to added-convo (multi-convo) agents.
- Exclude `web_search`/`file_search` from eligibility — their results are turned
  into user-visible attachments/citations only by the foreground toolEndCallback.

* 🩹 fix: Address Codex round 2 on background tool calls

- Idempotency scoped to run+turn: provider tool-call ids repeat across turns
  (e.g. `call_0`), so key the dedupe map by `runId::toolCallId` and sweep
  orphaned mappings — a later turn no longer collides with a retained task.
- Artifacts preserved: a backgrounded tool's artifact is processed through the
  same `toolEndCallback` as the foreground path (images/files/citations no
  longer silently dropped), best-effort/guarded.
- Forward the `run_in_background` capability to connected-agent discovery and
  subagent `processAgent` init, so a child agent's own event-driven tools work
  the same as when it runs as primary.
- Strip the injected flag on foreground calls of background-capable tools
  (the model may emit it as `false`) so strict MCP/action schemas don't reject.
- `check_background_task` list path returns metadata only (result_available /
  result_chars), never full results — prevents context overflow; the full
  result is returned only when a specific id is requested.

* 🩹 fix: Address Codex round 3 on background tool calls

- Exclude background-capable tools from eager execution (run.ts): a speculative
  eager dispatch of a `run_in_background` call could launch the detached task
  with partial/stale args, and that side effect can't be canceled.
- Reserve the `check_background_task` name: overwrite a colliding user/MCP tool
  with the host poll schema (with a warning) so the advertised schema matches
  the executor's interception instead of hijacking a mismatched tool.
- Don't inject background schemas into pure subagents (spawn-tool child graphs)
  whose tools don't reach the host interceptor; keep it for primary/added/
  connected agents. Subagent background is the durable follow-up.
- Thread `backgroundToolsAvailable` + `backgroundToolNames` through the
  OpenAI-compatible and Responses agent routes (was chat-only), so the same
  agent/model spec behaves consistently across surfaces.
- Exclude image-generation built-ins (dalle/flux/gemini_image_gen/image_gen_oai/
  image_edit_oai) — artifact-first tools whose files can't reliably attach to an
  already-saved turn when backgrounded.

* 🩹 fix: Address Codex round 4 on background tool calls

- Sanitize self-spawn subagent inputs: strip `run_in_background` + the
  `check_background_task` def from the parent AgentInputs reused for self-spawn,
  so the isolated child (direct/child-graph path) doesn't advertise a background
  schema it can't honor. The SDK resolver keeps a provided `agentInputs` even
  with `self: true`.
- Exclude `check_background_task` from PTC (`run_tools_with_code`) tool
  definitions — it's host-only and not callable from generated code.
- Parse stringified JSON args before deciding background dispatch and before
  stripping the flag, so string-delivered `run_in_background` is honored and
  never leaks to strict object-schema tools.
- Skip injection for tools that already declare their own `run_in_background`
  param (would otherwise hijack/strip it), and for non-object (string-input)
  schemas (would otherwise rewrite the input contract).

* 🩹 fix: Address Codex round 5 on background tool calls

- check_background_task now parses stringified JSON args, so providers that
  deliver args as a string can retrieve a specific task by id (not just list).
- Include agentId in the background dedupe key (`agentId::runId::toolCallId`):
  two agents in the same run emitting the same provider id (e.g. `call_0`) now
  launch independent tasks instead of colliding.
- Self-spawn sanitization also strips the background entries from the reused
  toolRegistry (not just toolDefinitions), so a child using tool_search/deferred
  loading can't rediscover the host-only run_in_background / check_background_task.

* 🩹 fix: Strip run_in_background from PTC target tool schemas (Codex round 6)

The PTC path already filtered out the host-only check_background_task poll tool
but still exposed target tool schemas with the injected `run_in_background` param
(the shared toolRegistry entries were mutated by applyBackgroundToolCalls). PTC
codegen doesn't go through the host background interceptor, so it could pass the
flag to an MCP/action tool (strict-schema rejection or silent foreground with no
poll). Sanitize the PTC toolDefs like the self-spawn path does.

* 🩹 fix: Sanitize background from explicit subagent inputs (Codex round 7)

A child agent reachable as a top-level/handoff agent is initialized WITH the
background capability, then reused as an explicit subagent via buildSubagentConfigs.
Round 4 only sanitized the self-spawn case; this now applies the same
stripBackgroundFromToolDefinitions/Registry to explicit child agentInputs when
`child.backgroundToolNames` is non-empty, so an isolated child graph doesn't
advertise a run_in_background / check_background_task contract it can't honor.

* 🩹 fix: Reap stuck/expired background tasks (Codex round 8)

- get() now sweeps before returning, so repeatedly polling a known
  background_task_id can't keep an expired completed task (and its retained
  result, up to 100k chars) alive past the one-hour completed TTL.
- sweep() now reaps `running` tasks older than a 30-min running TTL, marking
  them errored. Previously a detached call that never settled (hung network /
  lost MCP connection) held a running slot forever, exhausting the
  per-conversation cap and rejecting every later dispatch.

* 🩹 fix: Evict oldest settled tasks instead of blocking at the cap (Codex round 9)

Only the running-task cap gates dispatch now. The total-tasks cap
(MAX_TASKS_PER_BUCKET) bounds memory but no longer rejects new background calls:
when full, it evicts the oldest settled (completed/error) tasks to make room.
Previously 200 quick background calls in one conversation would block all new
dispatches for up to the completed-task TTL, since polling doesn't remove settled
tasks. Running is already capped, so room always frees.

* 📝 docs: Frame background tool calls as within-turn (Codex P1 contract)

Codex escalated the request-lifecycle findings to P1 on the grounds that the
advertised "poll later" contract can't be honored for genuinely long-running
calls (request-scoped MCP connections + the run abort signal are torn down at
turn end). Align the model-facing contract with what the same-run implementation
actually delivers: the run_in_background param, check_background_task, and the
dispatch handle now instruct the model to collect the result WITHIN THE SAME TURN
(backgrounded work isn't guaranteed to survive past the turn). This is
within-turn parallelism; cross-turn survival of long-running calls remains the
deliberate durable subagent follow-up. Copy/comment-only; no behavior change.

* ♻️ refactor: Cross-turn background tool calls, leak-free

Extend background tool calls from within-turn to cross-turn on a single
process, since the mechanism already supports it: the run's abort signal
never reaches the detached invoke (the graph forwards only configurable/
metadata to the tool-execute handler), so the floating promise keeps
running past turn completion and its result stays in the in-process
registry for a later turn to poll (get/list key only on
user::conversation + id, never the dispatch run/turn).

Guarantee no connection leak: ephemeral request-scoped MCP tools (runtime
{{LIBRECHAT_BODY_*}} placeholders) capture their request-scoped store at
creation and fall back to it, so config manipulation can't redirect them;
their connection is torn down at request end. Tag such tools in
createToolInstance and run them in the foreground instead of backgrounding
them. Pooled/app-level MCP and structured tools are unaffected and survive
cross-turn via their managed pools.

Reword the model-facing contract (run_in_background, check_background_task,
handle message, fileoverview) from within-turn to cross-turn on this server
(not across restart/replica, which stays the durable follow-up).

Tests: cross-turn poll retrieval; ephemeral MCP tool runs foreground.

* 🐛 fix: Guard ephemeral MCP tag against a null server config

createToolInstance can be reached with a null/stale capturedServerConfig
(cached availableTools + getServerConfig returns null, as several MCP unit
tests construct tools). The new unconditional requiresEphemeralUserConnection
call then dereferenced config.source and threw during tool construction
(CI: Tests api shard 2/3). Guard with the same serverConfig ? ... : false
pattern the other callers use; a missing config is not request-scoped.

* 🎨 fix: Deliver backgrounded tool artifacts on the poll turn

A slow backgrounded MCP/action tool resolves after its dispatch turn is
finalized: createToolEndCallback only appends to that turn's artifactPromises
(already awaited) and writes to a closed stream, so the artifact (file/citation/
UI resource) was silently dropped — check_background_task recorded only the
hasArtifact boolean. The cross-turn contract made this the common case.

Hold the artifact on the task and deliver it through the LIVE poll turn's
toolEndCallback the first time check_background_task collects that id (once,
then cleared to free memory), attributed to the original tool. Same-turn and
cross-turn now share this path since the model must poll to collect any result.

Tests: registry claim-once; artifact delivered on poll not dispatch, idempotent.

* ✨ feat: Agent-builder toggle for background tool calls + cap tool descriptions

Add a per-MCP-tool "run in background" toggle in the agent builder, mirroring
the programmatic/deferred pattern: gated on the admin `run_in_background`
capability via useAgentCapabilities, read/written on tool_options[id]
.run_in_background through useMCPToolOptions (per-tool + bulk mark-all), and
rendered as a Zap toggle in MCPToolItem and McpSection with new locale keys.

Also cap the section tool/server descriptions (McpSection, ToolSection,
SkillSection) with max-h-40 overflow-y-auto so a long description scrolls
instead of overflowing the dialog, matching MCPToolItem's existing cap.

Tests: MCPToolItem renders/toggles the background button only when enabled.

* 🧪 fix: Mock new background hook functions in McpSection spec

* 🎨 fix: Restore background artifact when poll-turn delivery fails

* 🛡️ fix: Harden background tool call edges from review findings

- Error immediately (matching foreground) when a background-requested tool
  failed to load, instead of returning a success handle for a dead task
- Exclude ephemeral request-scoped MCP tools at injection time so the model
  never sees a run_in_background param the executor would silently downgrade;
  flip the execute-time tag to fail closed on a missing server config
- Source image-tool background exclusions from the shared imageGenTools set
  (adds missing stable-diffusion, an artifact-first live tool) instead of a
  hand-copied list
- Add check_background_task to the eager-execution exclusion list: artifact
  collection is a one-shot claim that must not fire from a speculative
  snapshot the SDK may discard
- Strip an imitated run_in_background arg on tools the executing agent never
  opted in (multi-agent history bleed), unless the tool's own schema declares
  the parameter
- Truncate oversized stored results with an explicit marker via the shared
  truncateMiddle (moved to utils/text) instead of a silent slice
- Document the at-most-once artifact delivery semantics honestly (the
  callback's downstream persistence is fire-and-forget, as in foreground)

* ♻️ refactor: Deduplicate background tool-call plumbing and tighten types

- Use the SDK's JsonSchemaType instead of a local duplicate; drop all
  as-unknown casts and type the poll-tool serializer explicitly
- Drop derivable BackgroundTask state (progress, hasArtifact) and the dead
  `enabled` param/return on applyBackgroundToolCalls (guarded at the call
  site), which also skips the defs pass when nothing opted in
- Fold the enable expression into synthesizeBackgroundToolOptions so the
  three load/added call sites can't drift
- Throttle the registry's all-buckets sweep and always sweep the accessed
  bucket, so a hot poll loop is no longer O(total tasks server-wide); bound
  retained artifact memory with a size cap
- Single-pass stripBackgroundFromToolDefinitions; pass metadata through to
  the poll-turn callback instead of a no-op reconstruction
- Collapse the client's copy-pasted boolean option families into a keyed
  factory (also removes the shared-object mutation in the bulk toggles) and
  the six toggle-button copies into one OptionToggle component

* 🧪 test: e2e coverage for cross-turn background tool calls

Proves the full contract through the real pipeline (mock harness): an agent
opts an MCP tool in via tool_options.run_in_background, the model dispatches
it detached and receives the synthetic handle while the tool is still running
(status=running in the rendered ack — the non-blocking guarantee without
timing assertions), the tool completes after its turn finalized, and a later
user turn recovers the task id from replayed history, polls
check_background_task, and renders the collected result.

- fake-mcp-server: slow_echo fixture tool (delayed echo)
- fake-model: E2E_BACKGROUND_DISPATCH / E2E_BACKGROUND_COLLECT markers
- e2e yaml: agents capabilities = defaults + run_in_background

* 🔧 fix: Close two background capability gaps from review

- Thread backgroundToolsAvailable through the OpenAI-compatible service
  (derived from app capabilities like codeEnvAvailable/statefulSessions),
  so agents with tool_options.run_in_background keep the feature on that
  route; fold the three capability derivations into one helper
- Index ephemeral MCP servers by normalizeServerName when excluding tools
  from background injection: tool names embed the normalized server name
  while mcpConfig keys the original, so exotic server names previously
  escaped the injection-time exclusion

* 🛂 fix: Fall back to configurable user identity for background task scoping

The in-repo routes merge req into the tool-execute configurable, but external
hosts of the exported OpenAI-compatible service inject their own loadTools and
may not — tasks would then register under an empty user id, collapsing
registry isolation to conversationId alone. Resolve the scoping id from
req.user.id, then configurable.user_id / user, and cover the isolation with a
foreign-user not_found test.

* 🧹 chore: Apply repo import sorter to PR-touched files
2026-07-13 12:51:36 -04:00
Danny Avila
b3f9cddbef
🧠 feat: Add GPT-5.6 reasoning.mode + reasoning.context (Responses API) (#14233)
Follow-up to #14206 (issue #14203 items 2-4). Adds two OpenAI Responses API
reasoning parameters that ride inside the `reasoning` object:

- reasoning_mode: standard | pro
- reasoning_context: auto | current_turn | all_turns

Wired end-to-end mirroring reasoning_summary: zod schema + query/base picks,
UI SettingDefinitions (openAI + openAICol2), data-schemas types, i18n, and
the backend (hasReasoningParams/getReasoningObject/applyReasoningConfig +
getOpenAILLMConfig threading + dropParams cleanup via removeReasoningField).
Responses-API-only: they flow into llmConfig.reasoning (OpenAI) or
modelKwargs.reasoning (custom useResponsesApi), and are excluded from
Chat Completions and OpenRouter.

Per-model gating (hiding pro/max where unsupported) and persisted reasoning
(#14203 item 5) remain separate follow-ups.
2026-07-13 09:52:42 -04:00
Danny Avila
53e369fba8
🧪 feat: stateful_code_sessions capability for warm Code API sandbox sessions (experimental) (#14150)
* ✨ feat: stateful_code_sessions capability for warm Code API sandbox sessions

Wire the @librechat/agents stateful sandbox sub-config behind a new,
off-by-default stateful_code_sessions agent capability. createRun sets
toolExecution.sandbox.statefulSessions when code execution is active in the
run AND the capability is enabled; execute_code and bash_tool factories get
the param so their descriptions hedge toward persistence. Rides the existing
variable-not-literal runConfig pattern, so it no-ops until @librechat/agents
is bumped to the version shipping the sandbox sub-config.

* ✨ feat: per-agent stateful code sessions (builder toggle + init gating)

Stateful sessions now require the agent's own opt-in, not just the admin
capability. New agent field stateful_code_sessions (schema + validation +
types) surfaces as a toggle in Agent Builder Advanced settings, gated on
the app capability and disabled without Code Interpreter. initializeAgent
resolves the per-agent truth (admin capability AND builder opt-in AND
code env) once: the registered bash_tool description, the execute_code
factory, and createRun's toolExecution.sandbox gate all read the same
resolved value. statefulSessionsAvailable threads through the same call
sites as codeEnvAvailable, including handoff discovery and added convos.

* 🐛 fix: propagate runtime_session_hint to sandbox executor in event-driven tool path

The event-driven ON_TOOL_EXECUTE handler built config.toolCall without the
resolved runtime_session_hint, so BashExecutor/CodeExecutor never sent
runtime_session_hint to the Code API. Every conversation then collapsed onto
the server-derived default session (no per-conversation isolation). Copy
tc.runtimeSessionHint onto toolCallConfig._runtime_session_hint, mirroring the
SDK direct-execution path.

* 🐛 fix: address Codex review findings for stateful code sessions

- OpenAI-compatible service (packages/api/src/agents/openai/service.ts) now
  derives and passes statefulSessionsAvailable alongside codeEnvAvailable, so
  the feature activates on that route (previously statefulCodeSessions resolved
  false there and createRun never sent toolExecution.sandbox).
- Thread runtime_session_hint through the host file-authoring tools
  (create_file/edit_file/read_file): those host branches return before the
  generic tool path, so readSandboxFile/writeSandboxFile now forward the
  per-conversation hint instead of falling back to the Code API default session.
- StatefulSessions builder toggle clears its form value when Code Interpreter is
  disabled, so a saved agent matches the disabled UI and re-enabling code doesn't
  silently reactivate stateful sessions.

* 🐛 fix: normalize stateful_code_sessions on save when Code Interpreter disabled

Addresses Codex review (round 2): a stale `stateful_code_sessions` opt-in
could persist when Code Interpreter (`execute_code`) is disabled from the
main agent builder without opening Advanced settings, silently reactivating
warm sessions if code was later re-enabled.

- AgentPanel: normalize in `composeAgentUpdatePayload` (the always-run save
  path) so `stateful_code_sessions` is forced to `false` whenever
  `execute_code !== true`, regardless of whether Advanced was opened.
- StatefulSessions: revert the mount-scoped useEffect (round-1 approach) —
  it only fired while the Advanced panel was mounted, missing this path.
- Add spec coverage for both branches of the normalization.
2026-07-12 08:12:04 -04:00