Commit graph

65 commits

Author SHA1 Message Date
Marco Beretta
b1591d291b
fix: Guard Mongo Cleanup In Api Services Tests (#17990) 2026-11-11 11:10:39 +01:00
Dustin Healy
9aa81cf1ab
🔐 fix: Redact MCP OAuth Diagnostics (#17603)
* 🔐 fix: Redact MCP OAuth Diagnostics

* fix: Keep OAuth Resource Credentials Out of Diagnostics

* test: Type the OAuth Completion Diagnostic Fixture
2026-11-10 21:04:00 -08:00
Marco Beretta
70dfb8abe0
🔒 fix: Gate GET Routes That Start OAuth Flows Behind Terms Acceptance (#17935) 2026-11-09 15:35:24 +01:00
lia-by-librechat[bot]
ba78cf5f5f
🫢 fix: Keep OAuth Credentials Out of Logs (#17409)
* 🫢 fix: Keep OAuth Credentials Out of Logs

* test: Match MCP OAuth SDK Fixture Types

* fix: Sanitize OAuth Failures Across Callback and Refresh Boundaries

* fix: Close OAuth Initiation Log Leaks and Preserve Vendor Recovery

---------

Co-authored-by: Lia <lia@librechat.ai>
2026-10-24 09:43:06 +02:00
lia-by-librechat[bot]
5a472bfef2
🙅 fix: Deny HTTP Access to System MCP OAuth Flows (#17408)
Co-authored-by: Lia <lia@librechat.ai>
2026-10-24 09:01:02 +02:00
Danny Avila
8c51a3f9f8
🙈 feat: Let Config Profiles Hide librechat.yaml MCP Servers and Remove Their Fields (#17327)
* 🙈 feat: Let Config Profiles Hide librechat.yaml MCP Servers and Remove Their Fields

Config profiles are the `__base__`, role, group and user documents written through `/api/admin/config`. A profile can tombstone `mcpServers.<name>`, and the merge then removes that server from the principal's resolved `mcpConfig`.

* 🛡️ fix: Apply MCP Hiding to Server Instructions, Failed Removals and the Startup Audit

- Server instructions bound for the model now leave out librechat.yaml servers hidden from the acting user, including one a model spec pins. `applyContextToAgent` passes the request user through `formatInstructionsForContext`, which drops the user's hidden set from the user-less catalog it reads.
- A config-tier candidate that removes a librechat.yaml value (a field, or a key of a record such as `headers`) and fails inspection now stands as a failed config-source entry instead of falling back to the librechat.yaml base that still carries the removed value. A failed change that removes nothing still falls back, and an entry with no transport (fields written over a hidden server) keeps the documented librechat.yaml fallback.
- The startup tombstone report reads every tenant's profiles in the system tenant scope, so it also runs from entry points without an outer system scope under strict tenant isolation, and names each profile's tenant.

* 🗂️ fix: Project the Startup MCP Tombstone Report Query and Run It Alongside Connections

* 🔑 fix: Keep Removed OAuth Headers Out of the MCP OAuth Callback

* 📝 docs: Note That Removed OAuth Headers Stay Out of MCP Sign-In

* 🔐 fix: Recheck the MCP OAuth Flow's Server Before Initiation Redirects

OAuth initiation redirected to a flow's stored authorization URL, or started
a new authorization from the flow's stored server URL, without resolving the
server again, so a principal whose profile hid the server after the flow was
created could still be sent to its authorization endpoint. The callback's
active-server check and its OAuth header fallback lived in the CJS route.

resolveMCPOAuthTarget (packages/api/src/mcp/oauth/target.ts) now owns both:
it resolves the server in the principal's current config with a strict read,
refuses a flow admitted for another server, a server the principal can no
longer use, or a redefined one, and selects the headers the flow sends (the
initiation snapshot, else the principal's config). Initiation calls it before
either redirect (404 or 400); the callback calls it before the exchange and
from the token-store fence.

* 📝 docs: Note That a Hidden MCP Server Ends OAuth Sign-Ins Already Started

* 🚩 feat: Publish the yamlMcpHide Admin Config Meta Flag

Profiles can hide librechat.yaml MCP servers and a hidden server is an
access boundary, so GET /api/admin/config/meta now reports
features.yamlMcpHide for the admin panel to gate its hide UI on. The
meta.spec probe checks the behaviour itself: the admin API stores an
mcpServers.<name> tombstone, the merged config drops the server, the
registry withholds it from principal lookups and checkout but not from
app-level lookups, process-backed servers stay, and an unreadable
principal config withholds every hideable server.

* 🧪 test: Align MCP Tombstone Specs With the Portable Config Store

The admin config handlers now write whole documents through
writeConfigDocument, so the end-to-end tombstone spec wires it in place
of the removed upsertConfig/patchConfigFields deps. A reset of a whole
MCP record now also prunes the parents it leaves empty.
2026-10-22 14:10:31 +02:00
Marco Beretta
9adabde72e
🗂️ feat: Audit MCP, Code Worker, Langfuse, Skill Credential, Passkey and Audit Log Access (#17054)
Some checks are pending
Backend Unit Tests / Build packages (push) Waiting to run
Backend Unit Tests / Codegraph select (push) Waiting to run
Backend Unit Tests / TypeScript type checks (push) Blocked by required conditions
Backend Unit Tests / Circular dependency checks (push) Waiting to run
Backend Unit Tests / Tests: api (shard 1/3) (push) Blocked by required conditions
Backend Unit Tests / Tests: api (shard 2/3) (push) Blocked by required conditions
Backend Unit Tests / Tests: api (shard 3/3) (push) Blocked by required conditions
Backend Unit Tests / Tests: data-provider (push) Blocked by required conditions
Backend Unit Tests / Tests: data-schemas (push) Blocked by required conditions
Backend Unit Tests / Tests: @librechat/api (shard 1/4) (push) Blocked by required conditions
Backend Unit Tests / Tests: @librechat/api (shard 2/4) (push) Blocked by required conditions
Backend Unit Tests / Tests: @librechat/api (shard 3/4) (push) Blocked by required conditions
Backend Unit Tests / Tests: @librechat/api (shard 4/4) (push) Blocked by required conditions
Codegraph E2E Votes / vote (full suite) (push) Waiting to run
Docker Dev Branch Images Build / publish (push) Waiting to run
Frontend Unit Tests / Client build recovery regression (push) Waiting to run
Frontend Unit Tests / Codegraph select (push) Waiting to run
Frontend Unit Tests / Build packages (push) Waiting to run
Frontend Unit Tests / TypeScript type checks (client) (push) Blocked by required conditions
Frontend Unit Tests / Tests: @librechat/chat (push) Blocked by required conditions
Frontend Unit Tests / Tests: @librechat/client (push) Blocked by required conditions
Frontend Unit Tests / Tests: Ubuntu (shard 1/2) (push) Blocked by required conditions
Frontend Unit Tests / Tests: Ubuntu (shard 2/2) (push) Blocked by required conditions
Frontend Unit Tests / Vite build verification (push) Blocked by required conditions
Sync Locize Translations & Create Translation PR / Sync Translation Keys with Locize (push) Waiting to run
Sync Locize Translations & Create Translation PR / Create Translation PR on Version Published (push) Blocked by required conditions
* fix: Remove Group Dependents Before Deleting the Group

* feat: Audit Code Worker Pairing and Langfuse Connection Changes

* feat: Audit Skill Sync Credential Changes

* feat: Audit Passkey Enrolment and Removal

* feat: Audit Audit Log Reads and CSV Exports

* feat: Audit Audit Log Retention Purges

* feat: Audit MCP Server Create, Update and Delete

* fix: Keep Purge Result When the Purge Audit Write Fails

* test: Cover Admin Integration Audit Scenarios

* fix: Tighten Admin Integration Audit Scope and Group Cleanup Order

* fix: Run Security Fences Before Awaiting Audit Writes

* fix: Move MCP Post-Commit Audit Ordering Into the Auditor and Audit Verify Reads

* fix: Fail Closed When Recording Audit Log Access

* fix: Audit Group Deletion Before the Cascade and Restore Dependents on Failure

* fix: Restore Only Config Overrides When Group Deletion Fails

* fix: Audit Worker Pairing and Export Before the Work and Append Failures

* fix: Record Group Deletion as Pending Without Delaying Revocation

* fix: Skip Config Restore When a Concurrent Delete Removed the Group

* fix: Restore Group Config When the Existence Recheck Fails

* fix: Keep Group Config Restricted When Delete State Is Unknown

* fix: Record Worker Pairing and Revocation as Pending Until Confirmed

* fix: Deliver the Pairing Code Before the Terminal Audit Write

* test: Assert Pairing Code Delivery Precedes the Terminal Audit Write

* fix: Send Revocation and Group Deletion Responses Before the Terminal Audit Write

* fix: Snapshot Group Config Before the Destructive Cascade Step

* refactor: Type the Group Config Snapshot Dependency Without Mongoose Shapes

* fix: Fence Shared MCP Users Before the Deletion Audit and Correlate Audit Attempts

* fix: Diff MCP Server Audits Against the Committed Config

* fix: Read the Committed MCP Config at Commit Time for the Audit Diff

* refactor: Start the Committed MCP Config Read Inside the Auditor

* fix: Audit Langfuse Patches When Activation Fails and Surface Failed Group Config Restores

* fix: Record Member Removals When Group Deletion Completes After a Failed Restore

* test: Serve the Committed Config on the MCP Update Audit Reread

* fix: Audit the Group Returned by the Compensating Deletion

* fix: Keep Member Removal Audits Fail-Open During Group Deletion

* fix: Retry ACL Cleanup After a Compensating Group Deletion

* fix: Mark Unavailable Audit Detail and Order Member Removals After the Pending Deletion

* fix: Guard Synchronous Committed-Config Read Failures in the MCP Auditor

* fix: Return the Persisted MCP Config and Audit Commits That Outlive Cache Invalidation

* style: Sort Imports in MCP Audit Files

* style: Sort Imports After Restack

* fix: Audit MCP Commits That Outlive Any Post-Write Step and Pend Audit Exports Until Complete

* test: Provide the Fail-Closed Setting in the Code Route Mock

* fix: Refresh MCP Read Caches When a Repository Step Fails After the Write

* fix: Finish Audit Exports on Response Completion and Keep Error Details Out of Messages and Logs

* fix: Watch Response Completion Before the Truncation Footer Write

* fix: Skip Audit for No-op MCP Deletes and Close Remaining Post-Failure Audit Gaps

* fix: Keep Causes Out of Structured Logs and Treat a Lost Compensating Delete as Not Found

* fix: Mark Failed MCP Post-Commit Steps and Null Langfuse Activation in Audits
2026-10-15 00:30:26 +02:00
lia-by-librechat[bot]
19a337da46
🛖 fix: Scope OAuth Binding Cookies to the Public API Path (#16649)
* 🛖 fix: Scope OAuth Binding Cookies to the Public API Path

* fix: Canonicalize OAuth Callback and Cookie Paths Together

---------

Co-authored-by: Lia <lia@librechat.ai>
2026-10-02 13:21:49 -04:00
Danny Avila
d349120ec3
🪢 fix: Restore MCP OAuth Browser Binding (#16634)
* fix: Require Browser Binding for MCP OAuth Callbacks

* fix: Refresh OAuth Browser Binding After Account Switch

* fix: Reject Malformed OAuth Binding Cookies Safely
2026-10-01 15:25:46 -04:00
lia-by-librechat[bot]
e396ea9c7b 🫸 fix: Gate MCP App Views and Actions on User Intent (#16297)
* 🫸 fix: Gate MCP App Views and Actions on User Intent

* 🧪 ci: Run MCP Apps Browser Checks on Repository Bot PRs

* 🧲 test: Open App Before Checking Tool-Call Frames

* 🔎 fix: Wait for Stable App Views in Restored Browser Tests

* 🧪 test: Trace Grouped App View Activation in Browser CI

* 🧪 test: Diagnose Grouped App Click Propagation

* 🧪 test: Distinguish Native and React App Click Delivery

* 🧪 test: Inspect Grouped App Button Hit Target

* 🎟️ test: Open Grouped App Views After Scrolling Into Place

* 🧯 fix: Revoke App Approval Before Teardown Handshake

* 🫸 fix: Configure App Limits and Suspend Denied Actions

* 🧾 test: Expect Configured MCP App Limits in API Suites

* 🧱 fix: Isolate Approved App Messages From Composer Drafts

* 🛟 fix: Suspend Aborted App Actions and Restore View Focus

* 🦮 fix: Preserve Focus on App-Initiated Teardown

* 🧳 fix: Exclude Draft Agent State From App Messages

* 🧭 test: Preserve Legacy Null File Overrides

* 🧰 fix: Retain Selected MCP Tools Across App Messages

* 🧾 fix: Acknowledge Accepted App Turns and Disclose Selected Tools

---------

Co-authored-by: Lia <lia@librechat.ai>
(cherry picked from commit 6866732a88)

Original-PR: https://github.com/LibreChat-AI/LibreChat/pull/16297
2026-10-01 03:46:17 +00:00
Dustin Healy
7b27e7941e 🧩 feat: MCP Apps support (#13831)
* feat: add MCP Apps support

* fix: preserve MCP App results and enforce sandbox policy

* fix: bind persisted MCP Apps to their server configuration

* fix: complete MCP Apps runtime integration

* fix: type the optional MCP App binding codec explicitly

* test: provide MCP Apps policy in bridge mocks

* fix: pass admitted App policy to the tool call owner

* fix: align MCP Apps CI contracts

* test: preserve MCP abort reasons in recovery mocks

* fix: preserve MCP compatibility across policy profiles

* test: use profiled MCP catalog generations

* test: expect the standard MCP session profile

* fix: remove the unused MCP abort import

* test: expect profiled MCP fixture arguments

* test: seed profiled MCP manager fixtures

* Fix request-header bearer resolution after canary merge

* Preserve MCP App views across canary message updates

* Document message-owned MCP App placement

* ci: Recheck MCP Apps Against Canary Base

---------

Co-authored-by: Lia <lia@librechat.ai>
(cherry picked from commit a98d40f61c)

Original-PR: https://github.com/LibreChat-AI/LibreChat/pull/13831
2026-10-01 03:46:16 +00:00
Danny Avila
1e0076376c
🪢 fix: Bind Preserved MCP API Keys to Server Configuration (#16136)
* fix: bind preserved MCP API keys to server configuration

* fix: surface MCP API key rebinding errors

* test: use typed MCP OAuth re-entry error
2026-09-20 12:33:20 -04:00
lia-by-librechat[bot]
2452f499a8
🗼 fix: Serialize MCP OAuth Token Refresh Across Replicas (#15993)
Some checks failed
Backend Unit Tests / Build packages (push) Waiting to run
Backend Unit Tests / Codegraph select (push) Waiting to run
Backend Unit Tests / TypeScript type checks (push) Blocked by required conditions
Backend Unit Tests / Circular dependency checks (push) Waiting to run
Backend Unit Tests / Tests: api (shard 1/3) (push) Blocked by required conditions
Backend Unit Tests / Tests: api (shard 2/3) (push) Blocked by required conditions
Backend Unit Tests / Tests: api (shard 3/3) (push) Blocked by required conditions
Backend Unit Tests / Tests: data-provider (push) Blocked by required conditions
Backend Unit Tests / Tests: data-schemas (push) Blocked by required conditions
Backend Unit Tests / Tests: @librechat/api (shard 1/4) (push) Blocked by required conditions
Backend Unit Tests / Tests: @librechat/api (shard 2/4) (push) Blocked by required conditions
Backend Unit Tests / Tests: @librechat/api (shard 3/4) (push) Blocked by required conditions
Backend Unit Tests / Tests: @librechat/api (shard 4/4) (push) Blocked by required conditions
Codegraph E2E Votes / vote (full suite) (push) Waiting to run
Docker Dev Branch Images Build / publish (push) Waiting to run
Docker Dev Images Build / publish (push) Waiting to run
Sync Helm Chart Tags / Ignore non-main push (push) Waiting to run
Sync Helm Chart Tags / Sync chart tags (push) Waiting to run
Frontend Unit Tests / Client build recovery regression (push) Has been cancelled
Frontend Unit Tests / Codegraph select (push) Has been cancelled
Frontend Unit Tests / Build packages (push) Has been cancelled
Sync Locize Translations & Create Translation PR / Sync Translation Keys with Locize (push) Has been cancelled
Frontend Unit Tests / TypeScript type checks (client) (push) Has been cancelled
Frontend Unit Tests / Tests: @librechat/client (push) Has been cancelled
Frontend Unit Tests / Tests: Ubuntu (shard 1/2) (push) Has been cancelled
Frontend Unit Tests / Tests: Ubuntu (shard 2/2) (push) Has been cancelled
Frontend Unit Tests / Vite build verification (push) Has been cancelled
Sync Locize Translations & Create Translation PR / Create Translation PR on Version Published (push) Has been cancelled
* fix: Serialize MCP OAuth token refresh across replicas

`MCPTokenStorage.inflightRefreshes` coalesces refresh-token redemptions
inside one Node process. Behind a load balancer without session affinity,
one user's concurrent requests land on different replicas, each reads the
same not-yet-rotated refresh token and redeems it. RFC 9700 servers treat
the second redemption as replay and revoke the whole grant family, so the
user is asked to authorize the MCP server again.

`forceRefreshTokens` now takes a cross-replica refresh flight before
redeeming, using the same `FlowStateManager.acquireLease` primitive the
OAuth teardown fence uses under a distinct key. A replica that waited for
the flight adopts the tokens the holder rotated instead of redeeming
again; when the flight is still held after the wait window it falls back
to the unfenced redemption every earlier release performed.

* fix: Redeem unfenced when the refresh flight lease store fails

* fix: Fence MCP refresh waiters instead of redeeming unfenced

A waiter that could not take the cross-replica refresh flight fell through to
an unfenced redemption after 10s, while the holder's own stale abort does not
fire until 60s. A refresh taking 11 to 60 seconds therefore still let two
replicas redeem one refresh token, the replay this fence exists to prevent.

The waiter now polls until it either adopts the tokens the holder rotated or
acquires the flight, and fails as MCPTokenRefreshUnavailableError rather than
redeeming beside a live holder. `getTokens` callers defer connection recovery
on that error, and the stored credential is left intact for a later attempt.

The flight is keyed by the stored credential (tenant, user, server name)
rather than the caller's OAuth binding digest, so a rolling config change
cannot hand two replicas different locks over one stored token. The wait is
an operator lever, `oauthRefreshWaitTimeout`, clamped to half the stale
window. An adoption read that fails no longer gives up the held flight.

* fix: Annotate derived refresh-flight constants for isolatedDeclarations

* fix: Recapture the credential generation a peer published on adoption

* fix: Hold the MCP refresh flight until redemption settles

Four corrections to the cross-replica refresh flight.

The flight lease equalled the window that aborts a stalled redemption, on
the claim that an expired flight could never belong to a redemption still
able to reach the token endpoint. Aborting proves no such thing: the
request may have been processed with its response lost, and a stalled
event loop can delay the abort past its own deadline. The lease now
outlives the abort, so a peer cannot redeem a credential the provider has
already rotated. A live replica still releases on settle, so the margin is
paid only by one that died holding the flight.

The credential snapshot was taken after the first failed acquisition, so a
holder that stored and released in that gap was snapshotted post-rotation:
the next attempt saw an unchanged record and redeemed the credential it
should have adopted. It is taken before the first attempt now.

`oauthRefreshWaitTimeout` accepted zero while the runtime mapped every
non-positive value to the default, so the config validated and then
behaved contrary to its value. Zero is rejected at load.

Adoption announced a second credential change through
`handleOAuthRefreshSuccess`, whose `onOAuthCredentialsChanged` advances
authorization state after persistence this replica did not perform. That
retired the generation recaptured beside it and fenced the build against
its own tool publication. Adoption now updates the local token-flow cache
and recaptures, without announcing.

* fix: Check every held MCP refresh flight for a peer's rotation

Four corrections, two of them consequences of moving the credential
observation ahead of the first lease attempt.

That move created a storage read inside `beginRefreshFlight`, and its
failure reached a handler written for a lease-store outage, so a transient
read error became an unfenced redemption beside a live peer. The flight
now handles its own reads: losing the observation costs adoption and
nothing else, and only the lease store failing may redeem unfenced.

The observation was also never compared when the first acquisition
succeeded. A peer that rotated and released before this replica contended
left an acquisition that looked uncontended, and its fresh credential was
redeemed a second time. Every acquisition now runs one rotation check,
which also collapses two code paths into one.

`getTokens` already loads the refresh record to decide a refresh is
needed, so it is passed on as the observation baseline instead of read
again, and the read taken under the flight is reused as the credential
redeemed. Two reads on a latency-counted path where there were three.

`runSilentRefresh` collapsed contention into null, sending the 401 path to
interactive OAuth, so a slow peer prompted the user to authorize a server
whose credential was about to be valid. The retryable outcome now travels
through the silent-refresh layers and the connection defers, matched by
name as well as identity because these errors cross the package boundary.

* chore: Regenerate the Agents OpenAPI spec for repositoryInstructions

The drift check added by #15928 fails on dev: #16008 added
`repositoryInstructions` to the agent schema without regenerating the
committed spec, so every branch merging dev inherits the failure. This is
the remedy the check itself prescribes and carries no hand-written change.
Drop this commit if dev regenerates first.

* fix: preserve MCP refresh outcomes across replica boundaries

* fix: keep MCP adoption fenced through publication

* fix: anchor MCP refresh adoption to rejected credentials

* fix: Complete OAuth Adoption Adapters and Isolate Legacy Flow Readers

* fix: Preserve OAuth Request Identity Through Recovery and Discovery

* fix: Fence OAuth Adoption Against All Credential Writers

* fix: Gate Coordinated OAuth Rollout and Preserve Unauthenticated Identity

* fix: Preserve OAuth Coalescing Across the Staged Rollout

* style: Sort OAuth Timeout Constant Import

* fix: Invalidate Both OAuth Token Flow Protocols on Rotation

* test: Complete the Rollback Token Flow Fixture

---------

Co-authored-by: Lia <lia@librechat.ai>
Co-authored-by: Danny Avila <danny@librechat.ai>
2026-09-16 21:58:23 -04:00
Danny Avila
6965f8e7d3
🛂 fix: Back Off MCP OAuth Discovery (#15783)
* fix: Back Off MCP OAuth Discovery

* fix: harden MCP discovery recovery state

* fix: synchronize MCP authorization recovery

* fix: fence MCP discovery authorization state

* fix: retain recovery generation through cache outages

* fix: close MCP authorization fence gaps

* fix: complete MCP authorization lifecycle

* fix: close MCP authorization recovery races

* fix: centralize MCP recovery lifecycle

* fix: fence ephemeral MCP catalog observers

* fix: close MCP authorization recovery races

* test: cover Redis token settlement race

* fix: prefer authoritative token settlement

* fix: harden MCP authorization retry recovery

* fix: back off stalled OAuth refreshes

* fix: bound durable authorization retries

* test: isolate authorization retry adapter

* fix: close MCP authorization recovery boundaries

* fix: defer agent authorization observers

* fix: close OAuth fence race windows

* fix: keep retry deferrals monotonic

* chore: sort startup hook imports
2026-09-09 22:21:34 -04:00
Danny Avila
601ed07870
🫧 fix: Retire Cancelled MCP OAuth State (#15702)
* fix: Clean Up MCP OAuth Teardown State

* fix: Harden OAuth Teardown Boundaries

* fix: harden MCP OAuth teardown races

* fix: avoid flow cache dependency cycle

* fix: bind OAuth teardown to credential generations

* test: cover active OAuth server fixtures

* style: sort OAuth cleanup imports

* fix: discard mismatched OAuth token snapshots

* fix: fence OAuth teardown across callback races

* test: align OAuth callback fixtures with server fencing

* test: complete OAuth callback server fixtures

* fix: version OAuth flow settlement

* fix: fence OAuth teardown snapshots

* test: align OAuth teardown fixtures

* test: complete OAuth snapshot fixtures

* fix: snapshot OAuth deletion lifecycle

* test: bind OAuth revocation snapshots

* test: assert per-record OAuth fence

* fix: harden MCP OAuth teardown consistency

* test: expect snapshotted OAuth readers
2026-09-07 15:43:02 -04:00
Danny Avila
49e1e32f3a
🧺 fix: Funnel MCP Catalog Work Through a Shared Gate (#15429)
* 🧯 fix: Bound MCP Catalog Work Across Requests

* 🛑 fix: Reject Saturated MCP Discovery

* fix: fairly bound MCP catalog work

* test: type catalog snapshot mock arguments

* test: assert snapshot deadline forwarding

* fix: harden MCP catalog scheduling
2026-08-31 14:13:31 -04:00
Dustin Healy
8fcab7e44f
🔄 fix: Recover Missing MCP Marketplace Catalogs (#15323)
* fix: recover missing MCP marketplace catalogs

* fix: make MCP catalog recovery passive

* test: type MCP catalog recovery fixtures

* fix: bound and back off passive MCP catalog recovery

Passive recovery runs inline on `GET /api/mcp/tools` and its results are
request-local by design, so every list request re-dialed the same cold
servers with the default connection timeout. Three limits keep that cost
proportional to what recovery can actually recover:

- Cap the discovery timeout at 5s instead of inheriting the connection
  default (`initTimeout ?? 30s`); a server configured to connect faster
  keeps its own shorter limit.
- Skip a server the config tier already marked `inspectionFailed`, leaving
  it to that tier's retry window rather than re-dialing it per request.
- Skip a server whose declared `customUserVars` are unset, matching the
  gate `reinitMCPServer` applies for issue #10969 — connecting without them
  fails auth, so the attempt is spent for nothing.

Servers that still fail discovery enter a one-minute per-process cooldown,
which is what stops an unreachable server from being re-dialed by every
subsequent list request. A server that recovers clears its own entry, and
expired entries are swept at most once per window so the map stays bounded.

Skipped servers render exactly as they did before recovery existed: present
in the catalog with an empty tool list.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Xr1Dabvdn1mzzyYgpJgU5B

* fix: bound passive MCP recovery by deadline, key cooldowns by config

Both follow-ups address the same mistake: recovery expressed its own
request-level constraints in terms borrowed from other layers.

`connectionTimeout` bounds one connection attempt, and
`MCPConnectionFactory.discoverToolsInternal` spends it twice — once on the
authenticated connection, then again in `attemptUnauthenticatedToolListing`
— so capping it bounded no total this layer could reason about. Recovery now
enforces its own wall-clock deadline per server with `withTimeout`, which
holds however many attempts the factory makes; `connectionTimeout` is left to
do only its own job, still honouring a shorter operator `initTimeout`. An
attempt abandoned by the deadline disposes its own connection when it
settles, and `Promise.race` keeps a handler on it, so a late rejection is
not unhandled.

A per-request budget now caps total recovery regardless of server count.
A server is dialed only if the remaining budget can fund a full deadline;
never dialing one is not evidence against it, so a skipped server records no
cooldown and a later request reaches it once those ahead are cached or
cooling down.

Cooldown identity now includes the publication generation — the same
effective-config identity the tool caches fence on — instead of just user and
server name. Correcting a server's URL or transport keys a new entry, so the
refetch the client issues on update is no longer skipped for up to a minute
by the previous configuration's failure.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Xr1Dabvdn1mzzyYgpJgU5B

* refactor: keep passive MCP recovery stateless and bounded by its own work

Reverts the cooldown, request budget and deadline race added in 257d5cf and
fc3e3c9, and keeps only the three stateless limits.

The tool cache refuses unfenced writes (`tools.ts`), and a discovery
connection owns no publication generation and is disposed, so a recovered
catalog cannot be retained by design. Those commits responded by building a
cache-shaped memory in front of it — per-process failure state, a scheduling
budget, an identity, an eviction sweep — and each round of review found
another way that hand-rolled cache differed from a real one: wrong identity
for configuration, wrong identity for credentials, no fairness across
requests, and a limiter slot released while its network operation was still
running. None of that machinery was asked for; all of it was compensation for
a result the architecture does not allow keeping.

Recovery is now stateless. It skips only what configuration alone proves
pointless — a server the config tier already marked `inspectionFailed`, and
one whose declared `customUserVars` are unset — and bounds the work itself
rather than racing it, so a limiter slot is held for exactly as long as its
network operation runs and the concurrency limit of three is real.

The attempt timeout is not a compromise: recovery exists for a server that is
reachable and authorized but whose catalog cache expired, and such a server
answers tools/list well inside 1.5s. Anything slower cannot be rescued here,
so failing fast costs nothing. The factory spends that value per attempt, so
a server's ceiling is it times the attempts made; the constant documents that
rather than hiding it behind a number tuned to today's attempt count.

Consequences that were bugs are now gone by construction: every cold server
is attempted on every request, so none is starved by those ahead of it, and
correcting a server's configuration or credentials takes effect on the next
refetch instead of waiting out a stale cooldown.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Xr1Dabvdn1mzzyYgpJgU5B

* fix: correct the inspection-failure skip and bound catalog fan-out

Three fixes that belong to this layer; a fourth issue does not, and is
described below.

The `inspectionFailed` skip was too broad. `MCPServersInitializer` stores a
YAML server that was unreachable at startup via `addServerStub`, which stamps
`source: 'yaml'`, and only config-tier entries get the timed retry in
`ensureSingleConfigServer`. Skipping every failed stub therefore hid a
recoverable server from the marketplace permanently — the exact state this
recovery exists to escape. It now defers only `source === 'config'`, matching
what `reinitMCPServer` already does.

Plugin auth is read only when some cold server actually declares
`customUserVars`, and only for those servers. The common unauthenticated case
no longer pays a MongoDB round trip whose result nothing can consume.

Snapshot refreshes are now bounded by the same limiter as discovery. They are
not local reads: both connection paths reach `fetchOrderedToolsSnapshot` and
issue a real `tools/list`, so a cache reset across many servers previously
burst unbounded outbound requests while discovery was capped at three.

Not fixed here, because it cannot be: `connectionTimeout` does not bound
discovery. It covers `connection.connect()` only, and `fetchToolsSnapshot`
then applies its own `TOOLS_LIST_TIMEOUT_MS` (30s) to `tools/list`, so a
server that connects fast and stalls while listing still holds its slot for
that window. The factory also does not cancel a timed-out connect before
starting the unauthenticated fallback. Bounding this end to end needs a
deadline threaded through `MCPConnectionFactory` into both `connect()` and
`fetchToolsSnapshot()`, which is a change to shared connection machinery
rather than to this caller. The constant's comment now states what it does
and does not bound instead of implying an end-to-end guarantee.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Xr1Dabvdn1mzzyYgpJgU5B

* fix: thread live-session OBO context into passive catalog discovery

The merge of #15334 sources OBO tokens from the live OpenID session via
request-boundary closures. Passive catalog recovery is a discovery call
site too; without these options an OBO server whose stored token went
stale fails recovery — the exact cold-catalog class this PR fixes.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Xr1Dabvdn1mzzyYgpJgU5B

---------

Co-authored-by: Claude <noreply@anthropic.com>
2026-08-30 06:55:30 -04:00
Danny Avila
fa913148fb
🔒 fix: Refresh MCP OBO Tokens From the Live OpenID Session (#15334)
* 🧊 fix: Inline-refresh OpenID session tokens at MCP OBO call time

Resolves the walk-away failure mode where MCP tool calls using OBO auth
fail with "No valid OpenID access token is available for OBO exchange"
after a user idles past their access-token lifetime. The strategy-time
snapshot on `user.federatedTokens` could expire mid-stream before
`resolveOboToken` ran, while `req.session.openidTokens` carried a still-
valid (or refreshable) token that nothing read.

- New OpenIDSessionRefresh service: per-user single-flighted closure that
  reads `req.session.openidTokens` at OBO time and inline-refreshes via
  `openid-client.refreshTokenGrant` when expired (30s skew), persisting
  via `req.session.save()`. No cookie writes (headers already flushed).
- `resolveOboToken` gains a required UpstreamTokenProvider parameter
  (typed as `() => Promise<OIDCTokens | null>`, reusing the shared shape
  from @librechat/data-schemas). Compile-time guarantee that every call
  site is updated.
- New `session_refresh_failed` OboTokenResolutionReason distinguishes
  "session expired and IdP rejected refresh" from "no upstream token
  ever existed."
- `req` threaded through createMCPTool/createMCPTools/createToolInstance
  to construct the closure with captured request, plus fail-closed
  guards in MCPConnectionFactory.getOboTokens and MCPManager.callTool
  when the closure isn't plumbed.
- Startup warning in MCPServersInitializer when OBO is configured but
  OPENID_REUSE_TOKENS is unset (the strategy populating
  user.federatedTokens is only registered under reuse, so OBO would
  fail every call without it).

Tests: 16 new in OpenIDSessionRefresh.spec.js; obo.spec.ts extended
for the new param + error reason; wiring smoke tests in MCPManager,
MCPConnectionFactory, MCPServersInitializer, and MCP.spec.js.

* 🛡️ fix: Harden OBO inline-refresh against token type and session edge cases

- Token-preference asymmetry: live-token reuse and expires_at derivation
  now strictly gate on the access_token, not the id_token. Added a
  required `tokenPreference` parameter on isLiveSessionTokenStillValid,
  buildOIDCTokensFromSession, and createOpenIDSessionTokenProvider
  so every call site is explicit. Dropped the bogus id_token-exp
  fallback in performIdpRefresh — id_token TTL is governed by IdP
  session policy and would mark a short-lived access_token reusable
  past its real lifetime.
- Missing req in /reinitialize route: the manual reconnect
  endpoint now forwards req into reinitMCPServer, so OBO servers can
  build a session-aware upstream-token closure instead of failing with
  missing_upstream_token.
- Single-flight key collisions: composed key as
  tenantId:openidIssuer:openidId:sessionId via getSingleFlightKey.
  Concurrent calls in the same session still coalesce; separate sessions
  never share an in-flight refresh, preventing refresh-token rotation
  from breaking sibling sessions and preventing cross-tenant token
  crossover when distinct users share an IdP sub.
- Opaque access token reuse): persist accessTokenExpiresAt
  (unix seconds, from tokenset.expires_in) on each refresh AND on initial
  login / SPA refresh in setOpenIDAuthTokens. New getAccessTokenExp
  helper falls back to it when the access token isn't a JWT, avoiding
  redundant inline refreshes for Microsoft Graph and Auth0 default
  audiences.
- Log hygiene: the single-flight key (containing sessionId,
  openidId, openidIssuer, tenantId) is now SHA-256-hashed in the
  "Joining in-flight refresh" debug log. Preserves cross-line correlation
  via a 12-char prefix without leaking credential or PII material.

Documented req.session.openidTokens shape contract via JSDoc typedef so
the new accessTokenExpiresAt field has a discoverable home alongside the
existing accessToken/idToken/refreshToken/expiresAt/lastRefreshedAt.

Tests: OpenIDSessionRefresh.spec.js up to 30 passing (added coverage for
opaque-token reuse, JWT-access-token-exp fallback, no-id_token-fallback
regression, cross-session no-coalesce, persistence on refresh, and a
guard against stale accessTokenExpiresAt carryover). AuthService.spec.js
adds two cases covering accessTokenExpiresAt persistence on login.
mcp.spec.js (route) gains a regression test asserting req flows into
reinitMCPServer.

* 🔍 fix: Detect OBO-only MCP admin config overrides

Admin Config overlays for YAML-defined MCP servers compare only
ADMIN_CONFIGURABLE_FIELDS to decide whether to lazy-init a config-tier override.
The OBO config field was added after that fingerprint list, so an override that
only added or changed `obo` was treated as unchanged YAML and skipped.

Include `obo` in the admin-configurable field list and add a regression test for
an OBO-only override.

* 🔊 fix: Mock MCP OAuth timeout in SDK integration test

MCPConnectionFactory.attemptToConnect reads mcpConfig.OAUTH_HANDLING_TIMEOUT
when building the OAuth connection timeout. The SDK OAuth integration test
mocked mcpConfig without that field, which made the timeout calculation produce
NaN and caused the test to fail before the OAuth refresh/start path completed.

Add OAUTH_HANDLING_TIMEOUT to the test mock.

* ♻️ refactor: Pass OBO upstream-token closure into MCP instead of req

Build the OpenID upstream-token provider at the request boundary and thread
only the closure through MCP handling, so the MCP service layer no longer
receives the raw Express request. The closure still reads/refreshes the live
session at tool-call time, preserving the walk-away recovery.

- Drop `req`/`capturedReq` from createMCPTools, createMCPTool, reconnectServer,
  createToolInstance, and reinitMCPServer; forward `upstreamTokenProvider`
  instead. Closure is constructed in loadTools, loadToolDefinitionsWrapper, and
  the reinitialize route, where req/res are in scope.
- OBO: fall back to user.federatedTokens when the provider yields no live
  session, so OIDC remote-agent calls (verified bearer, no session) still work.
- Inline refresh: mirror a rotated refresh token to the refreshToken cookie via
  a shared setRefreshTokenCookie helper, guarded by !res.headersSent (no-op on
  the streaming path; session copy stays authoritative).
- Single-flight: hydrate a joining request's own session from the resolved
  tokens so a later OBO call doesn't replay a rotated-away refresh token.

Addresses owner feedback and three review findings.

* 🔒 fix: Recover OIDC refresh-token rotation after SSE OBO refresh

When an inline OBO refresh rotates the OpenID refresh token after SSE headers
have already been sent, the browser refreshToken cookie cannot be updated. Store
a short-lived encrypted bridge from the stale cookie token to the rotated token
so /api/auth/refresh can recover after express-session loss.

Use the signed openid_user_id cookie to load user context for bridge validation,
retry only on invalid_grant, and delete the bridge only after the bridged refresh
succeeds.

* 🔨 fix: hydrate joined OIDC refresh sessions with stable refresh tokens

Update single-flight OIDC refresh joiners whenever refreshed access token
state changes, even if the IdP keeps the refresh token unchanged.

This prevents joined requests from retaining stale accessToken or
accessTokenExpiresAt values and redundantly refreshing later in the same run.

* 🌉 Persist OIDC refresh-token recovery bridges in MongoDB

Store SSE OBO refresh-token recovery bridges in MongoDB instead of
process-local memory so /api/auth/refresh can recover after worker
restarts or cross-worker routing.

Derive bridge expiry from REFRESH_TOKEN_EXPIRY so the recovery window
matches the stale refreshToken cookie it repairs, and delete bridges
after successful recovery.

* 🤝 Coordinate OIDC inline refreshes across workers

Add a short-lived Mongo-backed refresh-flight record so concurrent
OBO refreshes for the same OpenID session do not redeem the same
rotating refresh token on different workers.

The winning worker performs the IdP refresh and stores an encrypted
result; joiners wait for that result, hydrate their request session,
and return without calling the IdP.

* ⚓ Keep OpenID marker cookies aligned on inline refresh

Refresh token_provider and openid_user_id with the same expiry as the
rotated refreshToken cookie when an inline OBO refresh can still write
headers.

Share the marker-cookie writer with the normal OpenID auth refresh path
so the fallback /api/auth/refresh branch continues to recognize valid
OpenID refresh tokens after session expiry.

* 🔑 fix: include refresh token in OIDC local refresh flight key

Key the process-local OIDC refresh coalescing by the current session
refresh token, matching the Mongo-backed flight key. This prevents a
request with a newly rotated token from joining an older pending refresh
and inheriting its failure/result.

* 🌉 fix: store OIDC refresh bridge without cookie response

Treat missing or non-cookie responses like headers-sent streaming
responses during inline OIDC refresh. When the IdP rotates the refresh
token and cookies cannot be written, persist a recovery bridge so a later
/auth/refresh can recover after session expiry.

* 🫙 fix: preserve stale OIDC cookie bridge key

Track the refresh token last written to the browser cookie separately
from the current session refresh token. When inline OIDC refreshes rotate
tokens without a writable response, keep bridging from the browser-stale
token directly to the latest session token.

* 🙌 fix: keep OIDC bridge recovery success on cleanup failure

Make refresh-token bridge cleanup best-effort after a bridged OIDC
refresh succeeds. A transient delete failure now logs a warning but does
not convert the already-refreshed session and cookies into a 403 response.

* 📦 test: Exclude RefreshTokenBridge from tenant-isolation coverage

Add RefreshTokenBridge to the tenant-isolation coverage allowlist because
refresh bridge lookups run during unauthenticated OpenID refresh recovery.
The controller first recovers user context from the signed OpenID marker
cookie, then the bridge methods apply explicit user and tenant filters.

Ambient tenant isolation would bind this recovery path to request-local
tenant context that is not available at the point the stale cookie is being
resolved

* ⚡ Fix OpenID refresh flight retry and marker hydration

Allow failed OpenID refresh flights to be reclaimed immediately instead of pinning transient errors.

Preserve the browser refresh-token marker when joined refreshes hydrate session tokens from a shared flight result.

Stabilize AuthService tests by isolating mocked module imports from prior suites.

* 🛠️ fix: centralize OBO identity scoping

Add shared auth identity helpers for app user ids, OpenID subjects,
tenant ids, and normalized OpenID issuers.

Thread a non-placeholder-visible OBO identity context from the real
request user through MCP connection, tool-call, reinit, and refresh
paths. Keep tenantId and openidIssuer out of createSafeUser so MCP
user placeholders do not expose those fields.

Scope OBO token cache and in-flight exchange keys by tenant, issuer,
OpenID subject, scopes, and a SHA-256 hash of the upstream assertion.
This prevents cross-tenant/cross-issuer collisions and avoids reusing
tokens minted from stale rotated assertions.

Use the shared identity helpers for OpenID refresh-flight keys and
refresh-token bridge recovery records so related OBO refresh paths share
the same identity normalization rules.

The helper is intended for auth-boundary and credential-cache code, not
as a blanket replacement for ordinary app user id ownership checks.

* 🛠️ fix: preserve OIDC refresh-token sync on save failures

Sync OpenID refresh-token cookie/bridge state before persisting the
session so a transient session-store failure cannot lose an IdP-rotated
refresh token.

Also trigger sync when the session refresh token differs from the
browser refresh-token marker, not only when the current grant rotates
the token. This lets later writable refreshes repair stale browser
cookies left behind by SSE refreshes.

Route refresh bridge identity through the shared identity helper with
the threaded OBO identity context, falling back to request/user context
when needed.

Add regression coverage for session-save failures, stale browser cookie
repair, non-writable bridge storage, and shared-helper identity fallback.

* 🛠️ fix: keep OIDC refresh bridge during recovery grace

After successful bridged refresh recovery, re-store the stale-cookie bridge
with a short grace TTL instead of deleting it immediately. This lets parallel
/api/auth/refresh requests that already sent the stale browser cookie recover
before they can observe the first response's Set-Cookie.

Retarget the bridge to the refresh token returned by the bridged retry so
B-to-C refresh-token rotation remains recoverable. The grace TTL is parsed with
math() and defaults to 60s, which shrinks the replay window from the original
REFRESH_TOKEN_EXPIRY bridge lifetime to the short recovery grace period.

Remove the now-unused explicit bridge delete path from the service and
data-schemas method surface. Add regression coverage for grace re-store,
identity symmetry, retry failure behavior, and same-key upsert replacement.

* 🛠️ fix: fail closed on OBO MCP user identity mismatch

Add an OBO-specific guard before MCP tool execution that requires the
effective invocation user and captured request user to both have ids and
to match. This prevents OBO tool calls from falling back to a separate
configurable.user_id identity after request-bound OBO context has already
been captured.

Keep the existing user id fallback behavior for non-OBO MCP calls.

Tests cover mismatched OBO users, missing user ids, and the matching-user
path ignoring a conflicting configurable.user_id.

* 🛠️ fix: Guard OpenID bridge retry user identity

Extract the shared OpenID refresh/user-resolution flow in AuthController
so the normal refresh path and bridge-recovery retry use the same grant,
claims, issuer, user lookup, and diagnostic logging code.

Preserve the existing path-specific behavior: the normal path still owns
migration updates and 401 login redirects, while the bridge retry still
falls through to the existing 403 invalid-token response.

Add a bridge-recovery guard that rejects retry results whose resolved
user id differs from the signed openid_user_id cookie before issuing
tokens or re-storing the grace bridge. Cover both the successful
matching-user recovery and the mismatched-user rejection.

* 🛠️ fix: type-safety polish on OBO data layer

Replace refresh token bridge query/update Record<string, unknown> usage
with typed Mongoose FilterQuery and UpdateQuery definitions.

Harden OpenID marker cookie JWT expiry handling by converting refresh
expiry milliseconds to integer seconds and rejecting invalid or
non-positive durations.

Add focused CSRF tests for fractional refresh expiry values and invalid
expiry configuration.

* 🛠️ fix: Bind OpenID session tokens to authenticated identity

Stamp OpenID session token state with the LibreChat user id, OpenID subject,
tenant id, and normalized issuer when tokens are stored.

Fail closed before OBO inline token reuse/refresh when the session token
identity does not match the current authenticated identity, preventing a stale
or mixed Express session from supplying another user's upstream assertion.

Also validate the normal /api/auth/refresh session-token reuse shortcut against
the signed marker-cookie user before returning cached session tokens.

Note: sessions created before this change carry no identity stamp and are
treated as a mismatch. This is self-healing — the reuse path forces a full IdP
refresh (which re-stamps the session) and the OBO path throws, surfacing as a
one-time re-authentication for active OBO users at deploy time. The session
re-stamps within one session lifetime (SESSION_EXPIRY, default 15 min).

* 🛠️ fix: Recover OpenID refresh token drift

Prefer the browser refresh-token cookie when it differs from the
server-side OpenID session state, and force a real IdP refresh in that
case instead of reusing stale session tokens.

Store a short-lived refresh-token bridge when inline OBO refresh writes
a rotated browser cookie but session persistence fails, so follow-up
refreshes can still recover from the old token.

Keep the bridge grace TTL centralized in RefreshTokenBridge so both
recovery paths use the same env-backed value.

Note: drift is measured against the last-synced browserRefreshToken
marker, so the SSE path (intentionally stale cookie, authoritative
session) does not false-positive. Sessions predating the marker have no
browserRefreshToken; for those, drift falls back to comparing the cookie
against the session refresh token and prefers the cookie on difference.
This is the same self-healing pre-change-session window as the identity
binding fix and re-syncs within one session lifetime.

Tests cover cookie/session drift selection, reusable-session bypass on
drift, bridge storage after session-save failure, and the shared bridge
constant wiring.

* 🛠️ fix: Harden OBO token caching and expiry handling

Reject malformed OBO grant responses before writing them to the exchanged-token cache so a missing access_token cannot poison the cache.

Store absolute expires_at values with cached OBO tokens and ignore legacy cache entries without usable expiry metadata. This keeps cached-token freshness based on the token’s real
remaining lifetime instead of reusing the original relative expires_in on cache hits.

Move OBO expiry normalization and skew helpers into packages/api and use them from both the JS exchange service and the TS MCP resolver. Apply a 30-second safety margin with a one-
second floor for short-lived tokens, covered by direct helper tests and caller-level regression tests.

Tests:
- packages/api: npm run build
- packages/api: npx jest src/mcp/oauth/expiry.spec.ts src/mcp/oauth/obo.spec.ts
- api: npx jest server/services/OboTokenService.spec.js

* 🛠️ fix: Harden OBO refresh-token bridge lookup and indexing

Reuse getValidOpenIDReuseUserId for the bridge-recovery user lookup in
refreshController instead of re-verifying openid_user_id inline. The shared
helper enforces the JWT_REFRESH_SECRET presence check and a strict
typeof payload.id === 'string' guard, rejecting tokens whose id claim is
present but not a string (e.g. a numeric id) that the inline check accepted.

Fail closed on issuer mismatch in getRefreshTokenBridge. Both the stored and
the expected issuer are now normalized and compared for equality, so a bridge
is recovered only when both sides agree (both absent, or both present and
equal after normalization). Previously the check was skipped whenever the
stored issuer was absent, allowing recovery across mismatched issuer context.

Drop the unused {oldRefreshTokenHash, userId, tenantId, openidIssuer} index
and the openidIssuer field on RefreshTokenBridgeQuery. The data-layer filter
only queries the 3-field {oldRefreshTokenHash, userId, tenantId} index; the
issuer is verified in application code, not the query. Hoist the repeated
model accessor into getRefreshTokenBridgeModel.

Note: issuer is now load-bearing for recovery. A bridge stored with an issuer
recovers only when the lookup supplies a matching issuer; the recovery lookup
reads user.openidIssuer via AUTH_REFRESH_USER_PROJECTION (an exclusion
projection that retains the field). If a user's persisted openidIssuer is
empty while the stored bridge has one, recovery fails closed (falls through to
normal re-authentication) until the bridge TTLs out — no security regression.

Tests cover invalid signed-cookie payloads bypassing the bridge, both
asymmetric issuer-presence cases, issuer normalization before comparison, and
an index-alignment assertion guarding against re-adding the dropped index.

* 🛠️ fix: Degrade OBO discovery on token resolution failures

Catch expected OboTokenResolutionError failures during MCP tool discovery and
fall back to unauthenticated tool listing instead of aborting discovery. This
keeps discovery aligned with the existing unauthenticated listing behavior while
preserving unexpected errors as real failures.

Also correct OBO tool-call freshness comment and tighten the OBO trust-check
permissions type to the existing role permission shape.

Tests:
- npx jest src/mcp/__tests__/MCPConnectionFactory.test.ts --runInBand --coverage=false
- npx jest src/mcp/oauth/obo.spec.ts --runInBand --coverage=false

* 🛠️ fix: tighten OBO tool-call errors, bridge logging, and flight typing

Move resolveToolCallUserId inside the tool-call try/catch so an OBO
identity mismatch surfaces with serverName/toolName context and the
standard tool-call-failed message instead of an opaque bare Error.

Raise the refresh-token bridge lookup failure log from debug to warn so
transient infrastructure failures on the unauthenticated /api/auth/refresh
path are observable, and guard the message access against non-Error values.

Replace the unknown+cast in isDuplicateKeyError with a hasErrorCode type
predicate so the duplicate-key check reads error.code without an assertion.

Preserve real math/isEnabled in the MCPConnectionFactory test mock (mock
only processMCPEnv) so mcpConfig timeouts no longer resolve to NaN, fixing
the TimeoutNaNWarning that masked slow OAuth retry behavior.

* 🧪 fix: Restore the Flight Uniqueness Index and Buffer the Graph Cache TTL

Two CI failures on the merge, both in suites this environment cannot run
(their MongoDB binary download is blocked).

`GraphApiService.spec.js` still asserted the unbuffered TTL. Graph tokens
route through the same `getTokenCacheTtlMs` as the OBO and openidStrategy
caches, so the entry now expires 30s before the credential does.

`openidRefreshFlight.spec.ts` dropped the database between tests, which takes
the indexes with it, and Mongoose builds them only once when the model is
compiled. Whether the unique `key` index survived into a test was a race with
that one-time build. Without it a second `create` inserts instead of raising a
duplicate-key error, so every worker believes it won the flight — the
mutual exclusion the file exists to prove. Indexes are now rebuilt after each
drop, which also makes the reclaim and complete cases reach those paths for
the right reason.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SxKWxwqxAGckYpRsYTqx3F

* fix: address OBO review findings

* 🔐 fix: Install Bridge Indexes and Carry OBO Through Assistant Recovery

Two findings from the Codex pass on d08c82f0d.

The refresh-token bridge relied on Mongoose auto-indexing for both of its
indexes, and `MONGO_AUTO_INDEX=false` is a supported deployment setting. A
bridge holds an encrypted refresh token and the TTL index is the only thing
that ever deletes one, so under that setting they would accumulate for the
life of the collection while concurrent upserts lost the compound uniqueness
the filter assumes. Installed before the first write, matching the flight
methods and the session and schedule methods before them.

`recoverServerTools`, the assistant create/update path that reruns
`reinitMCPServer` when a referenced server's catalog and connection snapshot
are both missing, was the last reinit site not carrying the upstream-token
closure. For an OBO server the factory rejects the connection outright, so the
assistant write failed with unavailable MCP definitions. It now builds the
provider at that request boundary like the other entry points; assistant
writes have no `res`, so a rotation there falls back to the recovery bridge.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SxKWxwqxAGckYpRsYTqx3F

* fix: harden OBO refresh coordination

* ⏳ fix: Keep Elapsed Expiries Elapsed and Revoke the Superseded Session

Two of the five findings from the Codex pass on fcdc15885 — the two that are
defects in code this branch introduced rather than design questions about the
bridge.

`getSkewedTokenExpiresAtMs` floored every result at a second in the future,
including an expiry the provider had already declared elapsed. An exchange
answering `expires_in: 0` or a past `expires_at` was handed to the MCP
connection stamped valid for another second, which only moves the failure
downstream. The floor now applies to a lifetime that is still live, which is
what it was for; an elapsed one stays elapsed so the caller rejects it. Same
for the cache TTL, which falls back to the elapsed-credential floor.

Bridge recovery left the stale token's durable Session behind. Only the token
it recovered through was passed as `existingRefreshToken`, so that one's
session was replaced while the token the browser actually presented kept its
record until its original expiry. That record, with the marker cookie still
bound to it, is what authorizes local image access for OpenID users — so a
copy of the stale cookie outlived the rotation it had lost. Revoked
explicitly on successful recovery.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SxKWxwqxAGckYpRsYTqx3F

* fix: close OBO refresh review findings

* 🎟️ fix: Carry the Bridged Token Through a Non-Rotating Recovery

Bridge recovery passes the browser's stale token as `existingRefreshToken` so
the durable Session naming it is the record replaced. That also makes the
stale token the fallback the installed session and the refresh cookie use when
a tokenset carries no `refresh_token` of its own, which holds only while the
recovery grant rotates.

An IdP that answers that grant without rotating sends the browser back to the
very token the bridge exists to retire: `storeOpenIDSession` installs and
deletes the same stale record in one call, and the cookie is rewritten to a
token the IdP already rejected — a sign-out on the next refresh. The grace
bridge one line above already guards this with `|| bridgedRefreshToken`; the
resolved tokenset now does the same, so leader and followers alike publish the
recovered token.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SxKWxwqxAGckYpRsYTqx3F

* 🚫 fix: Reject an OBO Exchange That Returns an Expired Credential

Preserving an elapsed expiry through the skew helper only helps if something
acts on it, and nothing did: `MCPManager.callTool` checks the access token and
nothing else before setting the Authorization header, so a credential the IdP
declared spent still went downstream to fail there. It is rejected at the
exchange now, where the reason is known, and retryably — the exchange itself
worked, so a fresh grant can succeed.

Completes the elapsed-expiry change in cb26a6f7d, which made the stamp honest
without giving anyone a reason to look at it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SxKWxwqxAGckYpRsYTqx3F

* fix: close OpenID refresh review findings

* fix: coordinate OpenID refresh entry points

* chore: sort OpenID flight imports

* fix: fence OpenID refreshes during logout

* fix: close OpenID logout publication races

* fix: narrow completed refresh flight

* test: cover bridge cleanup failure after ownership loss

The compensating delete in storeRefreshTokenBridgeWithLease swallows its own
failure so the lease error stays the one the caller sees. Nothing asserted
that, so removing the inner catch left every suite green while callers began
receiving the cleanup error instead.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SxKWxwqxAGckYpRsYTqx3F

* fix: compensate OpenID bridges only on proven ownership loss

The post-write lease assertion deletes the bridge it just published when it
throws, but it threw for two different reasons: a coordination record that is
no longer ours, and a coordination read that simply failed. Treating the second
as the first destroys the only mapping from the token the browser still holds
to the one the IdP already rotated to, so a transient Mongo error on the
headers-already-sent path signed the user out.

Tag the ownership error where the lease raises it and compensate only for that,
preserving the bridge whenever ownership is merely undetermined. A preserved
bridge stays behind the logout revocation fence, so the safe default costs
nothing.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SxKWxwqxAGckYpRsYTqx3F

* fix: reject spent OpenID refresh results

Two ways a refresh could report success while handing back a credential
nothing can use.

An inline refresh carries the previous id_token forward when the IdP omits
one on rotation, so tokenset.id_token is not necessarily freshly issued.
setOpenIDAuthTokens applied its freshness guard only to the session copy and
took tokenset.id_token unconditionally, so /refresh returned an expired
bearer even though the grant produced a usable access token. Skip it only
when it is provably expired: an id_token whose expiry cannot be read stays
preferred, since access_token may be opaque or scoped to another audience.

normalizeExpiresIn preserves a zero or negative lifetime rather than
discarding it, so a grant declaring an already-spent access token still
published, rotating the refresh token and returning a token every freshness
check rejects. Each OBO call then repeated the grant. Reject an elapsed
lifetime before publishing; an unknown lifetime still publishes.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SxKWxwqxAGckYpRsYTqx3F

* fix: harden OpenID refresh publication

* fix: keep identity and results intact through OpenID refresh cleanup

Two follow-ons from the last round's fixes.

Stripping an expired carried-forward id_token from the refresh result removed
the only identity material a rotation without id_token leaves behind. The
result is rebuilt by buildOIDCTokensFromSession, so it carries no provider
claims() either, and getTokenClaims accepts only those two — bridge recovery
failed with "no usable identity claims" before setOpenIDAuthTokens could hand
back the fresh access token. The stripped token now travels in a
non-enumerable marker, alongside the existing browser and predecessor markers,
which identity resolution reads and the authentication response never sees.

The lease drained a pending renewal by awaiting it in finally, so a transient
coordination failure there threw from finally and replaced the operation's
result. The refresh had already settled and published, so the caller saw a
failure on credentials that had rotated. Proven ownership loss is recorded on
ownershipLost and checked before the return, so the drain has nothing to add
but noise; it now absorbs and logs.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SxKWxwqxAGckYpRsYTqx3F

* refactor: fence OpenID recovery publication

* fix: close OpenID publication transaction

* refactor: make OpenID publication transactional

* fix: satisfy OpenID publication type checks

* fix: fence OpenID session publication

* fix: authorize OpenID refresh publication

* fix: bind OpenID replay generations

* fix: fence OpenID response generations

* fix: authorize OpenID token delivery

* fix: linearize OpenID publication delivery

* chore: sort OpenID refresh flight imports

---------

Co-authored-by: J.C. Bartle <jcbartle@users.noreply.github.com>
Co-authored-by: jbartle <jbartle@rand.org>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: jcbartle <7274202+jcbartle@users.noreply.github.com>
2026-08-29 23:55:58 -04:00
Danny Avila
8ae94afa91
🪡 fix: Thread Parent Message ID Through MCP Request-Scoped Bodies (#15095)
* fix: Unify MCP request-scoped headers

* fix: address request-scoped MCP review findings

* test: preserve request scope on status errors

* fix: treat authorized on-demand MCP servers as ready

* refactor: separate MCP readiness from connection state

* fix: preserve on-demand MCP readiness labels

* test: satisfy OpenAI conversation ownership guard

* fix: keep MCP action predicates boolean

* fix: close deferred MCP request context gaps

* fix: preserve on-demand MCP configuration actions

* fix: fail closed on unavailable MCP parent context

* test: complete MCP connecting-state mocks

* fix: preserve missing MCP parent on continuations

* fix: align native MCP request identities

* fix: preserve edited MCP parent identity

* test: use scoped Agent initializer fixture

* test: expose MCP request body helper

* fix: preserve MCP turn identity across resume

* style: sort stream metadata imports

* fix: carry normalized MCP identity to execution
2026-08-21 16:33:01 -04:00
Danny Avila
c5276fc63d
⏱️ feat: Run Scheduled Chats Through Durable Agent Triggers (#14939)
* feat: Scheduled Chats — agent-centric scheduled runs creating real conversations

Squash of the full review-hardened branch (PR 14540, supersedes 14373) onto
latest dev, preserving the exact verified tree. History prior to this commit
lived on the pre-squash branch; every invariant below survived 25 Codex review
rounds plus two external audit rounds (R26) with regression tests that fail
without their fixes.

Feature:
- Schedules CRUD + side-panel UI (cadence dialog, run cards, Run Now), roles/
  permissions (SCHEDULES:USE), interface.schedules availability, per-user limits
  and capacity slots, timezone-aware cadence with DST-conservative floors and
  misfire grace.
- Engine: single-process claim/fire loop with leases, loopback POST dispatch
  (signed schedule-fire JWT claims, per-occurrence idempotency key inside the
  route's clientRequestId charset), overlap/balance/capacity/duplicate skip
  policies with auto-disable streaks (too_many_failures, insufficient_balance),
  reconciliation from retained terminal-job evidence, erasure sweep.
- Scheduled runs create real conversations through the resumable agents chat
  path: HITL pauses surface on the card (requires_action), resumes re-apply the
  fire boundary's admission policy (revision fence, enabled, global kill switch,
  SCHEDULES:USE, availability) before continuing a billed generation.

Correctness invariants (the audit surface):
- Settlement discipline: every persistence-producing write happens-before a
  run's terminal outcome write; Stop/complete/pause race through single-winner
  terminal CAS claims (dev's TerminalJobClaim substrate) with retained,
  completedAt-less evidence for scheduled fires plus an owner-intended outcome
  stamp (scheduleOutcome) the reconciler prefers over re-derived success —
  round-tripped through the Redis hash mapper.
- Swallowed generation failures (client error content parts) classify to
  error/skipped_balance instead of success on both initial and resumed paths;
  stale stamps are refreshed evidence-first when persistence plus the Mongo
  outcome write both fail.
- Abort honesty: delivery judged by generation ownership and the CAS's actual
  from-status; republication escalates to the transport's acknowledged variant;
  the Stop route settles only genuinely paused runs.
- Account deletion: one-way barrier (deletionRequestedAt) with auth-cache
  tombstone-before-stamp, boundary rechecks across all auth strategies, quiesce
  of scheduled + interactive work with durable per-stream abort fences
  (positive-evidence acknowledgement only), owner-side finalization markers for
  post-terminal billed writes, deferred-deletion sweep (explicitly ensured
  partial index) that completes cascades autonomously. Remote OpenAI-compatible/
  Responses requests are documented as outside the quiesce and tracked in issue
  14594.
- Store compatibility: finalization markers optional on the legacy IJobStore
  contract with coherent degradation (registration fails -> synchronous title
  fallback; count reads 0).

* fix: fence terminal response persistence from deletion; retain stale-pause evidence

Two blockers from the third external review round.

Terminal persistence visible to account deletion:

- The finalization marker was registered only when post-terminal TITLE work was
  possible, but every persistence-owning terminal CAS opens the same window: the
  claim drops the job out of the active set BEFORE the response save (and the
  background user-message/convo saves), so a deletion quiesce landing there saw
  neither an active job nor a marker and could cascade while the admitted request
  could still recreate messages. Both controllers now register the marker before
  every persistence-owning claim — the fresh-turn path and the HITL resume — and
  release it only after their pending saves (and any post-terminal title) have
  landed, on success and failure paths alike. The TTL bounds a crash.
- settleAbortFence no longer clears a complete/error fence while
  `terminalPersistencePending` is set: terminal at the CAS is not settled while
  the owner is still persisting. The job facade now surfaces the flag.
- The marker trio is REQUIRED by the runtime store contract (assertJobStoreV2
  refuses a store without it at configure time, keeping the failure loud and
  deterministic) while remaining optional on the legacy public IJobStore type for
  source compatibility. The silent degrade path from the previous round is gone —
  it was not deletion-safe.

Stale-pause recovery retains scheduled evidence:

- Three crash/timeout recovery paths — ApprovalLifecycle.failStalePausePersistence
  and the InMemory/Redis stale-pause cleanups — unconditionally stamped
  `completedAt`, putting a scheduled fire's failed-pause error terminal on the
  short completed TTL. A Mongo outage longer than that TTL erased the evidence and
  the reconciler recovered the run as `interrupted` instead of `error`. All three
  now follow the controller-observed path from the previous round: scheduled jobs
  omit `completedAt` (retained-evidence TTL) and stamp the error outcome.

The PR description now explicitly narrows the deletion guarantee for the remote
OpenAI-compatible/Responses paths (tracked in issue 14594).

* fix: deletion-fence marker protocol — generation-scoped, atomic, fail-closed, all terminal paths

One consolidated pass over the finalization-marker mechanism, per the fourth
external review round. The invariant it establishes: NO persistence-owning
terminal CAS runs without a durable, generation-qualified marker covering the
window it opens, and every consumer treats a pending terminal as unsettled.

- Generation-scoped markers. Entries were keyed (userId, streamId), so a
  Stop-superseded generation finishing late could clear the marker its
  replacement registered on the same conversation. Marker fields are now
  qualified by the generation's createdAt; clears must present the same
  identity, and an unqualified legacy clear cannot drop a qualified entry.
- Atomic Redis registration. HSET-then-EXPIRE loses the fresh marker when the
  user's existing hash expires between the two commands (or the process dies
  there); registration is now a single Lua script carrying both.
- Fail closed everywhere. Registration failure (after one retry) now REFUSES
  the terminal CAS instead of proceeding uncovered: the completion claim throws
  into the error path, the error path skips completeJob and leaves the job
  ACTIVE — deletion-visible by itself, recovered by the stale-running reaper —
  and abortJob returns a new retryable `fence_unavailable` failure the Stop
  route answers with 503 and the deletion quiesce treats as an unacknowledged
  stop (fence kept). The previous round's log-and-proceed is gone.
- Every terminal path enrolled. abortJob now owns its window (register before
  the abort CAS, clear in its finally — the Stop route's checkpoint prune and
  partial save run inside beforePublish, between CAS and publication); the
  interactive and background generation-error paths register before their
  completeJob; the resume controller's error finalization registers before its
  completeJob. A lost or thrown claim releases the marker after pending saves
  flush instead of holding the user's deletion behind the TTL.
- Every pending terminal unsettled. settleAbortFence defers on
  terminalPersistencePending for ALL statuses — including `aborted`, whose
  route-side persistence the previous guard missed.

Also: a direct Redis regression for scheduled stale-pause retention (the P2
test gap), and the PR description no longer claims to carry every commit.

* fix: lease-token lifecycle fences — same-generation isolation, admission fence, undelivered-Stop retention, legacy abort enrollment

Fifth external review round; four P1s, handled as the requested consolidated
lifecycle-fence pass.

- Lease tokens. Marker fields were (streamId, createdAt), shared by every
  contender on the same generation — completion and Stop, or two racing Stops —
  so a losing contender's cleanup erased the winner's still-live marker. Every
  registrant now carries a unique lease token in the field and may only ever
  clear its own lease; unqualified legacy clears cannot touch qualified entries.

- Admission fence. Authentication can pass before the deletion barrier goes up,
  and the durable createJob is several async steps later — a deletion quiesce in
  that window saw neither an active job nor a marker and could cascade before
  the admitted request created its job. The controller now registers an
  admission lease and THEN rereads the deletion barrier: the ordering guarantees
  either this request observes the barrier (403, lease released, slot/claim
  cleanup) or the quiesce observes the lease and defers. Held until createJob is
  durable; released on every refusal and initialization-error path. Fail closed
  when the lease itself cannot be registered (503 retryable).

- Undelivered-Stop retention. abortJob released its lease in a finally even when
  delivery AND publication had provably failed — the job reads terminal
  (invisible to active-set scans), a user Stop writes no durable abort fence,
  and the remote owner keeps generating and will persist its abort-catch writes
  whenever the signal finally lands. The lease is now retained in exactly that
  case, and each resignal attempt heartbeats a fresh lease so the fence outlives
  the TTL for as long as delivery is still being driven. The abort-winning
  turn's own loser-side pending saves are additionally fenced in the controller
  catch (best-effort — those writes are already in flight).

- Legacy abort enrollment. abortMiddleware (assistants abort route fallback for
  non-assistants endpoints) awaited abortJob and then spent usage and saved the
  stopped response AFTER the abort's lease was released. Both writes now run
  inside `beforePublish`, between the abort CAS and publication, covered by the
  same lease as every other abort.

Barrier tests, each verified to fail without its fix: same-generation lease
isolation (store), racing two-Stop loser cleanup (manager, stale-read forced
CAS race), undelivered-Stop lease retention, resignal heartbeat, admission
refusal with lease-before-reread ordering plus release-on-durable-create, and
legacy-abort persistence inside beforePublish.

* fix: heartbeat-backed owner-lifecycle leases close the settlement handoff races

Sixth external review round: the remaining P1 interleavings were one structural
problem — lease handoffs that were not atomic — resolved as the requested
consolidated lifecycle-lease pass.

- Quiesce reads leases BEFORE the active-job scan. The admission-lease -> durable
  -job handoff is only atomic against a reader in the OPPOSITE order of the
  writer: writers hold the lease strictly until the job is active-set visible,
  so leases-first shows every interleaving either the lease or the job.
  Jobs-first allowed a request to create its job after the scan and release its
  lease before the count — hiding both, cascading, and letting the new
  generation persist into a deleted account.

- The abort acknowledgement is fenced by the owner-lifecycle lease. Redis ACKed
  the moment the owner's AbortController tripped; the stopping side released its
  lease on that ACK while the owner's asynchronous abort-catch persistence was
  still ahead. The transport now awaits a manager-installed pre-ACK hook that
  registers a DETERMINISTIC owner lease (exactly one owner exists per
  generation, and determinism is what lets the signal-time registrant and the
  owner's catch-side release agree across processes) before the acknowledgement
  is persisted or published; an owned same-replica abort bridges to the same
  lease before tripping its local controller. Both generation-owner catches
  (fresh turn, resume) release it once their writes land.

- A failed replacement handoff no longer orphans the predecessor. The atomic
  replacement removes it from active storage, and an unconfirmed handoff
  terminalizes the replacement too — leaving nothing a quiesce could discover
  while the predecessor's provider may still be generating. Its owner lease is
  now retained at the point the receipt fails delivery; the owner replica renews
  it through the pre-ACK fence when the signal finally lands.

- Leases HEARTBEAT while held. The five-minute store TTL only bounds a crashed
  holder; live persistence — a stalled save, a long deferred title — must never
  outlive its own fence. holdUserFinalization registers and renews every minute
  until released; the controllers' completion/error/admission leases all hold.
  The undelivered-Stop retention moved to a deterministic `stop` lease that
  every resignal attempt renews and the first successful one clears (no more
  opaque leases accumulating to TTL), and a THROWN abort transition releases the
  contender lease instead of leaking it.

- The user-document abort-fence mutations now invalidate the auth user-doc
  cache, matching every other user-doc write.

Barrier tests, each verified to fail without its fix: quiesce lease-scan
ordering (plus the observed-lease defer), pre-ACK fence ordering at the
transport, owned-abort owner-lease bridging, replacement-handoff predecessor
retention, held-lease heartbeat past the TTL, and failed-then-successful
resignal reaping the retained stop lease.

* fix: one manager-owned owner-lease span across every abort delivery path

Seventh external review round; four lifecycle-fence gaps, closed by making the
owner-lifecycle lease a single manager-owned, heartbeat-held span.

- Fail-closed acknowledgements. The pre-ACK hook registered a one-shot lease and
  the transport ACKed even when it failed; a same-replica owned abort likewise
  swallowed registration failure. The hook now acquires a HELD owner lease
  (heartbeat until the owner's catch releases it via releaseOwnerLease) and a
  rejection SUPPRESSES the acknowledgement — the stopping side stays retryable
  behind its retention lease, and every resignal re-drives the handler. The hook
  also stopped gating on `job.createdAt === generationId`: during a replacement
  handoff the store holds the replacement while the abort targets the
  predecessor, and that gate silently skipped exactly the generation being
  acknowledged (the store job is owner identity, never a generation gate). A
  local owned abort acquires the same held lease before tripping its provider;
  post-CAS the trip cannot be withheld, so acquisition failure downgrades
  delivery and the retention handoff keeps the user fenced.

- Committed-but-lost-reply disambiguation. A thrown abort transition released
  the contender lease as if nothing had happened, but a Lua CAS can commit and
  lose its reply — an aborted job invisible to active-set scans whose provider
  was never signalled, with no fence left. The throw path now re-reads the exact
  generation: only a job still live under the caller's identity proves no
  commit; committed or ambiguous outcomes hand the fence to the deterministic
  stop lease (kept on the contender lease if even that fails) before rethrowing.

- Replacement handoff covered end to end. A LOCAL replacement abort acquires the
  predecessor's held owner lease before the trip (failure reports the receipt
  undelivered, engaging retention). Failed-handoff retention is no longer a
  swallowed one-shot: it heartbeats with the durable acknowledgement proof as
  its renewal predicate — acquisition failures keep retrying for as long as the
  fence is needed, and the retainer stands down (without clearing the shared
  field) once the remote owner ACKs and thereby holds its own lease.

- A LOCAL resignal delivery hands off to the owner lease BEFORE clearing the
  retained stop lease, and keeps the stop lease when that handoff fails.

Barrier tests: hook rejection suppressing the ACK, commit-then-lost-reply
retention with its provably-uncommitted counterpart, local-resignal owner
handoff ordering, pre-ACK owner lease held past the store TTL until release
(and provably stopped after), and failed-handoff retention retrying on its
heartbeat — verified fail-before/pass-after by stashing the fixes.

* fix: finish scheduled chat lifecycle hardening

* fix: close scheduled chat review follow-ups

* fix: generation-fence abort recovery evidence

* test: wait for settled approval tool output

* test: preserve scheduled init reconciliation option

* refactor: rebuild scheduled chats on durable agent triggers

* test: reset MCP cache mock between cases

* test: isolate scheduler startup in server specs

* fix: harden scheduled run lifecycle

* fix: normalize schedule capacity conflicts

* test: type schedule collision fixture

* fix: re-fence scheduled resume and expiry

* fix: fence scheduled resume handoffs

* fix: release superseded manual schedule leases

* fix: release failed run-now claims

* fix: release superseded engine claims

* fix: repair schedule dialog interaction and rework its form

The agent picker was unusable: ControlCombobox portals its popover to the
body by default, which lands it outside the dialog's Radix focus trap. Clicks
passed through it, it could not be tabbed into, and the trap fighting Ariakit
for focus locked the page up on selection. The prop is documented for exactly
this case — pass `portal={false}` and give the dialog `overflow-visible`, as
ProjectButton already does. The time and day dropdowns defaulted the same way.

Alongside that:

- Extract the agent builder's instructions editor (special-variable menu plus
  expand-to-fullscreen) into a controlled `VariableEditor` and use it for the
  schedule prompt. Insertions now route through `onChange`, so react-hook-form
  sees them — the schedule PATCH is built from `dirtyFields`, and a `setValue`
  that skipped dirty tracking would have dropped an inserted variable silently.
- Replace the hand-rolled frequency buttons with the shared `Radio`. They marked
  the selection with `bg-surface-hover` on an outline button whose hover is the
  same token, so the selected option was indistinguishable from a hovered one;
  `Radio` is also a real radiogroup rather than four `aria-pressed` toggles.
- Wrap the fields in a real `<form>` and associate the footer button by id, so
  Enter submits. Group the frequency, day and time controls in fieldsets.
- Add placeholders for name and prompt, match the textarea fill to the other
  fields, and label the hourly case as minutes past the hour.
- Widen the dialog to `md:max-w-3xl` and pair name with agent so the form fits
  without scrolling on desktop.
- Move scheduled chats below skills and above prompts in the side nav.

The new dialog spec fails when the portal fix is reverted.

* test: cover scheduled and subagent deletion drains

* refactor: own the form-control appearance in the client primitives

Addresses the codex finding on ScheduleDialog: a feature-local `FIELD_CLASS`
restated the `Input` primitive's border, radius, height and background so it
could be pasted onto the schedule dropdowns, leaving those controls with no
connection to the primitive they were imitating.

Move that appearance into `packages/client/src/components/Field.ts` as the
single source `Input` and `Textarea` now compose, and give `Dropdown` and
`ControlCombobox` a `variant="field"` that applies it. The schedule dialog
passes the variant and carries no class strings of its own.

This also repairs a break the dev merge would otherwise have introduced: the
newer `Dropdown` splits `className` (wrapper) from `triggerClassName`, so the
old pasted classes would have landed on the wrapper and left the triggers
unstyled.

The semantic-token guard now watches the shared module and asserts each
primitive still composes it, which covers more than the two files it read
before.

* fix: keep an explicit schedules disable from becoming an opt-in

`use` is two things at once for a dual-purpose runtime interface field: a
permission bit, which DB overrides strip, and the runtime disable signal that
`getLimits` reads. Stripping it from `{ use: false, maxPerUser: 2 }` leaves an
object, and `getLimits` treats any object without `use: false` as enabled — so
an admin override written to stop scheduled billing for a role or user started
it instead.

Collapse an explicit disable to the boolean form before the strip, on both
paths that accept it: the `interface.schedules` field patch, which admitted the
object wholesale because bare runtime paths deliberately bypass the permission
gate, and the overrides merge, which reached the composite-field branch and
kept `maxPerUser`. Objects that only narrow limits are untouched, so a
principal can still be given a smaller cap.

Both regressions fail without the normalizer.

* fix(schedules): Wave A — null-balance CAS, atomic paused-card clear, clustered erasure sweep

Slice 1 (thread r3804518381): route the existing-null balance initialization
through a { user, tokenCredits: null } compare-and-set instead of a blind $set,
so a concurrent initializer/charge landing between the preflight read and the
write is never handed back its spent starting balance. On a CAS miss the
preflight re-reads the winner. The absent-record $setOnInsert path and the
credited-record refill-config sync are unchanged. Adds the initializeNullBalance
adapter (no upsert) and regression coverage for winner/miss/sync cases.

Slice 2 (thread r3804518388): updateScheduleById now drops a `requires_action`
lastRun projection atomically with the configRevision bump. Any pause present at
edit time was projected under the pre-edit revision and can never be replaced by
its own revision-fenced terminal outcome — a disabling edit would strand the
card on "Needs approval" forever. Implemented as classic-operator CAS branches
(DocumentDB rules out a conditional pipeline $unset), fenced on the card STILL
being the pause so a terminal outcome or newer occurrence that races in is
preserved. Terminal history survives untouched.

Slice 4 (thread r3803826204): expose initializeScheduleErasureSweep from the
schedule runtime facade and start it in every clustered (experimental) worker
after Mongo is up. It re-drives eraseScheduleIfDrained for soft-deleted rows so
a hidden prompt cannot outlive its drain when the delete/erase-on-settle
attempts miss. It arms nothing else and never infers owner death from a
process-local missing job (isTopologySafeToArm gates that).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq

* fix(schedules): Wave B.1 — reversible account-deletion schedule suspension

Thread r3804518383. Account-deletion quiesce marked every schedule `deleting`,
disabled it, and cleared nextRunAt destructively. When a later cascade step (or
the drain itself) failed, the controller cancelled the user-deletion fence —
restoring the user — but the schedules stayed `deleting` and were erased by the
sweep, silently losing all of a live user's scheduled prompts.

Replace the destructive marking with a REVERSIBLE, token-fenced suspension:

- suspendUserSchedulesForDeletion(userId, token) snapshots each schedule's prior
  enabled/nextRunAt under a per-attempt token, then fences firing (disable, clear
  nextRunAt, rotate claimToken). It never sets `deleting`, so a suspended row is
  not erasure-eligible. Snapshotting reads then bulkWrites (a classic update
  cannot copy field values under DocumentDB), fenced per row so an already-
  suspended/soft-deleted/edited row is left alone; idempotent per token.
- restoreUserSchedulesFromDeletion(userId, token) reverses it, re-enabling and
  re-arming only rows still carrying the exact attempt token and not independently
  deleted — so an owner-deleted or newer-attempt-suspended schedule is never
  resurrected.
- deleteUserController generates the attempt token, passes it to quiesce, and on
  any failure that cancels the user-deletion fence restores the suspended rows. A
  successful deletion hard-deletes them (and their snapshots) via the existing
  cascade and never restores.

Adds a `deletionSuspension` embedded field (excluded from the wire schedule),
data-method regression tests (suspend/restore/fence/idempotency/no-resurrect),
and controller tests (restore on drain-false and post-quiesce cascade failure,
no restore on success).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq

* fix(schedules): Wave B.2 — wire the interactive Stop persistence protocol

Thread r3804255932. The durable Stop primitives (requestRunAbort 'stop',
getScheduleRunAbortState, markRunAbortPersisted) already existed but production
only used requestRunAbort(..., 'deletion'). An interactive Stop flipped the job
to `aborted` and then persisted its partial message + checkpoint inside
`beforePublish`, without ever stamping the schedule Stop or acknowledging it —
so reconciliation, the generation owner, or a concurrent schedule/account
deletion could terminalize the run and release its capacity (and erase data)
mid-write.

Wire the request -> persist -> acknowledge -> settle barrier through the
schedule runtime (the route never touches raw Mongo):

- Expose beginScheduledStop / acknowledgeScheduledStopPersistence on the service.
- The abort route stamps the Stop BEFORE signalling abortJob (a serialized
  'in_progress' loser returns 409 STOP_IN_PROGRESS without a second abort),
  acknowledges only after beforePublish persistence succeeds, and on a
  persistence failure leaves the barrier unresolved so the run stays preserved
  (client retries; stale-owner timeout is the bounded recovery). A failed abort
  releases the stamp it placed so a replacement/retry is never blocked through
  its predecessor.
- recordScheduleOutcome (the owner settlement path) now waits, bounded, for the
  Stop acknowledgement before terminalizing; a resolved/non-stop/stale marker
  proceeds immediately. The paused Stop settles only after its own ack.

Adds service-layer barrier tests and route-level ordering/persistence-failure/
in-progress tests; the data-layer serialization is already covered.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq

* fix(schedules): Wave C.1 — reconcile durable trigger delivery with the reservation

Threads r3803826192 (manual limiter) and r3804255924 (PII/moderation), plus the
independently-found long-Retry-After race. fireSchedule reserves a `started` run
and a global capacity slot BEFORE the durable trigger delivery reaches the chat
route, where an interactive limiter (manual Run Now), PII, or moderation can
reject it before any generation job exists — dead-lettering the delivery while
the run sat `started` until the 30-minute orphan sweep mislabeled it interrupted.
And a valid delivery deferred by Retry-After (up to 24h) could be orphan-settled
and have its capacity released, then fire anyway.

Translate durable delivery state into the schedule outcome:

- Store the deterministic trigger deliveryKey on the ScheduleRun reservation
  (computed from the envelope BEFORE enqueue, so an ambiguous commit still has it).
- Add a getTriggerDelivery engine dep (wired to the merged trigger service's
  getDelivery) that reads the durable delivery by key.
- Schedule reconciliation, for a jobless `started` run: staging/pending/leased →
  admission is live, never orphan; dead → record `error` from the durable
  lastError and release capacity promptly (no 30-minute wait), through the
  ordinary outcome/auto-disable path; succeeded or no record → the existing
  legacy orphan policy (interrupted only past the cutoff); a delivery lookup
  failure defers rather than orphaning a possibly-live delivery. Limiter/PII/
  moderation middleware writes no schedule state.

Adds reconcile state-mapping tests (dead/pending/leased/staging/succeeded/none/
lookup-failure) and a fire test that the reservation's deliveryKey equals the
enqueued delivery's idempotency key.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq

* fix(stream): Wave C.2 — durable retry/ack for terminal host lifecycle actions

Thread r3804518375. Approval expiry won the `requires_action → aborted` CAS and
then invoked the host hook best-effort: `runApprovalExpiredHandler` swallowed a
failure, and because later sweeps enumerate only `requires_action` jobs, the now-
aborted job was never offered again. In the clustered entrypoint (no schedule
reconciler) the ScheduleRun stayed `requires_action` and its retained job
persisted indefinitely.

Make the host lifecycle work durable rather than schedule-specific:

- Add a generic `terminalHostActionPending` marker, set ATOMICALLY in the same
  terminal transition (ApprovalLifecycle.expireWithIdentity), only when a host
  adapter is installed.
- Retain and index such jobs: both stores keep them out of terminal reaping and
  expose getTerminalHostActionJobs(); Redis adds a set + extended (24h-bounded)
  TTL, in-memory a bounded 24h retention so a permanently-failing hook cannot leak.
- The manager clears the marker only after the adapter acknowledges success,
  fenced by generation identity (clearTerminalHostAction), so a replacement
  generation can neither clear nor execute its predecessor's action.
- cleanup()/expireStaleApprovals() enumerates unacknowledged terminal host actions
  across restarts and replicas and retries the idempotent hook; the relay only
  re-invokes while the marker is unacknowledged, so a successful ack prevents
  duplicate work. Store-won expiry marks it too, so a loser-replica relay still
  crosses the hook.
- Terminal SSE notification continues regardless of host-hook outcome.

Covers in-memory behavior (retry after failure, restart/other-replica retry, ack
prevents duplicates, identity fence, terminal notification on failure, no marker
accumulation for non-scheduled jobs) and updates the Redis cluster-membership
contract test for the new index.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq

* style(schedules): satisfy import sorting in fire.ts and fire.spec.ts

CI "Static checks" failed on IMPORT_SORT for the two files Wave C.1 added imports
to (the AgentTriggerEnvelope type import and getAgentTriggerIdempotencyKey).
Applied scripts/sort-imports.mts to exactly those files — imports-only reordering,
no behavior change. Other files reported by a repo-wide check are pre-existing on
dev and deliberately left untouched so this PR is not widened.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq

* fix(schedules): repair the deletion CLI and order restore before the fence release

Addresses three findings from the fresh Codex review.

P1 — config/delete-user.js called methods.disableUserSchedulesForDeletion, which
Wave B removed in favor of suspendUserSchedulesForDeletion. The file is
@ts-nocheck and its spec mocked the removed name, so neither typecheck nor tests
caught it; the real CLI would throw a TypeError before deleting anything and then
only unwind the fence. The CLI now uses the tokenized protocol: it mints a
suspension token, suspends with it, and restores that exact attempt's rows in its
finally block when the deletion does not commit. Its spec mocks the real methods,
so the breakage can no longer hide.

P2 — both the HTTP controller and the CLI released the user-deletion fence BEFORE
restoring schedules. That fence is what refuses new schedule writes/claims, so the
gap let an owner PATCH edit a still-suspended row and have its enabled/next-run
state overwritten by the older snapshot, and let a second deletion attempt
re-suspend under a new token — making the first restore a no-op and stranding the
disabled snapshot permanently. Restore now runs first, while writes are still
fenced.

Hardening for the same defect class across a crash: suspendUserSchedulesForDeletion
now ADOPTS an existing suspension's snapshot when re-suspending a row abandoned by
an earlier attempt, instead of re-capturing the row's current (already-suspended)
state. Without this, an attempt that died before restoring would have its
successor snapshot "disabled, no next run" and permanently strand the schedule.

Tests: CLI restore-before-fence ordering and no-restore-on-success; the same
ordering assertion on both controller post-quiesce failure paths; a data-method
regression that a second attempt adopts the abandoned snapshot and restores the
original enabled/next-run state.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq

* fix(schedules): converge dead deliveries in topology-safe maintenance

Codex finding: the `dead` delivery mapping added in Wave C.1 lives only inside
startScheduleEngine's reconciler, but the clustered entrypoint arms no engine — it
runs erasure-only maintenance. A delivery queued before a restart into clustered
mode and then rejected before generation creation (interactive limiter, PII,
moderation) dead-letters while its ScheduleRun stays `started`, holding a global
capacity slot indefinitely for an ordinary non-deleting schedule.

Add a dead-delivery convergence pass to the erasure sweep, so every topology that
runs schedule maintenance settles it. The pass is POSITIVE-EVIDENCE-ONLY and is
therefore safe where absence-based reconciliation is not: a `dead` delivery is
durable shared state proving no generation owns the reservation. It settles only
when the job is confirmed absent or identity-mismatched (an identity-matched job
still owns the run), defers on an unknown job lookup, on an in-flight abort, and
on an in-flight resume hand-off, ignores legacy reservations with no deliveryKey,
and applies a short grace so an accepted delivery still creating its generation is
never settled mid-handoff. Auto-disable policy is deliberately left to the armed
engine; this path records the failure and frees the slot.

Deliberately does NOT touch api/server/experimental.js — the clustered entrypoint
already starts this sweep, so the convergence arrives through the existing
initializer and the shared-file footprint stays as-is.

Tests: settles a dead delivery as error under an explicitly UNSAFE topology,
leaves live deliveries alone, never settles under an identity-matched running
generation (delivery is not even consulted), defers an in-flight abort, and
ignores a reservation with no deliveryKey.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq

* fix(stream): refresh host-action retention on each retry attempt

Codex finding: unacknowledged terminal host-action evidence was capped at the
24h pause TTL, so a host dependency (Mongo) unreachable for longer than that let
the Redis key — and its pending marker — expire with no generation-fenced
acknowledgement, stranding the ScheduleRun where no reconciler is armed.

Measure retention from the LAST retry rather than from the terminal transition:
enumerating a pending host action IS the retry attempt, so both stores refresh
its retention as they hand it to the hook (Redis re-EXPIREs the job key; the
in-memory store stamps terminalHostActionRefreshedAt and bounds from it). Evidence
therefore survives as long as some replica is still actively retrying, while a
deployment that stops sweeping entirely still lets it age out — so this does not
reintroduce the unbounded leak the cap existed to prevent.

Test: after a failed hook, a later cleanup pass keeps the marker pending and moves
its retention basis forward.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq

* fix(schedules): defer settlement when the Stop barrier times out

Codex finding: waitForStopPersistence returned after its 5s poll budget even when
the Stop was still fresh and unacknowledged, and recordScheduleOutcome then went
straight on to terminalize the run — releasing its capacity, deletion, and erasure
barriers while beforePublish may still have been writing. Slow checkpoint cleanup
is indistinguishable from a dead route on that signal, so the timeout was being
treated as if the barrier had been satisfied.

The poll budget now means "undecided", not "clear". On timeout with a fresh,
unacknowledged Stop the barrier DEFERS: recordScheduleOutcome returns false
without recording, leaving the run active/preserved. Settlement then happens
either when the route acknowledges, or once the existing stale-owner cutoff
(ABORT_OWNER_PRESUMED_ALIVE_MS) authorizes a later attempt — which the loop
already treats as clear-to-settle. Callers with durable retry (the approval-expiry
host action, reconciliation) re-drive it, so a deferral converges rather than
stranding the run.

Test: a fresh Stop that never acknowledges within the budget reports not-settled
and records no outcome.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq

* fix(schedules): require definite delivery failure, extract its message, converge deferred Stops

Third Codex round. Three findings in code from this closeout, plus one pre-existing
P1 that is a one-line operator-safety fix.

P1 — dead-delivery settlement demanded too little evidence. `dead` does not prove a
request was rejected: the trigger host marks response timeouts and invalid success
responses `certainty: 'ambiguous'`, and the engine dead-letters those once retries
are exhausted. The erasure sweep treated every dead letter as positive evidence, so
an ambiguous one sitting over a generation a peer had accepted could terminalize the
run and release its capacity mid-flight. It now settles only on a DEFINITE rejection,
unless job absence is deployment-authoritative (safe topology), where the
confirmed-absent job is itself the evidence.

P1 — `lastError` is an `AgentTriggerDeliveryFailure` object, not a string. A
duplicated local interface declared it `string` (against CLAUDE.md's no-duplicate-
types rule), so both the sweep and the engine reconciler passed the object into the
String-typed run/schedule `error` fields; Mongoose would reject the cast, the per-row
catch would swallow it, and the run would keep its global capacity slot. The dep type
now reuses the canonical `AgentTriggerDeliveryFailure` and both call sites pass
`.message`. Re-typing immediately surfaced a stale test that had asserted a string.

P1 (pre-existing) — the base-config global stop is honored in `getLimits` via
`isRuntimeDisabled` rather than a literal `=== false`. The stop has two shapes, and
deepMerge turns base `{ use: false }` plus a principal override of `true` into
`{ use: true }`, so the literal check reported the feature enabled and Run Now
dispatched straight through fireSchedule, bypassing the operator's emergency stop.
Now the same predicate the engine gate already uses.

P2 — a Stop whose settlement DEFERRED past the poll budget had no convergence path
where no reconciler is armed. `acknowledgeScheduledStopPersistence` now optionally
re-drives the terminal outcome once the barrier clears; `recordRunOutcome` is
match-guarded and idempotent, so an owner that already settled makes it a no-op. The
abort route passes it for a running generation; a paused job still settles explicitly.

Tests: ambiguous dead letters refused under unsafe topology but settled when absence
is authoritative, definite rejections settled either way, the failure message carried
through, and the abort route's re-drive present for running / absent for paused.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq

* fix(schedules): converge terminal runs in clustered workers, retry suspension restore

Fourth Codex round on 74f7af8. Both findings are in code from this closeout.

P2 - a clustered worker never settled a run whose generation finished but whose
outcome write failed. `recordScheduleOutcome` retries three times and then returns
false; the owner honors that by PRESERVING the terminal job as the only surviving
evidence, and the armed engine's reconciler replays exactly that. The clustered
entrypoint arms no engine, and its sweep only covered deleting schedules
(settleAbandonedRuns) and dead deliveries - an identity-matched job was skipped
outright. So an ordinary live schedule's run stayed `started`, held its GLOBAL
capacity slot, and kept a preserved job that carries no `completedAt` and is
therefore invisible to the store's finished-job sweep: both leaked until store
expiry.

The live-schedule pass now also converges from a retained terminal job, mirroring
the reconciler branch it stands in for: honor the owner's stamped outcome over the
generic status (so a balance refusal still walks its streak rather than resetting
it), clear the reserved conversationId when the generation never emitted its
created event, and delete the retained job only AFTER the outcome write is durable.

This stays positive-evidence-only and safe in every topology. Presence, not
absence, is the evidence: an identity-matched job is authoritative wherever it is
observed - a shared store shows the real generation, a process-local store can
only be showing this process's own - which is why it needs no
canInferOwnerDeathFromMissingJob fence, unlike the absence-based paths. The
in-flight abort/resume fences still defer, so an `aborted` job cannot settle a run
whose owner may still be persisting. Both cases now share one pass over one window
rather than two, and `retainedOutcome` moved to types.ts so the sweep reuses the
engine's mapping instead of duplicating it (and stays independent of the engine).

P2 - a failed restore stranded a live account's schedules. Cancelling an account
deletion restores the suspended rows while the deletion fence is still armed and
then releases that fence; nothing re-drives the restore afterwards, so one
transient write failure left the user with silently disabled, next-run-less
schedules. The restore is now retried at the single choke point both the HTTP
controller and the CLI share. Retrying is safe because each attempt re-reads only
the rows STILL carrying the token: a partially-applied unordered write converges
on exactly the stragglers, and a fully-applied one finds nothing.

The fence is still released when every attempt fails, deliberately: retaining it
would refuse the live account's schedule writes AND make beginAgentTriggerUserDeletion
report `in_progress` forever, blocking the retry that is the convergence path (a
later attempt adopts this snapshot, so its cancel restores these exact rows). Both
callers now log the user id and suspension token so the state stays recoverable by
hand if that never happens.

Tests: retained terminal job settled and its evidence released, stamped outcome
preferred over the generic status, stamped failure reason carried, reserved
conversationId cleared for a never-created conversation, identity-mismatched
terminal job ignored, abort-in-flight deferred; restore retried past a transient
write failure and converging on a partially applied restore.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq

* fix(schedules): converge unprojected pauses and release replayed bookkeeping jobs

Two of the four findings I flagged as pre-existing to my closeout commits. Both are
in code this PR introduces, so merging would have shipped them; both are the same
capacity/evidence-leak class the last four rounds have been closing. The remaining
two (slotless rows escaping the per-user cap, and the non-rotating `started`
reconciliation bucket) are genuinely latent and deliberately left alone.

An UNPROJECTED PAUSE held a global capacity slot forever. The pause projection is
what moves a run row off `started`; `recordScheduleOutcome` already retries it, but
the request controller discarded the result, so three failed attempts were dropped
silently. The armed engine's reconciler replays that state, but the clustered sweep
did not: a paused job is not terminal, so the retained-job path returned without
settling, and the dead-delivery path never inspects an identity-matched job at all.
The row stayed `started` with no cutoff that would ever clear it.

The call site now surfaces the failure, and the sweep converges it, mirroring the
reconciler's pause branch: project `requires_action` (which frees the slot) but do
NOT release the job's evidence — unlike a terminal job it is still live, awaiting an
approval. The resume hand-off fence still defers, so re-projecting cannot release a
slot a continuation just claimed.

The BOOKKEEPING REPLAY pass leaked its retained job. A run reaches that pass only
because its owner crashed before bookkeeping — which is also before it could release
the job it retained for exactly this recovery. The active-run pass clears its own;
this one never did, and a preserved job is deliberately kept WITHOUT `completedAt`
so the store's finished-job sweep cannot reap it early, so nothing else ever would.
It now clears after `finalizeBookkeeping` succeeds — identity-guarded, a no-op when
no job is retained, and skipped entirely when the replay itself failed, since the
retained job is the only surviving evidence in that case.

Tests: pause projected and the slot freed while the live job's evidence is kept,
paused job deferred during a resume hand-off, retained job released once replayed
bookkeeping is durable, and retained job kept when the replay fails.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq

* fix(schedules): fence the clustered pause replay against a concurrent resume claim

Codex round 5, on code I added in f735341. The finding is correct and the exposure
is one I introduced.

The pause replay lives in a sweep that runs in EVERY clustered replica, so several
sweepers can observe the same unprojected pause. Its guards were all derived from an
in-memory row SNAPSHOT: hasResumeHandoffInFlight reads the snapshot's
resumeClaimedAt, and recordRunOutcome's pause branch matched any row currently in
`started`/`requires_action` with no fence of its own. The race: sweeper A projects
the pause and frees the slot; the owner's approval then claims a fresh one
(markRunResumeClaimed takes the row to `started` WITH resumeClaimedAt in one write);
sweeper B, still holding the pre-projection snapshot, passes its hand-off check and
replays — `$unset: { capacitySlot, resumeClaimedAt }` under a continuation that is
already running. The run reverts to `requires_action` while its generation proceeds
outside global capacity.

The engine's reconciler makes the same call and has the same snapshot-derived guard,
but v1 arms exactly one engine, so it has no concurrent racer; the sweep is the first
thing to run this transition in parallel. Left the engine alone rather than widening
the change: its `requires_action` re-affirmation is deliberate and single-writer.

Fixed where the race is, in the write itself. `recordRunOutcome` takes an optional
`requireNoResumeClaim`, which adds `resumeClaimedAt: { $exists: false }` to the pause
filter, and the sweep sets it. Because the stamp is written in the SAME update that
moves the row to `started`, its absence is atomic proof no resume owns the row. The
flag is deliberately NOT set by the generation owner: its own re-pause legitimately
clears the stamp as the hand-off's completion signal.

The fence cannot block the recovery it exists to enable: markRunResumeClaimed only
matches `requires_action`, so a genuinely stuck `started` row can carry no resume
claim — it is in fact blocking its own approval until this replay frees it.

Tests: a replay from a stale snapshot leaves a resume-claimed row's status, slot, and
claim stamp intact, while a stuck row with no claim still recovers; and the sweep is
asserted to send the fence.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq

* fix(schedules): fence stale resume claims by age, not existence

Codex round 6, on the fence I added in ee00c60. Correct again, and the defect is
the mirror image of the one it fixed.

The fence was an EXISTENCE check (`resumeClaimedAt: { $exists: false }`) while its
caller's guard is a FRESHNESS check (hasResumeHandoffInFlight, bounded by
RESUME_HANDOFF_STALE_MS). They agree while a claim is fresh and disagree once it is
abandoned: a worker that dies after markRunResumeClaimed takes the row to `started`
and stamps resumeClaimedAt — but before the continuation resumes or
releaseRunResumeClaim rolls it back — leaves the stamp set forever. Past the bound
the sweep correctly stops deferring and tries to recover the row, but the write
rejected it purely because the field still existed. The row stayed `started` holding
its global capacity slot, and its approval was unresumable for good, since
markRunResumeClaimed only matches `requires_action`. That is exactly the stuck state
this replay exists to clear, so the fence had reintroduced it for the crashed-resume
case.

`requireNoResumeClaim: boolean` becomes `resumeClaimStaleBefore: Date`, and the
filter matches a row with no claim OR a claim older than that cutoff. The sweep
passes the SAME bound its in-flight check uses, so the two can no longer disagree.
A genuinely racing claim is by construction fresh — it is created after the sweeper's
snapshot — so the race from round 5 stays closed.

Tests: a row whose resume claim was abandoned by a dead worker now recovers (status,
slot and stamp all cleared), alongside the existing two — a fresh claim still repels
a stale-snapshot replay, and an unclaimed stuck row still recovers.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq

---------

Co-authored-by: Claude <noreply@anthropic.com>
2026-08-20 11:51:30 -04:00
Danny Avila
1a3e2aebcb
🛰️ fix: Attach Request-Scoped MCP Servers (#14780)
* fix: attach request-scoped MCP servers

* fix: satisfy MCP static checks

* fix: format MCP runtime hint
2026-08-12 23:43:02 -04:00
Danny Avila
dccef82254
🪶 chore: Aggregate Empty MCP Tool Logs (#14767)
* fix: aggregate empty MCP tool logs

* fix: retain server names in MCP tool logs
2026-08-12 22:29:49 -04:00
Danny Avila
7fc62023eb
🧷 fix: Safely Recover Runtime MCP OAuth Rejections (#14684)
* fix runtime MCP OAuth recovery

* style: sort LC-008 imports

* fix: single-flight runtime OAuth handlers

* fix: retain transport OAuth failures for recovery

* fix(mcp): preserve OAuth recovery connections

* test(mcp): type request-scoped config fixture

* fix(mcp): harden shared OAuth recovery

* fix(mcp): bound OAuth recovery escalation

* style(mcp): sort OAuth integration imports

* fix(mcp): harden OAuth recovery boundaries

* fix(mcp): abort shared recovery waiters

* fix(mcp): bound request OAuth recovery phases

* fix(mcp): close OAuth recovery ownership gaps

* fix(mcp): retry borrowers closed by OAuth recovery

* fix(mcp): drain borrowers before OAuth reconnect

* fix(mcp): preserve eviction across OAuth recovery

* fix(mcp): unify OAuth recovery leases

* fix(mcp): serialize cache reuse with recovery

* fix(mcp): make recovery checkout atomic

* test(mcp): use numeric config timestamp

* fix(mcp): reacquire recovery checkouts

* fix(mcp): retain shared recovery disposal

* fix(mcp): restart checkout after recovery takeover

* fix(mcp): close recovery lifecycle gaps

* refactor(mcp): deepen OAuth recovery lifecycle

* fix(mcp): harden OAuth lifecycle disposal

* style(mcp): sort OAuth lifecycle imports

* fix: lease MCP OAuth lifecycle edges

* fix(mcp): isolate shared OAuth flows from aborts

---------

Co-authored-by: Dennis Schenk <dennis@gridonic.ch>
2026-08-10 10:38:34 -04:00
Danny Avila
1bccc2bc18
📡 fix: Refresh MCP Tools After List-Changed Notifications (#14686)
* fix(mcp): handle dynamic tool list changes

Co-authored-by: Pascal Garber <pascal@artandcode.studio>

* test(mcp): fix CI validation

* fix(mcp): keep dynamic tool catalogs live

* fix(mcp): harden dynamic catalog lifecycle

* test(mcp): use typed startup connection

* test(mcp): isolate dynamic e2e fixtures

* fix(mcp): refresh tools after reconnect

* fix(mcp): close dynamic catalog cache gaps

* test(mcp): update OAuth connection mocks

* fix(mcp): preserve app snapshot ownership

* style(mcp): sort connection imports

* fix(mcp): close review race conditions

* fix(mcp): preserve cache ownership edges

* fix(mcp): harden recovery lifecycle

* fix(mcp): guard tool-less app refresh

* fix(mcp): fence distributed cache races

* fix(mcp): retire stale connection state

* fix(mcp): keep tool snapshots authoritative

* fix(mcp): fence stale app tool publications

* style(mcp): sort repository test imports

* test(mcp): mock empty startup publication

* fix(mcp): preserve app publication generations

* fix(mcp): harden publication recovery races

* fix(mcp): address tool catalogs by runtime config

* fix(mcp): load scoped catalogs for assistant writes

* fix(mcp): harden catalog publication recovery

* fix(mcp): serialize forced connection replacement

* fix(mcp): serialize ordinary creation with replacements

* fix(mcp): harden catalog fallback boundaries

* fix(mcp): close lifecycle fencing gaps

* fix(mcp): preserve catalog authority on failures

* fix(mcp): compensate failed catalog mutations

* fix(mcp): fence catalog refresh ordering

* style(mcp): sort agent loader imports

* fix(mcp): cancel stale connection creation

* fix(mcp): fence catalog coordination

* fix(mcp): close catalog race windows

* fix(mcp): harden cross-pod catalog fencing

* fix(mcp): close catalog lifecycle edges

* style(mcp): sort assistant imports

* fix(mcp): reject stale recovery authority

* fix(mcp): restore static catalog on every startup

* fix(mcp): order app catalog publications

* style(mcp): sort catalog revision imports

* fix(mcp): separate catalog allocation and commit fences

---------

Co-authored-by: Pascal Garber <pascal@artandcode.studio>
2026-08-08 13:50:21 -04:00
Danny Avila
56175af0b5
🎟️ fix: Reconcile MCP OAuth Readiness Across Pods (#14629)
* fix: stabilize MCP OAuth readiness across pods

* fix: harden MCP readiness review findings

* fix: resolve CI type check and terminal OAuth polling

* fix: address MCP OAuth readiness review

* fix: align MCP OAuth readiness state

* test: stabilize MCP OAuth readiness assertion

* fix: reject stale MCP OAuth callbacks

* fix: close distributed MCP OAuth readiness gaps

* style: sort Redis MCP test imports

* fix: preserve MCP OAuth polling across rolling pods

* fix: finalize distributed MCP OAuth readiness

* fix: preserve runtime-detected MCP OAuth

* fix: report runtime MCP OAuth readiness

* fix: preserve live MCP OAuth classification

* style: sort MCP connection imports
2026-08-05 19:42:26 -04:00
Danny Avila
db6ba5392a
🪢 fix: Bind MCP OAuth Secrets to Trusted Endpoints (#14578)
* fix: bind MCP OAuth secrets to trusted endpoints

* fix: bind stored MCP OAuth clients during refresh

* fix: address MCP OAuth review findings

* fix: bind stored MCP OAuth credentials

* fix: make MCP OAuth credentials generation-safe

* test: update MCP OAuth uninstall binding fixtures

* fix: harden MCP OAuth credential persistence

* fix: scope MCP OAuth refresh single-flight

* style: sort MCP OAuth token imports
2026-08-02 13:38:58 -04:00
Danny Avila
6f45a9e32e
🔗 fix: Normalize MCP Tool Keys at Every Producer, Resolve Raw Names via Aliases (#14553)
* 🔗 fix: Normalize MCP Tool Keys at Every Producer, Resolve Raw Names via Aliases

Tool keys had two spellings that could diverge for any server whose name
contains characters outside [a-zA-Z0-9_.-]: the tool cache (and registry
inspector) built keys with the RAW server name, while runtime instances
are named with normalizeServerName(serverName). Three code comments
already asserted "tool keys embed the normalized server name" - no
producer honored it. For a special-character server that meant:

- definitions-only mode shipped raw def names the model echoed back,
  but the executor's tool map held the normalized instance name, so
  every call failed with "Tool not found";
- per-tool tool_options (defer_loading / allowed_callers /
  run_in_background / describe_intent) were persisted under raw keys
  that never matched the definition names the option passes resolve
  against, so builder settings were silently inert;
- tool-key parsing against normalized candidate lists silently fell
  back to last-delimiter splitting, which mis-parses delimiter-bearing
  tool names.

The reconciliation is one contract enforced in three moves:

1. PRODUCERS NORMALIZE. The tool cache (packages/api/src/mcp/tools.ts)
   and the registry inspector build keys with the normalized server
   name, matching the instance names MCP.js has always assigned. The
   builder's tool ids, agent.tools entries, tool_options keys, and
   definition names all flow from these keys, so every model-facing
   name now agrees. The cache STORE stays keyed by the raw config name.

2. CONFIG LOOKUPS RESOLVE ALIASES. New shared helpers in data-provider
   (buildServerNameAliases, normalizeMCPToolKey) map a parsed
   normalized name back to the raw config name that the registry,
   config maps, tool cache, and plugin-auth rows are keyed by. Applied
   in the definitions loader closure, handleTools grouping,
   createMCPTool's parsing fallback, getUserMCPAuthMap, and the MCP
   tools endpoint - matching both spellings so legacy raw keys keep
   resolving.

3. LEGACY DATA HEALS AT ONE BOUNDARY. initializeAgent rewrites
   raw-keyed agent.tools entries and tool_options keys to the
   normalized form (normalizeAgentToolKeys) before anything consumes
   them, so agents persisted under the old convention load their tools
   AND have all four per-tool options honored. Placeholder and
   server-pin tokens stay raw - they are config-identity references,
   not model-facing names.

Servers whose names are already in the safe character set (the common
case) produce byte-identical keys before and after; the fast path
allocates nothing. Stale Redis-cached raw keys self-heal via the
existing reconnect-on-missing path within one cache cycle.

* 🧯 fix: Deterministic Alias Collisions + Raw Names in Definition Metadata

Two review findings on the normalization contract:

- Two configured server names that normalize to the same segment (e.g.
  'Sales Force' and 'Sales:Force' -> 'Sales_Force') produce inherently
  ambiguous tool keys; the alias map silently resolved last-wins, so a
  tool selected from one server could execute against the other's
  config. buildServerNameAliases now resolves collisions to the FIRST
  configured name deterministically, and resolveMCPServerContext warns
  once per colliding pair per process so the operator can rename one
  server. A collision-resistant identifier would change every existing
  tool key, so detection + stable routing is the right treatment here;
  startup-time config validation can follow separately.

- The definitions loader resolved parsed (normalized) server names to
  raw only inside the ToolService closure, while the definition
  metadata (serverName -> mcpRawServerName) kept the normalized value.
  Server instructions are keyed by raw config names, so a
  special-character server's instructions were silently omitted in
  definitions-only mode. loadToolDefinitions now takes rawServerNames,
  resolves the boundary against both spellings, and stores the RAW
  name in definition metadata - consistent with the instance path.

* 🧯 fix: Heal Stale Caches, Skill Allowed-Tools, and Builder Selectors

Three review findings on the normalization rollout, all in the
transition class:

- Stale cache entries (P1): the definitions-only loader treats the
  per-server tool map as authoritative and never reconnects on a
  per-key miss, so a pre-change raw-keyed Redis entry would make a
  special-character server's tools vanish for up to the cache TTL.
  getMCPServerTools now heals legacy raw-keyed entries to the
  normalized format at read time (keys and function names), covering
  every consumer with no coordinated invalidation; safe names return
  the map untouched.

- Skill allowed-tools: a skill declaring a raw MCP key in
  allowed-tools bypassed the initialize-boundary heal (the union runs
  after it) and would neither dedupe against healed agent tools nor
  match the normalized tool map. The primes' allowedTools now pass
  through the same normalizeAgentToolKeys heal before unioning.

- Builder selectors: matchesMcpServer and useVisibleTools parsed tool
  ids against raw server names only, so an attached special-character
  server rendered as an unselected orphan card. Both now accept the
  normalized spelling and resolve it back to the raw map key, keeping
  legacy raw ids working.

* 🧯 fix: Fail Closed on Normalized Server-Name Collisions

Escalation of the collision finding: a deterministic first-wins alias
plus a warning still let the tools listing publish BOTH colliding
servers, so a tool selected under the shadowed second server would
silently execute against the first server's configuration (their
model-facing keys are identical, so routing cannot ever distinguish
them).

- findShadowedServerNames identifies later-configured names whose
  normalized form an earlier different name claimed.
- getMCPTools excludes shadowed servers from the published listing
  entirely (with a warn naming the collision), so their tools are
  never selectable - nothing ambiguous can be picked.
- Server creation reserves both spellings: a generated slug may not
  collide with a raw config name OR the normalized form its tool keys
  would carry.

Collision-resistant model-facing IDs remain out of scope: changing
normalizeServerName's output would rewrite every existing tool key
(agent documents, caches, instance names) for ALL servers to handle a
misconfiguration that is now blocked from exposure instead.

* ✅ fix: Dedupe Reserved Server-Name Spellings at Creation

The reservation list appended normalized forms unconditionally, which
duplicated every safe name (raw === normalized) and broke the
route-level contract test pinning the exact list. Dedupe via a Set so
safe names contribute one entry, while special-character names still
reserve both spellings; adds the special-character reservation case.

* 🧯 fix: Never Heal a Shadowed Server's Keys; Align Authorization Tie-Break

Persisted references were the remaining collision vector: an agent or
skill saved with the shadowed later server's raw key was HEALED into
the shared normalized key, authorized through a last-wins map, and
routed first-wins - authorized as one server, executed as another.

- normalizeAgentToolKeys now refuses to rewrite keys of shadowed
  servers (findShadowedServerNames): rewriting would produce exactly
  the first server's key. Left raw, the key cannot match the
  normalized-keyed tool map and the tool fails visibly - broken beats
  misrouted. Covers agent.tools, tool_options, and skill
  allowed-tools through the shared heal.

- filterAuthorizedTools (agents/v1.js) builds its normalized-to-raw
  map via the shared buildServerNameAliases instead of a last-wins
  Map constructor, so authorization resolves a colliding key to the
  SAME first server execution routes to.

* 🧯 fix: Direct Identity Wins Over Aliases; Heal Client Forms and Degraded Contexts

Four review findings on the normalization edges:

- Alias hijack (P1): a user-DB server named exactly like an operator
  server's normalized form ('foo' vs YAML 'foo!') had its tools
  rerouted to the operator server by unconditional alias resolution.
  Resolution is now DIRECT-FIRST everywhere: the parsed name is tried
  as-is, and only when nothing resolves is it treated as a normalized
  spelling (definitions loader, handleTools grouping, createMCPTool
  fallback). buildServerNameAliases seats identity entries before
  derived ones so a literal name owns its slot regardless of config
  order, findShadowedServerNames and the collision warning derive from
  the same construction, and getUserMCPAuthMap fetches auth under both
  spellings so either owner finds its rows.

- Builder double-match: a normalized name containing the delimiter
  ('foo mcp bar' -> 'foo_mcp_bar') also suffix-matched a server named
  'bar', selecting both cards and making removal strip the wrong tool.
  matchesMcpServer now resolves the token ONCE against the full
  configured list (longest boundary, both spellings) when the caller
  supplies it; selection and removal share the resolution.

- Builder legacy ids: an agent saved with raw-keyed ids showed its
  tools unchecked while the runtime heal kept them active, and
  selection updates never replaced the legacy entries. McpSection maps
  legacy raw ids to their current normalized ids when deriving and
  rewriting this server's selection.

- Degraded context: a transient ensureConfigServers failure returned
  an entirely empty context, leaving normalized keys unresolvable for
  the request. resolveMCPServerContext now keeps the name lists (they
  derive from the config snapshot alone) and degrades only the
  lazy-init configs.

* 🧯 fix: Collision Detection Sees Accessible Servers; Shadowed Refs Fail Closed End to End

Round follow-ups on the collision design, all in the
DB-server-visibility class:

- The legacy-key heal detected collisions against operator-config
  names only, so healing could still produce a key that direct-first
  resolution routes to an invisible user-DB server. initializeAgent
  gains an optional getAccessibleMcpServerNames dep (wired through
  ToolService for controllers that mock it, directly elsewhere),
  consulted ONLY when a configured name needs normalization - zero
  cost for safe-name deployments. The heal then sees the full
  accessible set and skips shadowed servers' keys.

- Wildcard and legacy raw tokens bypassed catalog filtering, letting a
  shadowed server's instances join a run under the same normalized
  names as the winner's. filterAuthorizedTools rejects tools of
  shadowed servers at authorization (its merged map sees DB + config),
  and handleTools skips them at execution.

- The builder migrated only tool selection, not tool_options: legacy
  raw option keys showed disabled while the runtime honored them, and
  toggles could not clear them. McpSection now migrates option keys to
  the current normalized ids (existing normalized entries win).

- A transient ensureConfigServers failure degraded to an EMPTY server
  context, leaving normalized keys unresolvable for the request.
  resolveMCPServerContext keeps the name lists (derived from the
  config snapshot alone) and degrades only the lazy-init configs.

* 🧯 fix: Complete the Collision Audit at Every Gate; Safer Heal Semantics

Round follow-ups hardening the collision audit:

- Execution guards now consult the FULL accessible set: the caller's
  heal threads its already-fetched names through loadTools, and
  handleTools fetches them itself when a configured name needs
  normalization (never for safe-name deployments) - so a cross-tier
  collision (user-DB 'foo' vs operator 'foo!') fails closed at eager
  execution instead of joining the run under one normalized name.

- Healing is SKIPPED when the collision audit cannot complete
  (transient lookup failure, or no dep): un-healed raw keys still
  resolve through the direct-first candidates, so skipping is safe
  while rewriting against an incomplete audit is not.

- The audit lookup is gated on the agent actually carrying
  delimiter-bearing keys (tools, tool_options, or skill
  allowed-tools), so non-MCP agents never pay a registry round-trip
  even on specially named deployments.

- normalizeAgentToolKeys gives the CURRENT (normalized) entry
  precedence when both spellings carry options, matching the builder's
  migration semantics instead of letting insertion order decide.

- The builder's toCurrentToolId resolves entries boundary-exactly
  against every configured server (longest match, both spellings), so
  a raw suffix shared with a LONGER server name can no longer reassign
  that server's selection or options while another dialog is open.

* 🧯 fix: Shared Collision Audit for Definitions Loading; Fail Closed on Audit Failure

Round follow-ups closing the remaining audit gaps:

- The definitions-only loader now consumes the same collision audit as
  eager loading: shadowed servers' entries (wildcards included) are
  dropped before definitions are emitted, so the default execution
  path can never resolve a shadowed server's normalized function name
  to another server. The audit names thread from initializeAgent's
  heal; the loader self-fetches only when a configured name needs
  normalization.

- resolveCollisionAuditNames centralizes the audit-resolution policy
  (threaded set > self-fetch when needed > incomplete on failure), and
  BOTH loaders now fail closed under an incomplete audit: any
  normalization-sensitive reference (its own name needs normalizing,
  or it equals the normalized form of a configured special-character
  name) is skipped with a warning instead of being audited against
  operator names alone. isNormalizationSensitiveName lives in
  packages/api as a pure helper so test mocks use the real predicate.

- normalizeAgentToolKeys collapses duplicate ids after healing
  (order-preserving): a document carrying both spellings converges on
  one key, never two instances with the same function name.

* 🧯 fix: Thread the Audit Everywhere; Identity-Aware Alias Fallback

Round follow-ups on audit plumbing:

- The OpenAI-compatible and Responses tool loaders now forward the
  already-resolved accessibleMcpServerNames instead of discarding it,
  so the definitions loader neither repeats the registry lookup nor
  fails closed on a transient second lookup after the first succeeded.

- The skill-only path threads its audit: when the baseline agent has
  no MCP keys but a primed skill's allowed-tools fetched the complete
  set, that set (not the operator-only list) reaches the loader, so
  the collision remains visible and the shadowed reference stays
  rejected end to end.

- OAuth discovery iterates the collision-FILTERED tool list, so a
  request can no longer emit an OAuth prompt, wait out the connection
  timeout, and reconnect a server whose definitions were deliberately
  rejected.

- The definitions loader's alias fallback is identity-aware: when the
  parsed name IS a known accessible server, a null tool fetch means
  temporarily unavailable (OAuth pending, missing user variables,
  disconnected) and no longer reroutes to the raw alias - previously
  the aliased operator server's definitions could be emitted under the
  unavailable DB server's names.

* 🧯 fix: Legacy-Key Definition Lookup; Retain Audit for Deferred Execution

- createMCPTool resolves tool definitions by BOTH spellings: the key as
  persisted plus the canonical normalized key built from the resolved
  server name. Assistants and direct tool calls persisted before the
  rollout bypass the agent-boundary heal and arrive with raw keys, while
  availableTools is now indexed canonically - previously every such call
  missed the index, burned a reconnect, and returned the unavailable
  stub permanently via the negative cache.

- The initialized agent retains accessibleMcpServerNames (the COMPLETE
  collision audit this initialization resolved), buildAgentToolContext
  copies it into every per-agent tool context, and loadToolsForExecution
  threads it into the eager loader as bare options. Deferred/event-driven
  execution therefore reuses the snapshot instead of repeating the merged
  registry read - a transient failure there could fail-closed a tool the
  same turn already advertised from the successful first audit.

- MCP.spec.js keeps @librechat/api pure helpers REAL (requireActual
  spread) so normalization paths are exercised rather than mirrored.

* 🧯 fix: Parse Legacy Keys Against Both Server-Name Spellings

createMCPTool's boundary candidates were normalized-only, so a legacy
raw key whose server name contains the delimiter (foo_mcp_bar!) missed
the suffix match and fell to the generic last-delimiter split - the
canonical rebuild then produced a key that could never hit the index
and the persisted call stubbed out. The candidate list now carries the
RAW resolved name (and raw config names on the parse-only path) next
to the normalized spellings.

* 🧯 fix: Honest Audit Completeness; Shadowed-Server Form-Key Guard

- resolveAllMcpConfigs tolerates ensureConfigServers failures, so the
  merged registry read can silently omit config-only servers while the
  audit still reported complete: true - a foo/foo! collision would go
  unseen and a persisted key could route to the wrong server. Both
  audit consumers now union the snapshot-derived raw config names back
  in (resolveCollisionAuditNames unions the caller's rawServerNames;
  the initializeAgent heal unions configRawServerNames), keeping the
  completeness label honest without an extra read: operator names come
  from the registry-independent config snapshot, user-DB names from the
  merged read that fails loudly into the existing incomplete path.

- The client tool_options migration now mirrors the runtime heal's
  fail-closed rule for SHADOWED servers: when the dialog's server has
  lost its normalized slot to another catalog name, legacy raw keys
  stay raw instead of being rewritten onto the winning server's key,
  where a later save would apply the wrong server's per-tool settings.
  The dialog's own server joins the alias construction so a stale
  catalog map can't misread as a collision.

* 🧯 fix: Heal Legacy Assistant MCP Tool Names on Save

The assistants create/update controllers look tools up in the cached
definitions by exact key, and the cache is now normalized-keyed - an
assistant saved before the convention resubmits its raw-suffixed MCP
name on every edit, so any save silently removed the tool.

healMcpToolNames pre-heals the payload's tool list: a delimiter-bearing
string that misses the cache resolves through the configured raw names
(longest-suffix, boundary-exact) and rewrites to the normalized key
only when that key actually exists in the cache. SHADOWED raw names
stay raw and fail closed, mirroring the runtime heal; the config read
happens only when a delimiter-bearing name actually misses, and read
failures propagate (write path) rather than silently dropping tools.
v2's update loop also stops re-reading the tool cache per iteration.

* 🧯 fix: Full-Audit Shadow Set + Dedupe in the Assistant Key Heal

- The assistant-save heal built its shadow set from operator config
  names alone, so a cross-tier collision (user-DB `foo` owning the
  normalized slot of operator `foo!`) looked unshadowed and the legacy
  key healed into the shared normalized name - which direct-first
  execution then binds to the DB server. The shadow set now comes from
  resolveCollisionAuditNames' full accessible audit, and an incomplete
  audit skips healing outright (every rewrite candidate is
  normalization-sensitive by construction, so raw-and-fail-closed is
  the only safe answer).

- Healed string entries dedupe order-preserving: a payload carrying
  both spellings of the same tool collapses to one entry instead of
  expanding into duplicate function definitions the provider rejects.
2026-08-01 07:39:24 -04:00
Danny Avila
78ec1940a2
🧹 fix: Clean Up MCP OAuth State Mappings on Uninstall + Reject Superseded Callbacks (#14549)
* 🧹 fix: Clean Up MCP OAuth State Mappings on Uninstall + Reject Superseded Callbacks

Disconnecting an OAuth MCP server deleted its mcp_oauth flows but left the
mcp_oauth_state:{state} mappings behind for the full TTL. Because flow ids
are deterministic (userId:serverName) and the CSRF token is HMAC(flowId), a
stale browser tab's callback could resolve its orphaned state to the NEXT
flow for the same server, pass CSRF, burn the fresh flow's one-shot CSRF
cookie, and fail the PKCE exchange, sabotaging the legitimate retry.

- Add MCPOAuthHandler.deleteFlowAndStateMapping: reads the flow's stored
  state and deletes the mapping before the flow (mapping-first so a crash
  between deletes fails closed instead of recreating the orphan)
- Route mcp_oauth deletions in clearStoredMCPOAuthState through the helper
  for both tenant-scoped and legacy flow ids
- Reject callbacks whose state does not match the resolved flow's stored
  state: the only control distinguishing a superseded attempt from the
  current one on a deterministic flow id

Fixes #14534

* fix: gate failFlow on state match in the OAuth error branch (Codex P1)

The provider-error branch failed the resolved flow on CSRF/session alone,
so a superseded error callback resolved through an orphaned mapping could
mark the current flow FAILED. Apply the same stored-state equality gate
before failFlow.

* fix: leave the flow in place when the state-mapping delete fails (Codex P2)

deleteFlow swallows storage errors and returns false, and
deleteStateMapping discarded that result, so a failed mapping delete
followed by a successful flow delete would silently recreate the orphan.
Surface the boolean from deleteStateMapping and throw from
deleteFlowAndStateMapping before touching the flow, so the caller's
allSettled warn branch fires and the next replacement retries both.

* fix: restore the state mapping when the flow delete fails (Codex P2)

The inverse partial failure of the round-3 fix: a successful mapping
delete followed by a silently failed flow delete left a PENDING flow
whose reused authorization URL could never resolve, dead-ending every
callback in invalid_state until the flow went stale. Check deleteFlow's
result, re-store the mapping on failure, and throw so the caller's
allSettled warn branch fires.

* fix: never leave a callback-capable flow behind on uninstall (Codex round 6)

Teardown runs after the server's tokens are deleted, so a preserved
flow+mapping pair (the round-3 early-throw path) let a lingering consent
tab complete the callback and recreate credentials post-uninstall. Now
that both callback branches gate on stored-state equality, an orphaned
mapping is the benign failure mode, so invert the order: delete the flow
first, attempt the mapping delete regardless, and reject when either
reports a storage failure. This supersedes the round-4 mapping restore,
which also preserved a callback-capable pair.

* fix: delete the flow even when its metadata read fails (Codex round 7)

A storage error on the initial getFlowState aborted teardown before any
delete ran, preserving the callback-capable flow after token deletion.
Tolerate the read failure, delete the flow blindly, skip the mapping it
could not identify (the callback gates neutralize the possible orphan),
and reject so the caller's warn branch fires.
2026-07-31 12:11:36 -04:00
Jens Schumann
ad74a282d1
🪃 fix: Resolve User Vars Before the First Post-OAuth Reconnect (#14538)
* fix: resolve customUserVars before first post-OAuth-callback MCP reconnect

The OAuth callback route reconnects the user's MCP connection immediately
after storing new tokens, but never resolves customUserVars before doing
so - unlike the /reinitialize route a few hundred lines below, which does.
As a result, headers/oauth_headers templates like `{{MY_KEY}}` are sent
to the MCP server literally, unsubstituted, on this first connection
attempt, even though the user's value is already saved. The upstream
server rejects it as an invalid credential.

Fixes #14537

* refactor: share getServerCustomUserVars reader from @librechat/api

The mcp_-prefixed key shape was built by getUserMCPAuthMap but re-derived
by hand at each read site (a private helper in services/MCP.js, and the
new callback-route extraction). Export a reader from the same module that
owns the writer and reuse it at both sites, so the key shape has a single
source of truth.

* chore: sort destructured require members in routes/mcp.js

---------

Co-authored-by: Danny Avila <danny@librechat.ai>
2026-07-30 22:17:06 -04:00
Danny Avila
8af6414e13
🪟 fix: Surface MCP Initialization Errors (#14529) 2026-07-30 13:22:11 -04:00
Danny Avila
49f4b659f6
🔐 fix: Honor Admin-Panel MCP Allowlist Overrides Without Restart (#13814)
* 🔐 fix: Honor Admin-Panel MCP Allowlist Overrides Without Restart

MCPServersRegistry was built once at boot from getAppConfig({ baseOnly:
true }), freezing allowedDomains/allowedAddresses to YAML. Admin-panel
mcpSettings overrides were ignored by both inspection (addServer/
reinspectServer/updateServer/lazyInitConfigServer) and runtime connection
enforcement (assertResolvedRuntimeConfigAllowed), so a domain allowed only
via the panel failed inspection and never connected.

Make the registry's effective allowlists mutable and refresh them from the
merged admin-panel config: seed at boot, and re-apply on every config
mutation via invalidateConfigCaches -> clearMcpConfigCache. Both inspection
and connection paths read the same getters, so both honor overrides without
a restart. Fail-safe: current allowlists are preserved when the merged read
fails.

* 🛡️ fix: Scope MCP allowlist refresh to global config, fail-safe on DB error

Address Codex P1 review findings on the allowlist-refresh path:

- Tenant-scoped config mutations no longer push one tenant's merged
  mcpSettings into the process-wide registry singleton (read by all MCP
  connection paths), which would leak allowlists across tenants. Only
  global (non-tenant) mutations refresh the registry; tenant mutations
  still evict the config-server cache.
- The refresh read now uses strictOverrides:true so a transient DB error
  throws instead of silently returning YAML base config — preserving the
  last-known allowlists rather than overwriting them with fallback values.
  Adds the strictOverrides option to getAppConfig (default off, no behavior
  change for existing callers).

* ♻️ refactor: Resolve MCP allowlists per-request (tenant-scoped) instead of a global singleton

Supersedes the prior global-mutation approach. MCP allowlists live in
mcpSettings, which is tenant/principal-scoped admin config, so a process-wide
singleton value is the wrong model — it caused cross-tenant bleed and stale
reads.

Instead, inject a resolver (from the app layer, where the merged config lives)
that the registry calls per inspection and per connection. It reads the ALS
tenant context via getAppConfig and accepts the acting user so user/role-scoped
overrides resolve; config-source inspection (no user) resolves at tenant scope.
Falls back to the YAML base allowlists when no resolver is set or the lookup
fails, so a transient error fails to the operator baseline rather than
disabling the allowlist.

Removes the now-unnecessary setAllowlists / boot-seed / invalidateConfigCaches
refresh / getAppConfig.strictOverrides machinery.

* 🔒 fix: Scope config-source cache by allowlist; resolve OAuth allowlists per-request

Address Codex review of the per-request resolver:

- Config-source cache key now folds in the resolved allowlists, not just the
  raw-config hash. Inspection results became allowlist-dependent, so without
  this a tenant whose allowlist rejects a URL could poison the shared key with
  an inspectionFailed stub for a tenant that allows it (and vice versa). The
  tenant-scoped allowlist is resolved once per ensureConfigServers pass and
  threaded through the cache key + inspection.
- The two remaining request-time OAuth allowlist reads now use the merged
  config instead of the YAML base getters: the fallback OAuth-initiate path
  (routes/mcp.js) via resolveAllowlists, and OAuth revocation
  (UserController.maybeUninstallOAuthMCP) via the request's already-merged
  appConfig.mcpSettings. Without this, an OAuth endpoint allowed only by an
  admin-panel override was rejected while inspection/connection allowed it.

* ✅ test: Update MCP OAuth registry/config mocks for per-request allowlists

CI fix for the Finding-12 change. The OAuth-initiate route now calls
registry.resolveAllowlists() and the revocation path reads the merged
appConfig.mcpSettings, so the affected specs' mocks were asserting the old
base-getter values:
- routes/__tests__/mcp.spec.js: add resolveAllowlists to the registry mock.
- UserController.mcpOAuth.spec.js: provide mcpSettings on the getAppConfig
  mock so revokeOAuthToken still receives the expected allowlists.

* 🧪 test: e2e proof that admin-panel MCP allowlist override takes effect

Adds a Playwright mock-harness spec for #13809. A URL-based MCP fixture
(e2e-http, streamable-http SDK server) boots inspectionFailed because its
origin is omitted from the YAML mcpSettings.allowedDomains; the spec adds that
origin via an admin config override (PUT /api/admin/config/user/:id) and
asserts the server reinitializes — exercising the real resolver path through
the backend + DB. Before the fix, reinspection used the frozen YAML allowlist
and the server stayed unreachable.

- e2e/setup/fake-mcp-http-server.js: streamable-HTTP MCP fixture (health GET /).
- e2e/playwright.config.mock.ts: boot the fixture as a second webServer.
- e2e/config/librechat.e2e.yaml: mcpSettings.allowedDomains (excludes 127.0.0.1)
  + the e2e-http server.
- e2e/specs/mock/mcp-allowlist-override.spec.ts: login → baseline reinit fails →
  apply override → reinit succeeds.
2026-06-17 20:14:53 -04:00
Dustin Healy
054fa4bfa7
🥽 fix: Restrict MCP Server URL Disclosure to Admins, Owners, and Editors (#13784)
* 🥽 fix: Redact Non-User-Sourced MCP Server URLs by ACL Edit Permission

GET /api/mcp/servers and GET /api/mcp/servers/:serverName return MCP server configs to any caller with MCP-use permission. For user-sourced configs (DB-stored, UI-submitted), the URL is the caller's own and is intentionally disclosed. For non-user-sourced configs (YAML or config-tier, operator-defined), the URL and OAuth flow endpoints (authorization_url, token_url) are operator-sensitive: they can encode internal infrastructure hostnames and are not editable through the API.

This change redacts those fields on non-user-sourced configs unless the caller has edit authority on the resource, using the same ACL check (PermissionBits.EDIT) that the PATCH and DELETE routes already enforce via canAccessMCPServerResource. Callers with broad MANAGE_MCP_SERVERS capability bypass the per-resource check, matching the existing capability bypass in canAccessResource. customUserVars is intentionally not redacted: its values are UI hint metadata (title, description, sensitive), not user-supplied secrets; blanking it would give non-editor callers a Configure form with no field labels.

* 🥽 fix: Correct getResourcePermissionsMap import path + tighten redact comments

The MCP server redaction commit imported getResourcePermissionsMap from ~/server/controllers/PermissionsController, but that controller is a consumer of the helper, not its exporter. The canonical export lives in ~/server/services/PermissionService (which controllers/agents/v1.js already imports from). Fixes the runtime getResourcePermissionsMap is not a function failure on GET /api/mcp/servers and the four downstream route-spec failures whose config mocks lacked a source field and were therefore wrongly treated as non-user-sourced; mocks now reflect the real registry behavior (addServer/updateServer tag DB-stored configs with source: 'user'). Trims narrating JSDoc on the redact helpers and resorts the librechat-data-provider destructure by length.

* chore: import order

* 🥽 fix: Redact OAuth Revocation Endpoint Alongside Authorization And Token URLs

The OAuth-URL strip path only dropped authorization_url and token_url. The UserOAuthOptionsSchema in packages/data-provider/src/mcp.ts (line 146) accepts revocation_endpoint as another operator-configurable URL, and the OAuth handler uses it to revoke tokens; it can hold the same internal IdP hostnames the existing strip is trying to hide. Adds revocation_endpoint to the destructure so a non-user-sourced YAML/config MCP server config no longer leaks the revocation URL to non-editor callers. The existing strip url and oauth flow URLs spec is extended with a revocation_endpoint value to lock in the new field.

* 🥽 fix: Gate Shared DB Server URL Disclosure On ACL Edit Permission

source-driven URL disclosure was incorrect for shared DB-backed MCP servers. ServerConfigsDB.mapDBServerToParsedConfig (packages/api/src/mcp/registry/db/ServerConfigsDB.ts:465) sets source: 'user' on every DB-stored config it returns, regardless of who is accessing it. A user with only VIEW share on a DB server, or with agent-mediated access, was therefore treated by the redaction layer as if they owned the URL, and GET /api/mcp/servers disclosed the owner's URL and OAuth flow URLs to viewers who could not edit the resource.

The redaction is now driven purely by ACL edit authority: computeCanEditByServer routes every dbId-bearing config through PermissionBits.EDIT regardless of source; redactServerSecrets strips on !canEdit regardless of source. POST and PATCH controllers explicitly pass canEdit: true since both endpoints establish edit authority (POST creates the resource, PATCH is gated on the EDIT middleware). Legacy/ephemeral configs without a dbId still fall back to the source heuristic.

* 📝 docs: correct redactServerSecrets URL-disclosure comment

---------

Co-authored-by: Danny Avila <danny@librechat.ai>
2026-06-16 11:20:52 -04:00
Danny Avila
49859c04a2
🗄️ fix: Gate Request-Scoped MCP Servers Out of Persistent Tool Cache (#13672)
* 🗄️ fix: Gate Request-Scoped MCP Servers Out of Persistent Tool Cache

PR #13626 established that request-scoped MCP servers (runtime
OPENID/GRAPH/BODY placeholders) must not use the persistent 12h tool
cache, but only gated three of five touchpoints. The panel endpoint
still back-filled the cache and the OAuth callback still wrote to it,
while agent loading read those entries ungated — pinning ephemeral
model-spec/agent toolsets to stale definitions for up to 12h.

Centralize the invariant in createMCPToolCacheService: a getServerConfig
resolver dep gates both writers and a new service-owned getMCPServerTools
read, so every current and future caller is covered. Callers that already
hold the parsed config pass it to skip resolution; the per-call skipCache
flag and duplicated call-site gates are removed in favor of the single
config-based mechanism. Resolution failures fail open to preserve prior
behavior.

* 🩹 fix: Address Codex Review on Cache Gating

- Repair getCachedTools.spec.js, which destructured the relocated
  getMCPServerTools directly from the module; its coverage now lives in
  the service-level tools.spec.ts.
- Resolve the merged (Config-tier-aware) server config in the OAuth
  callback before writing tool definitions, so the cache gate detects
  request-scoped servers supplied via admin Config overlays that the
  base registry lookup cannot see.
- Discover tools actively for request-scoped servers in the panel
  endpoint via ephemeral reinitialization: such servers have no stored
  app/user connections, so the previous getServerToolFunctions fallback
  returned an empty toolset once the cache read was gated.

* 🧵 fix: Address Second Codex Review on Cache Gating

- Resolve the merged server config before the OAuth callback reconnects,
  so the connection itself uses Config-tier overlays rather than only
  the subsequent cache write.
- Pass Config-tier candidates into the panel's request-scoped discovery,
  matching the reinitialize route: reinitMCPServer forwards configServers
  (not the provided serverConfig) to its OAuth discovery fallback.
- Document the accepted read-path trade-off: the gate resolver sees base
  configs only, all writers pass merged configs, so a pre-gating or
  overlay-divergent entry survives at most one cache TTL.

* 🚏 chore: Rework Cache Gating for BODY-Only Request Scoping

After #13673 narrowed requiresEphemeralUserConnection to BODY
placeholders, the central gate follows the predicate unchanged, but the
panel's active discovery no longer serves a purpose: the only remaining
request-scoped class cannot connect outside a chat turn, so the
reinitialization attempt would always fail at the missing-body check.
Remove that path; OpenID/Graph servers are persistent user-scoped again
and flow through the stored-connection and cache lookups as before.

Flip test fixtures that used OPENID placeholders to denote
request-scoped configs over to BODY placeholders.

* 🪟 fix: Check Config Overlays in Agent-Loading Cache Reads

The cache service's registry resolver sees only base YAML/DB configs, so
a BODY placeholder introduced by a request-tier Config overlay was
invisible to the gate on the agent-loading read path: model-spec and
ephemeral-agent expansion could read a leftover persistent entry and pin
stale concrete tool names instead of the mcp_all fresh-discovery path.

Check the raw overlay candidate inline in loadEphemeralAgent and
loadAddedAgent — a pure placeholder scan with no extra IO — and skip the
cache read when the overlay makes the server request-scoped. Widen
UserScopedConnectionConfig so raw (pre-inspection) configs qualify for
the scoping predicates, which only check key presence.

* 🧪 test: Guard Run-Scoped MCP Definition Handoff Boundaries

The original ClickHouse breaker storm regressed precisely at field
pass-through boundaries that unit tests of each end could not see:
initializeAgent dropping mcpAvailableTools from its destructure, and the
agent tool context losing it on the way into ON_TOOL_EXECUTE. Add direct
guards on both hops: the loadTools result must surface on the
initialized agent, and the captured toolExecuteOptions closure must
forward it to loadToolsForExecution.
2026-06-13 11:26:49 -04:00
Danny Avila
c27d6b85a4
🤫 refactor: Silent MCP OAuth Refresh on Mid-Session 401 (#13369)
* 🤫 fix: Silent MCP OAuth Refresh on Mid-Session 401

Avoids the hourly interactive re-auth prompt when an MCP server
(e.g. Azure Entra ID) returns 401 mid-session by attempting a refresh
token exchange first, and only falling back to the interactive OAuth
flow when no refresh token is stored or the refresh server rejects it.

Resolves #13364.

* fix: Use distinct flow type for silent token refresh to avoid cache hit

Addresses the Codex review on PR #13369: `attemptSilentTokenRefresh` was
reusing the `'mcp_get_tokens'` flow type, so
`FlowStateManager.createFlowWithHandler` would short-circuit and return
the same tokens cached by an earlier `getOAuthTokens` call — the very
tokens the server just rejected — without executing the forced-refresh
handler.

Switch silent refresh to the distinct `'mcp_force_refresh_tokens'` flow
type so coalescing still works but stale `mcp_get_tokens` cache entries
are not reused. After a successful refresh, invalidate the
`mcp_get_tokens` flow cache so the next `getOAuthTokens` call reads the
freshly persisted tokens from storage rather than the stale cached
value.

Add a regression test that simulates the real
`FlowStateManager.createFlowWithHandler` cache-hit behavior for
`mcp_get_tokens` and verifies the silent refresh handler still runs and
returns the freshly refreshed tokens.

* fix: Address Codex round-2 review on silent MCP OAuth refresh

Three follow-up findings from Codex on PR #13369:

1. The new `mcp_force_refresh_tokens` flow type was itself cached by
   `FlowStateManager.createFlowWithHandler`, so a subsequent 401 within
   the refreshed token's `expires_at` could re-serve the just-rejected
   token without ever re-running the refresh handler.

2. The factory's `oauthRequired` listener was removed immediately after
   the initial `attemptToConnect` succeeded, so a real mid-session 401
   emitted by `MCPConnection.connectClient` during transport recovery
   had no listener — the OAuth handled-promise would simply time out
   instead of triggering the silent refresh.

3. Routing the silent refresh through a distinct flow type broke
   coalescing with the `mcp_get_tokens` lock used by `getOAuthTokens`,
   letting two paths concurrently redeem the same stored refresh token.
   For providers that rotate refresh tokens (e.g. Azure Entra) the
   second redemption is rejected, kicking the user back into interactive
   OAuth despite a successful refresh elsewhere.

Resolution:

- Drop `FlowStateManager` from the silent-refresh path entirely. Replace
  with a process-local `inflightSilentRefreshes` Map keyed by
  `userId:serverName` that holds only the in-flight Promise (no cached
  result), so every fresh 401 after settlement triggers a fresh
  redemption while concurrent 401s for the same user/server still share
  one redemption.
- Stop calling `cleanupOAuthHandlers()` on successful initial connect,
  keeping the OAuth handler attached for the connection's lifetime so
  mid-session 401s actually reach `attemptSilentTokenRefresh`.
- Add a regression test reproducing the stale-cache scenario by faking
  the `mcp_get_tokens` cache hit and asserting silent refresh still runs
  against storage and returns the fresh tokens.
- Add a coalescing test asserting two concurrent oauthRequired events
  for the same user/server result in a single `forceRefreshTokens` call.
- Clear `inflightSilentRefreshes` in `beforeEach` to prevent
  cross-test leakage; switch the silent-refresh test mocks to
  `mockResolvedValueOnce` / `mockImplementationOnce` so leftover mock
  state cannot leak into later test cases.

Acknowledged remaining gap: the silent refresh still races
`getOAuthTokens`'s `mcp_get_tokens` flow when both run concurrently
(narrow window when an existing connection's local `expires_at` is
still valid but the server invalidated the token, and a new connection
is being created in parallel). The race is self-healing on the next
401 and documented inline.

* fix: Address Codex round-3 review on silent MCP OAuth refresh

Three more findings from Codex on PR #13369:

1. The in-flight silent-refresh promise was unbounded. If
   `forceRefreshTokens()` ever hung (slow provider, dropped TCP), the
   `inflightSilentRefreshes` lock stayed occupied forever and every
   later 401 for the same user/server joined the stuck promise instead
   of starting a fresh attempt or falling back to interactive OAuth.

2. The interactive-OAuth fallback didn't invalidate the
   `mcp_get_tokens` flow cache after persisting fresh tokens. For
   providers that don't issue refresh tokens (so silent refresh
   returns null), the old cache could still feed stale access tokens
   to the next `getOAuthTokens` call until its TTL expired — causing
   an immediate reconnect with the same just-rejected token.

3. When silent refresh failed, the handler fell through to
   `handleOAuthRequired()` whose recent-completion fast path can
   reuse a COMPLETED `mcp_oauth` flow within `PENDING_STALE_MS`. Those
   cached tokens are exactly the ones the server just rejected, so
   the connection would keep adopting them and looping on 401s until
   the cache aged out.

Resolution:

- Wrap `runSilentRefresh()` with a 60-second `withTimeout` (well under
  `connectClient`'s 120s OAuth timeout). On timeout the `.catch`
  resolves to null and the `finally` clears the in-flight entry, so
  the next 401 starts fresh and falls through to interactive OAuth.
- Extract two helpers — `invalidateGetTokensFlow` and
  `invalidateCompletedOAuthFlow` — and call them from the right
  branches: clear `mcp_get_tokens` after silent-refresh success AND
  after interactive-OAuth `storeTokens`; clear the COMPLETED
  `mcp_oauth` state (plus its CSRF mapping) before falling through to
  interactive OAuth so the fast-reuse path can't re-serve the
  rejected tokens.
- Add three regression tests: hung refresh release-the-lock under
  fake timers, completed-OAuth cache invalidation pre-fallback, and
  `mcp_get_tokens` invalidation after interactive token store.

* fix: Address Codex round-4 review on silent MCP OAuth refresh

Three more findings from Codex on PR #13369:

1. (P1) The silent-refresh in-flight lock keyed only by
   `userId:serverName`. In multi-tenant setups where two tenants share a
   userId (e.g. username-based IDs) and the same MCP server name, a
   concurrent mid-session 401 from tenant B would join tenant A's
   in-flight refresh and adopt tenant A's freshly minted tokens onto a
   tenant-B connection — a cross-tenant credential leak.

2. (P2) `invalidateGetTokensFlow` deleted the `mcp_get_tokens` flow
   state regardless of its status. When another connection was
   currently in `getOAuthTokens()` (PENDING flow) and joiners were
   monitoring it, the unconditional delete made those waiters see
   "Flow state not found" and unnecessarily fall back to interactive
   OAuth — even though fresh tokens were already being written.

3. (P2) The 60s `withTimeout` wrapping `runSilentRefresh()` only races
   the promise; it does not cancel the underlying `forceRefreshTokens`
   /  refresh-token HTTP request. If the request returned after a
   subsequent interactive OAuth had stored newer tokens, the late
   completion would `storeTokens` over the newer state. This requires
   a provider that doesn't rotate refresh tokens AND a refresh slower
   than 60s AND a successful interactive OAuth in that window — narrow
   but real.

Resolution:

- Capture `getTenantId()` into a new `factory.tenantId` field at
  factory construction time (before the OAuth handler closes over it
  outside the original request's async context) and include it in the
  silent-refresh lock key as `tenantId:userId:serverName`.
- `invalidateGetTokensFlow` now calls `getFlowState` first and only
  deletes when `status === 'COMPLETED'`. PENDING lookups are left
  alone so concurrent `getOAuthTokens` waiters via `monitorFlow` can
  still settle.
- For (3), document the race as a known limitation inline. Fully
  closing it requires threading an `AbortSignal` through
  `MCPTokenStorage.forceRefreshTokens` and the OAuth refresh handler
  to skip the late `storeTokens` after timeout — out of scope for this
  PR's surgical change.
- Add `getTenantId` to the `MCPOAuthConnectionEvents` test's
  `@librechat/data-schemas` mock so the factory constructor doesn't
  blow up under that suite.
- Add three regression tests: per-tenant lock isolation, PENDING-state
  preservation under `invalidateGetTokensFlow`, and (reused) the
  existing interactive-store invalidation test now driven through
  `getFlowState` returning the COMPLETED state.

* fix: Address silent MCP OAuth refresh review

Restore captured tenant context around token storage and OAuth fallback paths so mid-session callbacks do not lose tenant scope.

Thread AbortSignal through forced refresh and OAuth token requests, cap silent refresh by the connection OAuth timeout, and prevent timed-out refreshes from writing stale credentials after fallback.

Complete pending mcp_get_tokens flows with fresh tokens, add missing FlowState createdAt test fixtures, and cover the new tenant/abort/cache behaviors.

* fix: Tighten tenant-scoped MCP token refresh

Cap silent refresh by both the factory connect timeout and the connection OAuth wait timeout so fallback OAuth wins before the outer connect attempt expires.

Tenant-scope mcp_get_tokens flow ids for both token lookup and refresh invalidation, preventing cross-tenant flow completion or cache deletion when tenants share user ids and server names.

Add regression tests for the omitted initTimeout budget and tenant-prefixed token flow locks.

* fix: Reserve MCP OAuth fallback budget

* fix: Harden MCP OAuth refresh races

* fix: Keep MCP OAuth fallback route-compatible

* test: Add SDK MCP OAuth refresh repro

* fix: Address MCP OAuth refresh review findings

* fix: Address MCP OAuth tenant review findings

* fix: Close MCP OAuth route tenant gaps

* fix: Preserve MCP OAuth refresh flow guards

* fix: Avoid reprocessing MCP OAuth reauth config

* fix: Release timed-out MCP refresh locks

* fix: Release MCP OAuth request callbacks

* fix: Tenant-scope remaining MCP OAuth flow lookups

* ci: Sort imports in MCP OAuth test suites
2026-06-10 13:12:42 -04:00
Danny Avila
a7f16911b2
⏳ fix: Extend and Decouple MCP OAuth Flow Timeouts (#13622)
* ⏳ fix: Extend and decouple MCP OAuth flow timeouts

The OAuth auth button disappeared after 2 minutes (the internal OAuth
handling timeout) while the flow state lived for 3 minutes, leaving users
who didn't click immediately stuck in an unrecoverable re-auth loop. The
handling timeouts also reused the connection/init timeout, so a short
initTimeout would shrink the OAuth window further.

- Add MCP_OAUTH_HANDLING_TIMEOUT (10m) and MCP_OAUTH_FLOW_TTL (15m) to mcpConfig
- Decouple the reactive/proactive OAuth waits from initTimeout/connectionTimeout
- Use OAUTH_FLOW_TTL for the FlowStateManager TTL and the UI status window
- Ensure the flow TTL outlives the handling timeout, fixing the
  "Flow state not found" race
- Remove dead FLOW_TTL constant and document new env vars

Fixes #13615

* ⏳ fix: Coordinate OAuth pending window with handling timeout

Address Codex review: the extended OAuth wait was still capped by other
timeouts that were not updated.

- Align PENDING_STALE_MS (button validity + pending-flow reuse window)
  with MCP_OAUTH_HANDLING_TIMEOUT so a flow stays reusable for the full
  wait instead of 2 minutes (Finding 3)
- Clamp MCP_OAUTH_FLOW_TTL to never fall below the handling timeout so a
  callback near the deadline still finds its flow state (Finding 2)
- Floor attemptToConnect's timeout to the handling window for OAuth
  servers so the reactive in-connect OAuth wait is not killed by the
  30s connection timeout (Finding 1)
- Update flow staleness tests to reference the threshold symbolically

* ⏳ fix: Align OAuth window across status, action flows, and client polling

Address Codex round 2: extending the server wait exposed three more
windows that were still capped or now over-extended.

- checkOAuthFlowStatus reports a PENDING flow as active only within the
  usable PENDING_STALE_MS window, not the longer Keyv retention TTL, so
  the connect button reappears instead of a stuck 'connecting' state
- Give Action (custom tool) OAuth its own FlowStateManager on the prior
  3-minute TTL so the longer MCP OAuth TTL can't leave an action tool
  call waiting up to 15 minutes
- Extend the MCP server-card client polling to the 10-minute handling
  window so a user who completes OAuth after 3 minutes is still picked up

* 🧪 test: Make stale-flow CSRF test track PENDING_STALE_MS

The CSRF-fallback stale-flow test hardcoded a 3-minute age, which is now
within the 10-minute PENDING_STALE_MS window and was wrongly treated as
active. Derive the age from PENDING_STALE_MS so it tracks the constant.

* ⏳ fix: Add grace buffers and surface OAuth timeout to the client

Address Codex round 3 (near-deadline edges):

- Clamp MCP_OAUTH_FLOW_TTL to handling timeout + 60s grace (not equality),
  so flow state outlives the wait instead of expiring at the same instant
- Extend attemptToConnect's OAuth floor by a 60s grace so a user who
  authorizes near the deadline still gets the post-OAuth reconnect
- Surface OAUTH_HANDLING_TIMEOUT on the connection-status response and
  have the client poll for the configured window instead of a hardcoded
  10 minutes, so a tuned server deadline isn't capped on the client

* ⏳ fix: Refresh client OAuth timeout from the first status refetch

If the connection-status cache is empty when polling starts, the client
captured the 10-minute fallback and never picked up a tuned oauthTimeout.
Re-read it after each refetch so a longer configured deadline is honored
even on a cold cache.

* 📝 refactor: Type oauthTimeout on MCPConnectionStatusResponse

Declare the oauthTimeout field on the shared response type in
data-provider instead of an ad-hoc inline cast in the client hook, and
replace the pre-existing 'as any' on the status query read with the
typed getQueryData. Type-level only; no runtime change.
2026-06-09 17:50:02 -04:00
Danny Avila
da5876331e
🔐 fix: Reuse MCP OAuth Authorization URL (#13532)
* fix: reuse MCP OAuth authorization URL

* fix: validate MCP OAuth initiate flow ID
2026-06-05 17:18:59 -04:00
jcbartle
268f095c1a
🔒 feat: Add On-Behalf-Of (OBO) token exchange support for MCP Servers (#13429)
Some checks failed
Docker Dev Images Build / build (Dockerfile, librechat-dev, node) (push) Waiting to run
Docker Dev Images Build / build (Dockerfile.multi, librechat-dev-api, api-build) (push) Waiting to run
GitNexus Index / index (push) Waiting to run
GitNexus Index / post-index (push) Blocked by required conditions
Sync Locize Translations & Create Translation PR / Sync Translation Keys with Locize (push) Waiting to run
Sync Locize Translations & Create Translation PR / Create Translation PR on Version Published (push) Blocked by required conditions
Sync Helm Chart Tags / Ignore non-main push (push) Waiting to run
Sync Helm Chart Tags / Sync chart tags (push) Waiting to run
Publish `librechat-data-provider` to NPM / pack (push) Has been cancelled
Publish `@librechat/data-schemas` to NPM / pack (push) Has been cancelled
Publish `librechat-data-provider` to NPM / publish-npm (push) Has been cancelled
Publish `@librechat/data-schemas` to NPM / publish-npm (push) Has been cancelled
* Add OBO (On-Behalf-Of) token exchange support for MCP server connections

Enables transparent authentication to Entra ID-backed MCP servers using the logged-in user's federated token via the OAuth 2.0 jwt-bearer grant. Configured via obo.scopes in librechat.yaml server config.

- Extract generic OboTokenService from GraphTokenService (jwt-bearer grant + cache)
- Refactor GraphTokenService to thin wrapper delegating to OboTokenService
- Add obo schema field to BaseOptionsSchema in data-provider
- Add resolveOboToken in packages/api/src/mcp/oauth/obo.ts (validates federated token, calls resolver, returns MCPOAuthTokens)
- Wire oboTokenResolver through MCPConnectionFactory, MCPManager, UserConnectionManager
- OBO tokens injected via request headers (not OAuth transport), refreshed on each tool call
- Explicit error on OBO failure (no fallthrough to standard OAuth redirect)
- Add unit tests for both resolveOboToken (9 tests) and exchangeOboToken (14 tests)

* Add OBO authentication option to MCP server UI configuration

  Enable users to configure On-Behalf-Of (OBO) token exchange for MCP servers created via the UI (MongoDB-stored), in addition to the existing YAML-based configuration.

  - Add "On-Behalf-Of (OBO)" radio option to MCP server auth section with scopes input field
  - Remove obo from omitServerManagedFields so the field passes UI schema validation
  - Add OBO to AuthTypeEnum, obo_scopes to AuthConfig, and OBO handling in form defaults and submission
  - Add .min(1) validation on obo.scopes to reject empty strings
  - Add English localization keys: com_ui_obo, com_ui_obo_scopes, com_ui_obo_scopes_description
  - Add 5 schema validation tests for OBO field acceptance, transport compatibility, and edge cases

* 🧊 fix: Add obo to safe properties in redactServerSecrets. Fixes the OBO configuration not showing up in the MCP UI after app restart

* Address linter errors

* 🧊 fix: fail closed on OBO refresh errors and retry transient token exchange failures

- stop tool calls from falling back to stale Authorization headers when per-call OBO refresh fails
- add one-time retry for transient Entra OBO exchange failures (network/429/5xx)
- preserve structured OBO failure reasons and retryability in resolveOboToken
- improve OBO auth error messaging for connection setup and tool execution
- add tests for transient vs permanent OBO failure paths

* Addressing linting errors / warnings

* 🧊 fix: isolate OBO MCP auth to user-scoped connections

- block OBO-enabled servers from app-level shared MCP connections
- bypass shared connection lookup for OBO servers in MCPManager.getConnection
- add regressions covering OBO connection scoping and preserve non-OBO app connection reuse

* 🛠️ refactor: centralize MCP user-scoped connection policy

- add shared requiresUserScopedConnection helper for OAuth, OBO, and customUserVars
- use the shared predicate in MCPManager and ConnectionsRepository
- add utils coverage for user-scoped connection policy

* 🧊 fix: restrict MCP OBO config to header-capable transports

- Move OBO configuration out of the shared MCP base options schema and allow it
only on SSE and streamable-http transports, where request headers are applied.
- Explicitly reject OBO on stdio and websocket configs to avoid accepted-but-
nonfunctional server definitions. Add schema coverage for admin/config parsing
and user-input websocket validation.

* 🧊 fix: single-flight concurrent OBO token exchanges

Concurrent tool calls that arrive on a cache miss were each issuing
their own jwt-bearer request to the IdP. Under that fan-out, Entra
intermittently returned errors that the retry classifier saw as
non-retryable, surfacing as:

  "The identity provider rejected the OBO token exchange.
   Cannot execute tool <name>. Re-authenticate the user or
   verify the configured OBO scopes and retry."

A user retry then hit the populated cache and succeeded, which matches
the observed flakiness — the cache was empty at the moment of fan-out
but populated by the time the user clicked retry.

- Coalesce concurrent exchanges in `OboTokenService.exchangeOboToken`
keyed by `${openidId}:${scopes}`. Callers that arrive while an exchange
is in flight share the same upstream request and receive the same
result. `fromCache=false` continues to force a fresh, independent
exchange (and is not joined by `fromCache=true` callers). The IdP
call, single-retry path, and cache write are unchanged — they were
moved into a `performOboExchange` helper so the coalescing wrapper
stays small.
- Tests cover: coalescing on the same key, isolation between different
keys, cleanup on success, cleanup on failure, and the
`fromCache=false` bypass.

* 🔒 feat: gate MCP OBO config behind MCP_SERVERS.CONFIGURE_OBO permission

OBO silently mints per-user delegated tokens from the caller's federated
access token and forwards them to whatever URL the server config points at.
Previously, anyone with MCP_SERVERS.CREATE could configure obo.scopes — so
if server creation is ever delegated beyond admins, a user could stand up
an attacker-controlled server, attach it to a shared agent, and exfiltrate
other users' downstream tokens on tool invocation.

Add a dedicated MCP_SERVERS.CONFIGURE_OBO permission (ADMIN: true, USER:
false by default) and enforce it at three layers so the safety property
no longer depends on CREATE staying admin-only:

- Create/update: POST/PATCH /api/mcp/servers returns 403 when the body
  carries `obo` and the caller's role lacks the permission.
- Runtime fail-closed: for DB-sourced configs, MCPConnectionFactory and
  MCPManager.callTool re-check the original author's role before each
  OBO exchange. If the author has been downgraded, the exchange is
  skipped (factory) or refused (callTool) — retained configs lose their
  privileges automatically.
- UI: the OBO option is hidden in the MCP server dialog for users
  without the permission; a CONFIGURE_OBO toggle is exposed in the MCP
  admin role editor.

Existing role docs receive the new sub-key via the permission backfill
in updateInterfacePermissions on next startup, preserving any
operator-set values. YAML/Config-sourced server configs are unaffected
since they're admin-controlled at the deployment level.

* 🧊 fix: wire OBO machinery for servers with requiresOAuth: false

The discovery and user-connection paths gated OAuth wiring (flow
manager, token methods, oboTokenResolver, oboTrustChecker) behind
isOAuthServer(), which only considers requiresOAuth/oauth fields.
A DB-stored OBO server with requiresOAuth: false therefore landed in
the non-OAuth branch, never received an oboTokenResolver, and the
factory's usesObo getter evaluated to false — sending a bare request
that the upstream rejected with invalid_token.

Add requiresOAuthMachinery() (OAuth OR OBO) and use it at those two
gates. isOAuthServer remains for the OAuth-handshake-only check
(shouldInitiateOAuthBeforeConnect), where OBO must not initiate a
handshake. Plumb the OBO resolver/trust-checker through
ToolDiscoveryOptions so reinitMCPServer can pass them on the
discovery path.

* 🧊 fix: lock all OBO-target fields (URL, proxy, headers, auth) without CONFIGURE_OBO

The CONFIGURE_OBO permission was meant to gate control of the endpoint
that receives OBO-minted per-user delegated tokens and the scopes that
are requested. The previous frontend lock + backend gate only covered
obo.scopes and the auth section, leaving url/proxy/headers/etc. editable
by anyone with UPDATE — meaning a non-permission user could still
redirect an existing OBO server's token flow to an attacker endpoint.

Switch to an allowlist policy: when editing an OBO server without
CONFIGURE_OBO, only title/description/iconPath are mutable. Backend
rejects any other field change with 403; frontend disables the
non-allowlist sections (URL, transport, auth, trust) via fieldset.
The comparison surface (MCP_USER_INPUT_FIELDS) is derived from
MCPServerUserInputSchema's union members so it stays in sync with the
schema. New schema fields land in the locked set by default — adding to
the allowlist is the only way to unlock them, which preserves the
security-review boundary.

* 🧊 fix: skip unauthenticated MCP inspection for OBO-only servers

MCPServerInspector.inspectServer() ran an unauthenticated temp connection
unless the config had requiresOAuth or customUserVars set. For OBO-only
servers without standard MCP OAuth advertisement, this caused
MCPConnectionFactory.create to attempt the connection without a user or
oboTokenResolver — failing on servers that reject the MCP initialize
handshake without a valid bearer token, which surfaced as
MCP_INSPECTION_FAILED on create/update.

Add `obo` to the skip list alongside requiresOAuth and customUserVars,
matching the existing pattern for user-scoped auth modes.

* Addressed linting error: watchedTitle is declared but never referenced (the auto-fill logic at line 156 uses getValues('title') instead). Deleted constant.
2026-06-01 22:36:18 -04:00
Danny Avila
100871c3ec
🛂 fix: Enforce MCP Permissions for Agent Tools (#13174)
Some checks are pending
Docker Dev Branch Images Build / build (Dockerfile, lc-dev, node) (push) Waiting to run
Docker Dev Branch Images Build / build (Dockerfile.multi, lc-dev-api, api-build) (push) Waiting to run
GitNexus Index / index (push) Waiting to run
GitNexus Index / post-index (push) Blocked by required conditions
* fix: Enforce MCP Permissions for Agent Tools

* fix: Measure MCP Image Limit by Decoded Size

* fix: gate cached MCP tools and tighten remote image URL detection

Addresses Codex review findings on the MCP permissions PR:

- filterAuthorizedTools previously fast-accepted any tool present in the
  global tool cache before reaching the MCP-use permission gate. App-level
  MCP tools (keyed `name_mcp_server` by MCPServerInspector and merged into
  the cache via mergeAppTools) therefore bypassed the canUseMCP check,
  letting a user without MCP_SERVERS.USE persist/bind them. Route all
  MCP-delimited tools through the permission + server-access gate
  regardless of cache presence.

- assertImageDataWithinLimit / image formatter used startsWith("http")
  to skip the size cap, which also matched base64 payloads that happen to
  begin with those chars. Require http:// or https:// via a shared
  isRemoteImageUrl helper so oversized inline base64 can no longer bypass
  MCP_IMAGE_DATA_MAX_BYTES.

Adds regression tests for both paths.

* fix: address Codex round-2 findings on MCP permissions PR

- parsers.ts: parseAsString dropped the image payload for unrecognized
  providers, returning only `Image result: <mimeType>`. Pre-PR these
  items survived via JSON.stringify(item). Keep the size guard but fall
  through to JSON.stringify so the data/URL is preserved.

- MCP.js: the runtime MCP-use check only read `configurable.user`, so
  paths that propagate `user_id` only (e.g. the OpenAI-compatible API in
  agents/openai/service.ts) rejected every MCP tool call for an
  authenticated user. Add resolveMCPPermissionUser: use the safe user
  directly when it already carries a role (no extra DB call), otherwise
  fall back to loading the role by user_id. Update fail-closed tests to
  the resolved behavior.

- v1.js: the update path only re-filtered newly added MCP tools, so a
  user who lost MCP_SERVERS.USE kept existing MCP bindings on edit while
  create/duplicate/revert stripped them. Strip all MCP tools on update
  when the permission is revoked; keep the narrower new-tool gating (and
  disconnect/registry preservation) when it is intact.

Updates and adds regression tests for all three paths.

* fix: populate safe user at producer instead of resolving in runtime MCP check

Corrects the Finding B approach from the previous commit. Rather than
loading the user by id inside the runtime MCP permission check, populate
`configurable.user` (and createRun's `user`) with the full safe user at
the producer, matching the in-repo agent controllers
(responses.js / openai.js) which already pass `createSafeUser(req.user)`.

- service.ts: derive `safeUser` via createSafeUser(req.user) and pass it
  to both createRun and processStream's configurable, so the role-bearing
  identity reaches the runtime `userCanUseMCPServers(configurable.user)`
  check. Falls back to a bare id when the host app attached no user,
  which correctly leaves MCP gated (fail closed).
- MCP.js: revert the resolveMCPPermissionUser DB-load fallback; the
  runtime check again reads configurable.user directly and fails closed
  when absent (defense in depth).
- MCP.spec.js: revert to the matching runtime test expectations.

* test: cover safe-user propagation in createAgentChatCompletion

Adds a focused spec for the OpenAI-compatible chat completion service
(the producer fixed for Codex Finding B). Injects mocked deps and asserts
that createRun and processStream's configurable.user carry the role from
req.user (with sensitive fields stripped by createSafeUser), and that an
unauthenticated request falls back to a bare { id: 'api-user' } so the
runtime MCP check fails closed.

* fix: address Codex round-3 findings + TS6133

- MCP.js (P1): the assistants required-action path invokes tool._call(
  toolInput) with no LangChain config, so the runtime check saw no
  configurable.user and rejected authorized users. createToolInstance now
  captures the creation-time user (req.user via createMCPTool) and _call
  falls back to it for both the permission check and userId. Still fails
  closed when neither config nor captured user carries a role.

- v1.js (P2): the update-path isMCPTool used a bare mcp_delimiter substring
  check, misclassifying action tools whose operationId contains "_mcp_"
  (e.g. sync_mcp_state_action_...) as MCP and dropping them on a
  permission-revoked edit. Delegate to the canonical isActionTool so only
  real MCP tools are gated. Regression test added.

- service.ts: drop the now-unused IUser import (TS6133); derive reqUser's
  type from createSafeUser's own parameter instead.

* fix: resolve TS7022 self-reference in service.spec mock res

The mock response object referenced `res` inside its own `status`/`json`
initializers without a type annotation, so tsc inferred `res` as `any`
(TS7022). Annotate the object and assign the self-referencing chainable
methods after declaration.

* fix: correct round-4 findings (isActionTool import, captured user, partial-update)

- v1.js: import isActionTool from librechat-data-provider (its real export;
  @librechat/api does not export it, so the prior import was undefined and
  threw TypeError). Exclude action tools from MCP classification in both the
  main filterAuthorizedTools loop and the update path, so action tools whose
  operationId contains _mcp_ (e.g. sync_mcp_state_action_...) are preserved
  regardless of MCP permission.
- v1.js: evaluate the effective tool set (updateData.tools ?? existingAgent.tools)
  so a tools-less PATCH by a user who lost MCP_SERVERS.USE still strips stale
  MCP bindings, matching create/duplicate/revert.
- MCP.js: createToolInstance now receives the construction-time user and _call
  falls back to it (permissionUser) when configurable.user is absent, fixing the
  assistants required-action path that invokes _call without a config and
  resolving the capturedUser no-undef/ReferenceError.
- Tests: action-tool preservation (authorized + denied), tools-less revocation
  PATCH, updated revocation test to expect all MCP tools stripped.

Affected specs pass locally: MCP 49/49, filterAuthorizedTools 49/49.

* fix: guard isActionTool against non-string tools; correct actionDelimiter import

Two test regressions from the prior commit:
- The main filterAuthorizedTools loop called isActionTool(tool) directly,
  but isActionTool does toolName.indexOf(...) and throws on null/undefined.
  Compute isActionToolName = typeof tool === 'string' && isActionTool(tool)
  once and reuse it, restoring graceful null/undefined handling.
- The action-tool test referenced Constants.actionDelimiter (undefined);
  actionDelimiter is a standalone librechat-data-provider export. Import and
  use it directly.

filterAuthorizedTools 36/36 and MCP 40/40 pass locally.

* fix: address MCP permission review follow-ups

* fix: preserve shared agent MCP tools
2026-05-30 16:19:49 -04:00
Danny Avila
bd64251eb9
🪪 fix: Prevent MCP Server Name Collisions (#13256)
* fix: prevent MCP server name collisions

* chore: address MCP registry review nits

* fix: reserve MCP config names from request context

* chore: format MCP registry changes

* chore: address MCP collision review findings
2026-05-22 20:46:14 -04:00
Danny Avila
4cce88be42
🪟 feat: Add allowedAddresses Exemption List For SSRF-Guarded Targets (#12933)
* 🪟 feat: Add allowedAddresses Exemption List For SSRF-Guarded Targets

LibreChat already blocks SSRF-prone targets (private IPs, loopback,
link-local, .internal/.local TLDs) at every server-side fetch site
that consumes user-controllable URLs — custom-endpoint baseURLs, MCP
servers, OpenAPI Actions, and OAuth endpoints. The only existing
escape hatch is `allowedDomains`, but that flips the field into a
strict whitelist: adding `127.0.0.1` to permit a self-hosted Ollama
also blocks every public destination that isn't in the list.

Introduce `allowedAddresses` as the orthogonal primitive: a private-
IP-space exemption list. When a hostname or its resolved IP appears
in the list, the SSRF block is bypassed for that target. Public
destinations remain reachable. Operators can now run self-hosted
LLMs / MCP servers / Action endpoints on private addresses without
weakening the default-deny posture for everything else.

Schema additions in `packages/data-provider/src/config.ts`:
- `endpoints.allowedAddresses` (new — gates `validateEndpointURL`)
- `mcpSettings.allowedAddresses` (parallel to `allowedDomains`)
- `actions.allowedAddresses` (parallel to `allowedDomains`)

Core changes in `packages/api/src/auth/`:
- New `isAddressAllowed(hostnameOrIP, allowedAddresses)` — pure,
  case-insensitive, bracket-stripped literal match.
- Threaded the list through `isSSRFTarget`, `resolveHostnameSSRF`,
  `isDomainAllowedCore`, `isActionDomainAllowed`, `isMCPDomainAllowed`,
  `isOAuthUrlAllowed`, and `validateEndpointURL`.
- Extended `createSSRFSafeAgents` and `createSSRFSafeUndiciConnect`
  to accept the list, building an SSRF-safe DNS lookup that exempts
  matching hostnames/IPs at TCP connect time (TOCTOU-safe).

Wiring:
- Custom and OpenAI endpoint initialize sites pass
  `endpoints.allowedAddresses` to `validateEndpointURL`.
- `MCPServersRegistry` stores `allowedAddresses` and exposes it via
  `getAllowedAddresses()`. The factory, connection class, manager,
  `UserConnectionManager`, and `ConnectionsRepository` all thread
  it through to the SSRF utilities.
- `MCPOAuthHandler.initiateOAuthFlow`, `refreshOAuthTokens`, and
  `validateOAuthUrl` accept the list and consult it on every URL
  validation along the OAuth chain.
- `ToolService`, `ActionService`, and the assistants/agents action
  routes pass `actions.allowedAddresses` to `isActionDomainAllowed`
  and to `createSSRFSafeAgents` for runtime action calls.
- `initializeMCPs.js` reads `mcpSettings.allowedAddresses` from the
  app config and forwards it to the registry constructor.

Documentation:
- `librechat.example.yaml` shows the new field next to each existing
  `allowedDomains` block, with a note clarifying that
  `allowedAddresses` is an exemption list (not a whitelist).

Tests:
- Unit tests for `isAddressAllowed` covering literal IPs, hostnames,
  IPv6 brackets, case insensitivity, and partial-match rejection.
- Exemption tests for every entry point: `isSSRFTarget`,
  `resolveHostnameSSRF`, `validateEndpointURL`, `isActionDomainAllowed`,
  `isMCPDomainAllowed`, `isOAuthUrlAllowed`.
- Existing tests updated to reflect the new optional parameter.

Default behavior is unchanged: omitted = empty list = no exemptions.

* 🩹 fix: Plumb allowedAddresses Through AppConfig endpoints Type

The initial PR added `endpoints.allowedAddresses` to the
data-provider config schema and consumed it in the endpoint
initialize sites, but the runtime `AppConfig.endpoints` shape in
`@librechat/data-schemas` was a hand-maintained subset that didn't
include the new field — so `tsc` rejected `appConfig.endpoints.allowedAddresses`.

Add the field to `AppConfig['endpoints']` in
`packages/data-schemas/src/types/app.ts` and forward it from the
loaded config in `packages/data-schemas/src/app/endpoints.ts` so the
runtime config carries the value.

Update `initializeMCPs.spec.js` to expect the third positional
argument (`allowedAddresses`) on the `createMCPServersRegistry` call.

* 🩹 fix: Enforce allowedDomains Before allowedAddresses In isOAuthUrlAllowed

The initial implementation checked the address exemption first, so a
URL whose hostname appeared in `allowedAddresses` would return true
even when the admin had configured `allowedDomains` as a strict bound
on OAuth endpoints. A malicious MCP server could advertise OAuth
metadata, token, or revocation URLs at any address the admin had
permitted for an unrelated reason (a self-hosted LLM at `127.0.0.1`,
for example) and pass validation, expanding SSRF reach beyond the
configured domain whitelist.

Reorder: when `allowedDomains` is set, treat it as authoritative —
return true only if the URL matches a domain entry, otherwise fall
through to false. The address exemption only applies when no
`allowedDomains` is configured (mirrors how the downstream SSRF check
in `validateOAuthUrl` consults `allowedAddresses`).

Add a regression test asserting that an `allowedAddresses` entry does
not broaden a configured `allowedDomains` list.

Reported by chatgpt-codex-connector on PR #12933.

* 🩹 fix: Forward allowedAddresses To Remaining OAuth Callers

Two `MCPOAuthHandler` callers still used the pre-feature signatures and
were silently dropping the new `allowedAddresses` argument:

- `api/server/routes/mcp.js` invoked `initiateOAuthFlow` with the old
  5-argument shape, so OAuth flows initiated through the route handler
  ignored the registry's `getAllowedAddresses()` and would reject any
  metadata/authorization/token URL on a permitted private host.
- `api/server/controllers/UserController.js#maybeUninstallOAuthMCP`
  invoked `revokeOAuthToken` without the address exemption, so
  uninstalling an OAuth-backed MCP server on a permitted private host
  would fail at the revocation step even though the rest of the MCP
  connection path now permits it.

Both sites now read `allowedAddresses` from the registry alongside
`allowedDomains` and forward it. Reported by Copilot on PR #12933.

* 🩹 fix: Update Test Mocks And Assertions For OAuth allowedAddresses

The previous commit started passing `allowedAddresses` to
`MCPOAuthHandler.initiateOAuthFlow` from `api/server/routes/mcp.js`
and to `MCPOAuthHandler.revokeOAuthToken` from
`api/server/controllers/UserController.js`, but the corresponding
test files mocked the registry without `getAllowedAddresses` (causing
`TypeError`s) and asserted the old positional shape on
`toHaveBeenCalledWith`.

Update the mocks and assertions to match the new arity:

- `api/server/routes/__tests__/mcp.spec.js`: add
  `getAllowedDomains`/`getAllowedAddresses` to the registry mock and
  expect the additional positional args on `initiateOAuthFlow`.
- `api/server/controllers/__tests__/maybeUninstallOAuthMCP.spec.js`:
  add a `getAllowedAddresses` mock alongside the existing
  `getAllowedDomains` and seed it in `setupOAuthServerFound`.
- `api/server/controllers/__tests__/UserController.mcpOAuth.spec.js`:
  add `getAllowedAddresses` to the registry mock and expect the
  trailing `null` arg on the three `revokeOAuthToken` assertions.

* 🛡️ fix: Address Comprehensive Review — Scope allowedAddresses To Private IP Space

Major findings from the comprehensive PR review (severity → fix):

**CRITICAL — `validateOAuthUrl` SSRF fallback bypass.** When `allowedDomains`
is configured and a URL fails the whitelist, the SSRF fallback in
`validateOAuthUrl` was still passing `allowedAddresses` to `isSSRFTarget` /
`resolveHostnameSSRF`, letting a malicious MCP server advertise OAuth
endpoints at any address the admin had permitted for an unrelated reason.
Suppress `allowedAddresses` in the fallback when `allowedDomains` is active —
the address exemption is opt-in for the no-whitelist mode only.

**MAJOR — WebSocket transport SSRF check ignored exemptions.** The
`constructTransport` WebSocket branch called `resolveHostnameSSRF(wsHostname)`
without `this.allowedAddresses`, so a permitted private MCP server would
pass `isMCPDomainAllowed` but be blocked at transport creation. Forward
the exemption.

**Scope `allowedAddresses` to private IP space only (operator directive).**
The exemption list is for permitting private/internal targets; it must not
be a back-door to broaden trust to public destinations.
- Schema (`packages/data-provider/src/config.ts`): new
  `allowedAddressesSchema` rejects URLs (`://`), paths/CIDR (`/`),
  whitespace, and public IPv4/IPv6 literals at config-load time. Wired
  into `endpoints`, `mcpSettings`, and `actions`.
- Runtime (`packages/api/src/auth/domain.ts`): `isAddressAllowed` now
  drops public-IP candidates and public-IP entries on the match path —
  defense in depth so a misconfigured runtime list never grants exemption.
- Hot path (`packages/api/src/auth/agent.ts`): `buildSSRFSafeLookup`
  pre-normalizes the list into a `Set<string>` once at construction and
  applies the same scoping filter, so the connect-time DNS lookup is an
  O(1) Set membership check instead of a full re-iterate-and-normalize on
  every outbound request.

**Test coverage for the connect-time and OAuth-fallback paths.**
- `agent.spec.ts`: new describe block exercising `buildSSRFSafeLookup` and
  `createSSRFSafe*` with `allowedAddresses` — hostname-literal exemption,
  resolved-IP exemption, public-IP scoping, URL/CIDR/whitespace rejection,
  and the default no-list block.
- `handler.allowedAddresses.test.ts` (new): integration tests for
  `validateOAuthUrl` — covers both the no-domains-set "permit private"
  path and the strict-bound regression where `allowedAddresses` must NOT
  bypass `allowedDomains`.

**Documentation & cleanup.**
- `connection.ts` redirect SSRF check: explicit comment that
  `allowedAddresses` is intentionally NOT consulted for redirect targets
  (server-controlled, must not inherit the admin's exemption).
- `MCPConnectionFactory.test.ts`: replaced an `eslint-disable` with a
  proper `import { getTenantId } from '@librechat/data-schemas'`. The
  disable was added to make a pre-existing `require()` quiet — the cleaner
  fix is to use the existing top-level import.

Updated `MCPConnectionSSRF.test.ts` WebSocket SSRF assertions to match the
new two-argument call shape (`hostname, allowedAddresses`).

* 🩹 fix: Require Absolute URL Before allowedAddresses Trust Bypass In isOAuthUrlAllowed

`parseDomainSpec` is lenient — it silently prepends `https://` to
schemeless inputs so it can match patterns like bare `example.com`.
That leniency leaked into `isOAuthUrlAllowed`'s new
`allowedAddresses` short-circuit: a value like `10.0.0.5/oauth` (no
scheme) would parse successfully via the prepended default, hit the
address-exemption path, return `true`, and skip `validateOAuthUrl`'s
strict `new URL(url)` parse-or-throw — only to fail later in OAuth
discovery with a less clear runtime error.

Add a strict `new URL(url)` gate at the top of `isOAuthUrlAllowed`.
Schemeless inputs now fall through to `validateOAuthUrl`'s explicit
"Invalid OAuth <field>" rejection. Tests added in both
`auth/domain.spec.ts` (unit) and the OAuth handler integration spec
(end-to-end).

Reported by chatgpt-codex-connector (P2) on PR #12933.

* 🛡️ fix: Address Follow-Up Comprehensive Review — Schema Tests, Shared Normalization, host:port

Auditing the second comprehensive review:

**F1 MAJOR — schema validation untested.** `allowedAddressesSchema` had
zero coverage, so a regression in the three refinement stages or the
three wiring locations (`endpoints` / `mcpSettings` / `actions`) would
silently let invalid entries reach the runtime. Added a dedicated
`describe('allowedAddressesSchema')` block in `config.spec.ts` covering:
valid private IPs (v4 + v6, including the previously-missed 192.0.0.0/24
range), accepted hostnames, all rejection categories (URLs, CIDR, paths,
whitespace tabs/newlines, host:port, public IP literals), and full
`configSchema.parse()` integration at each of the three nesting points.

**F2 MINOR — `isPrivateIPv4Literal` divergence.** The schema reimpl in
`packages/data-provider` was discarding the `c` octet, so the
`192.0.0.0/24` (RFC 5736 IETF protocol assignments) range that the
authoritative `isPrivateIPv4` accepts was being rejected with a
misleading "public IP" error. Destructure `c` and add the missing range
check; covered by the new schema tests.

**F3 MINOR — DRY violation across `domain.ts` and `agent.ts`.** Both
files had independent normalization implementations with a subtle
whitespace-check divergence (`/\s/` vs `.includes(' ')`). Extracted the
shared logic into a new `packages/api/src/auth/allowedAddresses.ts`
module that both consumers import:
  - `normalizeAddressEntry(entry)` — single-entry shape check
  - `looksLikeHostPort(entry)` — host:port detector (used by F4)
  - `normalizeAllowedAddressesSet(list)` — pre-normalized Set for the
    connect-time hot path
  - `isAddressInAllowedSet(candidate, set)` — membership check that
    enforces private-IP scoping on the candidate

Both `isAddressAllowed` (preflight) and `buildSSRFSafeLookup` (connect)
now go through the same primitives; the whitespace divergence is gone.

To break the import cycle (`allowedAddresses` needs `isPrivateIP`,
`domain` previously owned it), extracted IP private-range detection
into a leaf `auth/ip.ts` module. `domain.ts` re-exports `isPrivateIP`
for backward compatibility with existing call sites.

**F4 MINOR — `host:port` silently misclassified.** Entries like
`localhost:8080` previously slipped through the URL/path guard, were
mis-detected as IPv6, failed `isPrivateIP`, and were silently dropped
with a misleading "public IP" schema error. Added an explicit
`looksLikeHostPort` check with a clear error: "allowedAddresses
entries must not include a port — list the bare hostname or IP only."
Bare `::1`, `[::1]`, and other valid IPv6 literals are intentionally
not matched (regex distinguishes by colon count and the bracketed
`[ipv6]:port` form).

**F5 MINOR — hostname-trust documentation gap.** Hostname entries
short-circuit `resolveHostnameSSRF` before any DNS lookup — that's a
deliberate design (admin trusts the name) but it means the exemption
follows whatever the name resolves to at runtime. Added an explicit
note in `librechat.example.yaml` for both `mcpSettings.allowedAddresses`
and `endpoints.allowedAddresses`: "a hostname entry trusts whatever IP
that name resolves to. Only list hostnames whose DNS you control.
Prefer literal IPs when you can."

**F6** (8 positional params) is flagged for follow-up; refactor to an
options object is a breaking-API change deferred to a separate PR.
**F7** (redirect/WebSocket asymmetry, NIT, conf 40) — skipping; the
existing inline comment is sufficient.

* 🧹 chore: Address Follow-Up NITs — Import Order And Mirror-Function Naming

Three NITs from the latest comprehensive review:

**NIT #1 (conf 85) — local import order.** AGENTS.md requires local
imports sorted longest-to-shortest. Both `domain.ts` and `agent.ts`
had `./ip` (shorter) before `./allowedAddresses` (longer). Swapped.

**NIT #2 (conf 60) — missing cross-reference.** The schema-side
`isHostPortShape` in `packages/data-provider/src/config.ts` had no
note pointing at the canonical runtime mirror. Added a JSDoc paragraph
explaining the mirror relationship and why a local copy exists (the
data-provider package can't import from `@librechat/api` without
creating a circular dependency).

**NIT #3 (conf 50) — naming inconsistency.** Renamed
`isHostPortShape` → `looksLikeHostPort` so the schema mirror matches
the runtime helper exactly. Kept as a separate function (not a shared
import) for the same circular-dependency reason; the matching name
makes it obvious they should stay in lockstep.
2026-05-03 21:43:59 -04:00
Danny Avila
5b5e2b0286
🛡️ fix: Handle MCP Tool Cache Lookup Failures (#12910)
* Handle MCP tool cache lookup failures

* Harden MCP cached tool lookup

* Cover full MCP tool cache outage

* Guard MCP tool cache store lookup
2026-05-02 09:21:28 +09:00
Dustin Healy
fc3189b718
🔐 fix: Restore Tenant Context in MCP OAuth Callback (#12782)
* fix: restore tenant context in MCP OAuth callback for multi-tenant deployments

The MCP OAuth callback is a cross-origin redirect from the OAuth
provider. SameSite=Strict cookies (including the JWT) are not sent,
leaving the callback with no tenant context. With
TENANT_ISOLATION_STRICT=true, all DB writes fail.

Stores tenantId in flow metadata at OAuth initiation time (when
the user is authenticated), then restores it via tenantStorage.run
in the callback, wrapping the entire post-validation body.

* test: address review findings for tenant context tests

- Assert tenantId flows through to initFlow in MCPConnectionFactory test
- Add beforeEach to tenant context tests to reset mocks independently
2026-04-22 14:05:51 -07:00
Denis Palnitsky
8ed0bcf5ca
♻️ fix: Reuse Existing MCP OAuth Client Registrations to Prevent client_id Mismatch (#11925)
* fix: reuse existing OAuth client registrations to prevent client_id mismatch

When using auto-discovered OAuth (DCR), LibreChat calls /register on every
flow initiation, getting a new client_id each time. When concurrent
connections or reconnections happen, the client_id used during /authorize
differs from the one used during /token, causing the server to reject the
exchange.

Before registering a new client, check if a valid client registration
already exists in the database and reuse it.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* Handle re-registration of OAuth clients when redirect_uri changes

* Add undefined fields for logo_uri and tos_uri in OAuth metadata tests

* test: add client registration reuse tests for horizontal scaling race condition

Reproduces the client_id mismatch bug that occurs in multi-replica deployments
where concurrent initiateOAuthFlow calls each register a new OAuth client.
Tests verify that the findToken-based client reuse prevents re-registration.

* fix: address review findings for client registration reuse

- Fix empty redirect_uris bug: invert condition so missing/empty
  redirect_uris triggers re-registration instead of silent reuse
- Revert undocumented config?.redirect_uri in auto-discovery path
- Change DB error logging from debug to warn for operator visibility
- Fix import order: move package type import to correct section
- Remove redundant type cast and misleading JSDoc comment
- Test file: remove dead imports, restore process.env.DOMAIN_SERVER,
  rename describe blocks, add empty redirect_uris edge case test,
  add concurrent reconnection test with pre-seeded token,
  scope documentation to reconnection stabilization

* fix: resolve type check errors for OAuthClientInformation redirect_uris

The SDK's OAuthClientInformation type lacks redirect_uris (only on
OAuthClientInformationFull). Cast to the local OAuthClientInformation
type in handler.ts when accessing deserialized client info from DB,
and use intersection types in tests for clientInfo with redirect_uris.

* fix: address follow-up review findings R1, R2, R3

- R1: Move `import type { TokenMethods }` to the type-imports section,
  before local types, per CLAUDE.md import order rules
- R2: Add unit test for empty redirect_uris in handler.test.ts to
  verify the inverted condition triggers re-registration
- R3: Use delete for process.env.DOMAIN_SERVER restoration when the
  original value was undefined to avoid coercion to string "undefined"

* fix: clear stale client registration on OAuth flow failure

When a stored client_id is no longer recognized by the OAuth server,
the flow fails but the stale client stays in MongoDB, causing every
retry to reuse the same invalid registration in an infinite loop.

On OAuth failure, clear the stored client registration so the next
attempt falls through to fresh Dynamic Client Registration.

- Add MCPTokenStorage.deleteClientRegistration() for targeted cleanup
- Call it from MCPConnectionFactory's OAuth failure path
- Add integration test proving recovery from stale client reuse

* fix: validate auth server identity and target cleanup to reused clients

- Gate client reuse on authorization server identity: compare stored
  issuer against freshly discovered metadata before reusing, preventing
  wrong-client reuse when the MCP server switches auth providers
- Add reusedStoredClient flag to MCPOAuthFlowMetadata so cleanup only
  runs when the failed flow actually reused a stored registration,
  not on unrelated failures (timeouts, user-denied consent, etc.)
- Add cleanup in returnOnOAuth path: when a prior flow that reused a
  stored client is detected as failed, clear the stale registration
  before re-initiating
- Add tests for issuer mismatch and reusedStoredClient flag assertions

* fix: address minor review findings N3, N5, N6

- N3: Type deleteClientRegistration param as TokenMethods['deleteTokens']
  instead of Promise<unknown>
- N5: Elevate deletion failure logging from debug to warn for operator
  visibility when stale client cleanup fails
- N6: Use getLogPrefix() instead of hardcoded log prefix to respect
  system-user privacy convention

* fix: correct stale-client cleanup in both OAuth paths

- Blocking path: remove result?.clientInfo guard that made cleanup
  unreachable (handleOAuthRequired returns null on failure, so
  result?.clientInfo was always false in the failure branch)
- returnOnOAuth path: only clear stored client when the prior flow
  status is FAILED, not on COMPLETED or PENDING flows, to avoid
  deleting valid registrations during normal flow replacement

* fix: remove redundant cast on clientMetadata

clientMetadata is already typed as Record<string, unknown>; the
as Record<string, unknown> cast was a no-op.

* fix: thread reusedStoredClient through return type instead of re-reading flow state

FlowStateManager.createFlow() deletes FAILED flow state before
rejecting, so getFlowState() after handleOAuthRequired() returns null
would find nothing — making the stale-client cleanup dead code.

Fix: hoist reusedStoredClient flag from flowMetadata into a local
variable, include it in handleOAuthRequired()'s return type (both
success and catch paths), and use result.reusedStoredClient directly
in the caller instead of a second getFlowState() round-trip.

* fix: selective stale-client cleanup in returnOnOAuth path

The returnOnOAuth cleanup was unreliable: it depended on reading
FAILED flow state, but FlowStateManager.monitorFlow() deletes FAILED
state before rejecting. Move cleanup into createFlow's catch handler
where flowMetadata.reusedStoredClient is still in scope.

Make cleanup selective in both paths: add isClientRejection() helper
that only matches errors indicating the OAuth server rejected the
client_id (invalid_client, unauthorized_client, client not found).
Timeouts, user-cancelled flows, and other transient failures no
longer wipe valid stored registrations.

Thread the error from handleOAuthRequired() through the return type
so the blocking path can also check isClientRejection().

* fix: tighten isClientRejection heuristic

Narrow 'client_id' match to 'client_id mismatch' to avoid
false-positive cleanup on unrelated errors that happen to
mention client_id.

* test: add isClientRejection tests and enforced client_id on test server

- Add isClientRejection unit tests: invalid_client, unauthorized_client,
  client_id mismatch, client not found, unknown client, and negative
  cases (timeout, flow state not found, user denied, null, undefined)
- Enhance OAuth test server with enforceClientId option: binds auth
  codes to the client_id that initiated /authorize, rejects token
  exchange with mismatched or unregistered client_id (401 invalid_client)
- Add integration tests proving the test server correctly rejects
  stale client_ids and accepts matching ones at /token

* fix: issuer validation, callback error propagation, and cleanup DRY

- Issuer check: re-register when storedIssuer is absent or non-string
  instead of silently reusing. Narrows unknown type with typeof guard
  and inverts condition so missing issuer → fresh DCR (safer default).
- OAuth callback route: call failFlow with the OAuth error when the
  authorization server redirects back with error= parameter, so the
  waiting flow receives the actual rejection instead of timing out.
  This lets isClientRejection match stale-client errors correctly.
- Extract duplicated cleanup block to clearStaleClientIfRejected()
  private method, called from both returnOnOAuth and blocking paths.
- Test fixes: add issuer to stored metadata in reuse tests, reset
  server to undefined in afterEach to prevent double-close.

* fix: gate failFlow behind callback validation, propagate reusedStoredClient on join

- OAuth callback: move failFlow call to after CSRF/session/active-flow
  validation so an attacker with only a leaked state parameter cannot
  force-fail a flow without passing the same integrity checks required
  for legitimate callbacks
- PENDING join path: propagate reusedStoredClient from flow metadata
  into the return object so joiners can trigger stale-client cleanup
  if the joined flow later fails with a client rejection

* fix: restore early oauthError/code redirects, gate only failFlow behind CSRF

The previous restructuring moved oauthError and missing-code checks
behind CSRF validation, breaking tests that expect those redirects
without cookies. The redirect itself is harmless (just shows an error
page). Only the failFlow call needs CSRF gating to prevent DoS.

Restructure: oauthError check stays early (redirects immediately),
but failFlow inside it runs the full CSRF/session/active-flow
validation before marking the flow as FAILED.

* fix: require deleteTokens for client reuse, add missing import in MCP.js

Client registration reuse without cleanup capability creates a
permanent failure loop: if the reused client is stale, the code
detects the rejection but cannot clear the stored registration
because deleteTokens is missing, so every retry reuses the same
broken client_id.

- MCPConnectionFactory: only pass findToken to initiateOAuthFlow
  when deleteTokens is also available, ensuring reuse is only
  enabled when recovery is possible
- api/server/services/MCP.js: add deleteTokens to the tokenMethods
  object (was the only MCP call site missing it)

* fix: set reusedStoredClient before createFlow in joined-flow path

When joining a PENDING flow, reusedStoredClient was only set on the
success return but not before the await. If createFlow throws (e.g.
invalid_client during token exchange), the outer catch returns the
local variable which was still false, skipping stale-client cleanup.

* fix: require browser binding (CSRF/session) for failFlow on OAuth error

hasActiveFlow only proves a PENDING flow exists, not that the caller
is the same browser that initiated it. An attacker with a leaked state
could force-fail the flow without any user binding. Require hasCsrf or
hasSession before calling failFlow on the oauthError path.

* fix: guard findToken with deleteTokens check in blocking OAuth path

Match the returnOnOAuth path's defense-in-depth: only enable client
registration reuse when deleteTokens is also available, ensuring
cleanup is possible if the reused client turns out to be stale.

* fix: address review findings — tests, types, normalization, docs

- Add deleteTokens method to InMemoryTokenStore matching TokenMethods
  contract; update test call site from deleteToken to deleteTokens
- Add MCPConnectionFactory test: returnOnOAuth flow fails with
  invalid_client → clearStaleClientIfRejected invoked automatically
- Add mcp.spec.js tests: OAuth error with CSRF → failFlow called;
  OAuth error without cookies → failFlow NOT called (DoS prevention)
- Add JSDoc to isClientRejection with RFC 6749 and vendor attribution
- Add inline comment explaining findToken/deleteTokens coupling guard
- Normalize issuer comparison: strip trailing slashes to prevent
  spurious re-registrations from URL formatting differences
- Fix dead-code: use local reusedStoredClient variable in PENDING
  join return instead of re-reading flowMeta

* fix: address final review nits N1-N4

- N1: Add session cookie failFlow test — validates the hasSession
  branch triggers failFlow on OAuth error callback
- N2: Replace setTimeout(50) with setImmediate for microtask drain
- N3: Add 'unknown client' attribution to isClientRejection JSDoc
- N4: Remove dead getFlowState mock from failFlow tests

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: Danny Avila <danny@librechat.ai>
2026-04-03 22:15:08 -04:00
Danny Avila
935288f841
🏗️ feat: 3-Tier MCP Server Architecture with Config-Source Lazy Init (#12435)
* feat: add MCPServerSource type, tenantMcpPolicy schema, and source-based dbSourced wiring

- Add `tenantMcpPolicy` to `mcpSettings` in YAML config schema with
  `enabled`, `maxServersPerTenant`, `allowedTransports`, and `allowedDomains`
- Add `MCPServerSource` type ('yaml' | 'config' | 'user') and `source`
  field to `ParsedServerConfig`
- Change `dbSourced` determination from `!!config.dbId` to
  `config.source === 'user'` across MCPManager, ConnectionsRepository,
  UserConnectionManager, and MCPServerInspector
- Set `source: 'user'` on all DB-sourced servers in ServerConfigsDB

* feat: three-layer MCPServersRegistry with config cache and lazy init

- Add `configCacheRepo` as third repository layer between YAML cache and
  DB for admin-defined config-source MCP servers
- Implement `ensureConfigServers()` that identifies config-override servers
  from resolved `getAppConfig()` mcpConfig, lazily inspects them, and
  caches parsed configs with `source: 'config'`
- Add `lazyInitConfigServer()` with timeout, stub-on-failure, and
  concurrent-init deduplication via `pendingConfigInits` map
- Extend `getAllServerConfigs()` with optional `configServers` param for
  three-way merge: YAML → Config → User
- Add `getServerConfig()` lookup through config cache layer
- Add `invalidateConfigCache()` for clearing config-source inspection
  results on admin config mutations
- Tag `source: 'yaml'` on CACHE-stored servers and `source: 'user'` on
  DB-stored servers in `addServer()` and `addServerStub()`

* feat: wire tenant context into MCP controllers, services, and cache invalidation

- Resolve config-source servers via `getAppConfig({ role, tenantId })`
  in `getMCPTools()` and `getMCPServersList()` controllers
- Pass `ensureConfigServers()` results through `getAllServerConfigs()`
  for three-way merge of YAML + Config + User servers
- Add tenant/role context to `getMCPSetupData()` and connection status
  routes via `getTenantId()` from ALS
- Add `clearMcpConfigCache()` to `invalidateConfigCaches()` so admin
  config mutations trigger re-inspection of config-source MCP servers

* feat: enforce tenantMcpPolicy on admin config mcpServers mutations

- Add `validateMcpServerPolicy()` helper that checks mcpServers against
  operator-defined `tenantMcpPolicy` (enabled, maxServersPerTenant,
  allowedTransports, allowedDomains)
- Wire validation into `upsertConfigOverrides` and `patchConfigField`
  handlers — rejects with 403 when policy is violated
- Infer transport type from config shape (command → stdio, url protocol
  → websocket/sse, type field → streamable-http)
- Validate server domains against policy allowlist when configured

* revert: remove tenantMcpPolicy schema and enforcement

The existing admin config CRUD routes already provide the mechanism
for granular MCP server prepopulation (groups, roles, users). The
tenantMcpPolicy gating adds unnecessary complexity that can be
revisited if needed in the future.

- Remove tenantMcpPolicy from mcpSettings Zod schema
- Remove validateMcpServerPolicy helper and TenantMcpPolicy interface
- Remove policy enforcement from upsertConfigOverrides and
  patchConfigField handlers

* test: update test assertions for source field and config-server wiring

- Use objectContaining in MCPServersRegistry reset test to account for
  new source: 'yaml' field on CACHE-stored configs
- Add getTenantId and ensureConfigServers mocks to MCP route tests
- Add getAppConfig mock to route test Config service mock
- Update getMCPSetupData assertion to expect second options argument
- Update getAllServerConfigs assertions for new configServers parameter

* fix: disconnect active connections when config-source servers are evicted

When admin config overrides change and config-source MCP servers are
removed, the invalidation now proactively disconnects active connections
for evicted servers instead of leaving them lingering until timeout.

- Return evicted server names from invalidateConfigCache()
- Disconnect app-level connections for evicted servers in
  clearMcpConfigCache() via MCPManager.appConnections.disconnect()

* fix: address code review findings (CRITICAL, MAJOR, MINOR)

CRITICAL fixes:
- Scope configCacheRepo keys by config content hash to prevent
  cross-tenant cache poisoning when two tenants define the same
  server name with different configurations
- Change dbSourced checks from `source === 'user'` to
  `source !== 'yaml' && source !== 'config'` so undefined source
  (pre-upgrade cached configs) fails closed to restricted mode

MAJOR fixes:
- Derive OAuth servers from already-computed mcpConfig instead of
  calling getOAuthServers() separately — config-source OAuth servers
  are now properly detected
- Add parseInt radix (10) and NaN guard with fallback to 30_000
  for CONFIG_SERVER_INIT_TIMEOUT_MS
- Add CONFIG_CACHE_NAMESPACE to aggregate-key branch in
  ServerConfigsCacheFactory to avoid SCAN-based Redis stalls
- Remove `if (role || tenantId)` guard in getMCPSetupData — config
  servers now always resolve regardless of tenant context

MINOR fixes:
- Extract resolveAllMcpConfigs() helper in mcp controller to
  eliminate 3x copy-pasted config resolution boilerplate
- Distinguish "not initialized" from real errors in
  clearMcpConfigCache — log actual failures instead of swallowing
- Remove narrative inline comments per style guide
- Remove dead try/catch inside Promise.allSettled in
  ensureConfigServers (inner method never throws)
- Memoize YAML server names to avoid repeated cacheConfigsRepo.getAll()
  calls per request

Test updates:
- Add ensureConfigServers mock to registry test fixtures
- Update getMCPSetupData assertions for inline OAuth derivation

* fix: address code review findings (CRITICAL, MAJOR, MINOR)

CRITICAL fixes:
- Break circular dependency: move CONFIG_CACHE_NAMESPACE from
  MCPServersRegistry to ServerConfigsCacheFactory
- Fix dbSourced fail-closed: use source field when present, fall back to
  legacy dbId check when absent (backward-compatible with pre-upgrade
  cached configs that lack source field)

MAJOR fixes:
- Add CONFIG_CACHE_NAMESPACE to aggregate-key set in
  ServerConfigsCacheFactory to avoid SCAN-based Redis stalls
- Add comprehensive test suite (ensureConfigServers.test.ts, 18 tests)
  covering lazy init, stub-on-failure, cross-tenant isolation via config
  hash keys, concurrent deduplication, merge order, and cache invalidation

MINOR fixes:
- Update MCPServerInspector test assertion for dbSourced change

* fix: restore getServerConfig lookup for config-source servers (NEW-1)

Add configNameToKey map that indexes server name → hash-based cache key
for O(1) lookup by name in getServerConfig. This restores the config
cache layer that was dropped when hash-based keys were introduced.

Without this fix, config-source servers appeared in tool listings
(via getAllServerConfigs) but getServerConfig returned undefined,
breaking all connection and tool call paths.

- Populate configNameToKey in ensureSingleConfigServer
- Clear configNameToKey in invalidateConfigCache and reset
- Clear stale read-through cache entries after lazy init
- Remove dead code in invalidateConfigCache (config.title, key parsing)
- Add getServerConfig tests for config-source server lookup

* fix: eliminate configNameToKey race via caller-provided configServers param

Replace the process-global configNameToKey map (last-writer-wins under
concurrent multi-tenant load) with a configServers parameter on
getServerConfig. Callers pass the pre-resolved config servers map
directly — no shared mutable state, no cross-tenant race.

- Add optional configServers param to getServerConfig; when provided,
  returns matching config directly without any global lookup
- Remove configNameToKey map entirely (was the source of the race)
- Extract server names from cache keys via lastIndexOf in
  invalidateConfigCache (safe for names containing colons)
- Use mcpConfig[serverName] directly in getMCPTools instead of a
  redundant getServerConfig call
- Add cross-tenant isolation test for getServerConfig

* fix: populate read-through cache after config server lazy init

After lazyInitConfigServer succeeds, write the parsed config to
readThroughCache keyed by serverName so that getServerConfig calls
from ConnectionsRepository, UserConnectionManager, and
MCPManager.callTool find the config without needing configServers.

Without this, config-source servers appeared in tool listings but
every connection attempt and tool call returned undefined.

* fix: user-scoped getServerConfig fallback to server-only cache key

When getServerConfig is called with a userId (e.g., from callTool or
UserConnectionManager), the cache key is serverName::userId. Config-source
servers are cached under the server-only key (no userId). Add a fallback
so user-scoped lookups find config-source servers in the read-through cache.

* fix: configCacheRepo fallback, isUserSourced DRY, cross-process race

CRITICAL: Add findInConfigCache fallback in getServerConfig so
config-source servers remain reachable after readThroughCache TTL
expires (5s). Without this, every tool call after 5s returned
undefined for config-source servers.

MAJOR: Extract isUserSourced() helper to mcp/utils.ts and replace
all 5 inline dbSourced ternary expressions (MCPManager x2,
ConnectionsRepository, UserConnectionManager, MCPServerInspector).

MAJOR: Fix cross-process Redis race in lazyInitConfigServer — when
configCacheRepo.add throws (key exists from another process), fall
back to reading the existing entry instead of returning undefined.

MINOR: Parallelize invalidateConfigCache awaits with Promise.all.
Remove redundant .catch(() => {}) inside Promise.allSettled.
Tighten dedup test assertion to toBe(1).
Add TTL-expiry tests for getServerConfig (with and without userId).

* feat: thread configServers through getAppToolFunctions and formatInstructionsForContext

Add optional configServers parameter to getAppToolFunctions,
getInstructions, and formatInstructionsForContext so config-source
server tools and instructions are visible to agent initialization
and context injection paths.

Existing callers (boot-time init, tests) pass no argument and
continue to work unchanged. Agent runtime paths can now thread
resolved config servers from request context.

* fix: stale failure stubs retry after 5 min, upsert for cross-process races

- Add CONFIG_STUB_RETRY_MS (5 min) — stale failure stubs are retried
  instead of permanently disabling config-source servers after transient
  errors (DNS outage, cold-start race)
- Extract upsertConfigCache() helper that tries add then falls back to
  update, preventing cross-process Redis races where a second instance's
  successful inspection result was discarded
- Add test for stale-stub retry after CONFIG_STUB_RETRY_MS

* fix: stamp updatedAt on failure stubs, null-guard callTool config, test cleanup

- Add updatedAt: Date.now() to failure stubs in lazyInitConfigServer so
  CONFIG_STUB_RETRY_MS (5 min) window works correctly — without it, stubs
  were always considered stale (updatedAt ?? 0 → epoch → always expired)
- Add null guard for rawConfig in MCPManager.callTool before passing to
  preProcessGraphTokens — prevents unsafe `as` cast on undefined
- Log double-failure in upsertConfigCache instead of silently swallowing
- Replace module-scope Date.now monkey-patch with jest.useFakeTimers /
  jest.setSystemTime / jest.useRealTimers in ensureConfigServers tests

* fix: server-only readThrough fallback only returns truthy values

Prevents a cached undefined from a prior no-userId lookup from
short-circuiting the DB query on a subsequent userId-scoped lookup.

* fix: remove findInConfigCache to eliminate cross-tenant config leakage

The findInConfigCache prefix scan (serverName:*) could return any
tenant's config after readThrough TTL expires, violating tenant
isolation. Config-source servers are now ONLY resolvable through:

1. The configServers param (callers with tenant context from ALS)
2. The readThrough cache (populated by ensureSingleConfigServer,
   5s TTL, repopulated on every HTTP request via resolveAllMcpConfigs)

Connection/tool-call paths without tenant context rely exclusively on
the readThrough cache. If it expires before the next HTTP request
repopulates it, the server is not found — which is correct because
there is no tenant context to determine which config to return.

- Remove findInConfigCache method and its call in getServerConfig
- Update server-only readThrough fallback to only return truthy values
  (prevents cached undefined from short-circuiting user-scoped DB lookup)
- Update tests to document tenant isolation behavior after cache expiry

* style: fix import order per AGENTS.md conventions

Sort package imports shortest-to-longest, local imports longest-to-shortest
across MCPServersRegistry, ConnectionsRepository, MCPManager,
UserConnectionManager, and MCPServerInspector.

* fix: eliminate cross-tenant readThrough contamination and TTL-expiry tool failures

Thread pre-resolved serverConfig from tool creation context into
callTool, removing dependency on the readThrough cache for config-source
servers. This fixes two issues:

- Cross-tenant contamination: the readThrough cache key was unscoped
  (just serverName), so concurrent multi-tenant requests for same-named
  servers would overwrite each other's entries
- TTL expiry: tool calls happening >5s after config resolution would
  fail with "Configuration not found" because the readThrough entry
  had expired

Changes:
- Add optional serverConfig param to MCPManager.callTool — uses
  provided config directly, falling back to getServerConfig lookup
  for YAML/user servers
- Thread serverConfig from createMCPTool through createToolInstance
  closure to callTool
- Remove readThrough write from ensureSingleConfigServer — config-source
  servers are only accessible via configServers param (tenant-scoped)
- Remove server-only readThrough fallback from getServerConfig
- Increase config cache hash from 8 to 16 hex chars (64-bit)
- Add isUserSourced boundary tests for all source/dbId combinations
- Fix double Object.keys call in getMCPTools controller
- Update test assertions for new getServerConfig behavior

* fix: cache base configs for config-server users; narrow upsertConfigCache error handling

- Refactor getAllServerConfigs to separate base config fetch (YAML + DB)
  from config-server layering. Base configs are cached via readThroughCacheAll
  regardless of whether configServers is provided, eliminating uncached
  MongoDB queries per request for config-server users
- Narrow upsertConfigCache catch to duplicate-key errors only;
  infrastructure errors (Redis timeouts, network failures) now propagate
  instead of being silently swallowed, preventing inspection storms
  during outages

* fix: restore correct merge order and document upsert error matching

- Restore YAML → Config → User DB precedence in getAllServerConfigs
  (user DB servers have highest precedence, matching the JSDoc contract)
- Add source comment on upsertConfigCache duplicate-key detection
  linking to the two cache implementations that define the error message

* feat: complete config-source server support across all execution paths

Wire configServers through the entire agent execution pipeline so
config-source MCP servers are fully functional — not just visible in
listings but executable in agent sessions.

- Thread configServers into handleTools.js agent tool pipeline: resolve
  config servers from tenant context before MCP tool iteration, pass to
  getServerConfig, createMCPTools, and createMCPTool
- Thread configServers into agent instructions pipeline:
  applyContextToAgent → getMCPInstructionsForServers →
  formatInstructionsForContext, resolved in client.js before agent
  context application
- Add configServers param to createMCPTool and createMCPTools for
  reconnect path fallback
- Add source field to redactServerSecrets allowlist for client UI
  differentiation of server tiers
- Narrow invalidateConfigCache to only clear readThroughCacheAll (merged
  results), preserving YAML individual-server readThrough entries
- Update context.spec.ts assertions for new configServers parameter

* fix: add missing mocks for config-source server dependencies in client.test.js

Mock getMCPServersRegistry, getAppConfig, and getTenantId that were added
to client.js but not reflected in the test file's jest.mock declarations.

* fix: update formatInstructionsForContext assertions for configServers param

The test assertions expected formatInstructionsForContext to be called with
only the server names array, but it now receives configServers as a second
argument after the config-source server feature wiring.

* fix: move configServers resolution before MCP tool loop to avoid TDZ

configServers was declared with `let` after the first tool loop but
referenced inside it via getServerConfig(), causing a ReferenceError
temporal dead zone. Move declaration and resolution before the loop,
using tools.some(mcpToolPattern) to gate the async resolution.

* fix: address review findings — cache bypass, discoverServerTools gap, DRY

- #2: getAllServerConfigs now always uses getBaseServerConfigs (cached via
  readThroughCacheAll) instead of bypassing it when configServers is present.
  Extracts user-DB entries from cached base by diffing against YAML keys
  to maintain YAML → Config → User DB merge order without extra MongoDB calls.

- #3: Add configServers param to ToolDiscoveryOptions and thread it through
  discoverServerTools → getServerConfig so config-source servers are
  discoverable during OAuth reconnection flows.

- #6: Replace inline import() type annotations in context.ts with proper
  import type { ParsedServerConfig } per AGENTS.md conventions.

- #7: Extract resolveConfigServers(req) helper in MCP.js and use it from
  handleTools.js and client.js, eliminating the duplicated 6-line config
  resolution pattern.

- #10: Restore removed "why" comment explaining getLoaded() vs getAll()
  choice in getMCPSetupData — documents non-obvious correctness constraint.

- #11: Fix incomplete JSDoc param type on resolveAllMcpConfigs.

* fix: consolidate imports, reorder constants, fix YAML-DB merge edge case

- Merge duplicate @librechat/data-schemas requires in MCP.js into one
- Move resolveConfigServers after module-level constants
- Fix getAllServerConfigs edge case where user-DB entry overriding a
  YAML entry with the same name was excluded from userDbConfigs; now
  uses reference equality check to detect DB-overwritten YAML keys

* fix: replace fragile string-match error detection with proper upsert method

Add upsert() to IServerConfigsRepositoryInterface and all implementations
(InMemory, Redis, RedisAggregateKey, DB). This eliminates the brittle
error message string match ('already exists in cache') in upsertConfigCache
that was the only thing preventing cross-process init races from silently
discarding inspection results.

Each implementation handles add-or-update atomically:
- InMemory: direct Map.set()
- Redis: direct cache.set()
- RedisAggregateKey: read-modify-write under write lock
- DB: delegates to update() (DB servers use explicit add() with ACL setup)

* fix: wire configServers through remaining HTTP endpoints

- getMCPServerById: use resolveAllMcpConfigs instead of bare getServerConfig
- reinitialize route: resolve configServers before getServerConfig
- auth-values route: resolve configServers before getServerConfig
- getOAuthHeaders: accept configServers param, thread from callers
- Update mcp.spec.js tests to mock getAllServerConfigs for GET by name

* fix: thread serverConfig through getConnection for config-source servers

Config-source servers exist only in configCacheRepo, not in YAML cache or
DB. When callTool → getConnection → getUserConnection → getServerConfig
runs without configServers, it returns undefined and throws. Fix by
threading the pre-resolved serverConfig (providedConfig) from callTool
through getConnection → getUserConnection → createUserConnectionInternal,
using it as a fallback before the registry lookup.

* fix: thread configServers through reinit, reconnect, and tool definition paths

Wire configServers through every remaining call chain that creates or
reconnects MCP server connections:

- reinitMCPServer: accepts serverConfig and configServers, uses them for
  getServerConfig fallback, getConnection, and discoverServerTools
- reconnectServer: accepts and passes configServers to reinitMCPServer
- createMCPTools/createMCPTool: pass configServers to reconnectServer
- ToolService.loadToolDefinitionsWrapper: resolves configServers from req,
  passes to both reinitMCPServer call sites
- reinitialize route: passes serverConfig and configServers to reinitMCPServer

* fix: address review findings — simplify merge, harden error paths, fix log labels

- Simplify getAllServerConfigs merge: replace fragile reference-equality
  loop with direct spread { ...yamlConfigs, ...configServers, ...base }
- Guard upsertConfigCache in lazyInitConfigServer catch block so cache
  failures don't mask the original inspection error
- Deduplicate getYamlServerNames cold-start with promise dedup pattern
- Remove dead `if (!mcpConfig)` guard in getMCPSetupData
- Fix hardcoded "App server" in ServerConfigsCacheRedisAggregateKey error
  messages — now uses this.namespace for correct Config/App labeling
- Remove misleading OAuth callback comment about readThrough cache
- Move resolveConfigServers after module-level constants in MCP.js

* fix: clear rejected yamlServerNames promise, fix config-source reinspect, fix reset log label

- Clear yamlServerNamesPromise on rejection so transient cache errors
  don't permanently prevent ensureConfigServers from working
- Skip reinspectServer for config-source servers (source: 'config') in
  reinitMCPServer — they lack a CACHE/DB storage location; retry is
  handled by CONFIG_STUB_RETRY_MS in ensureConfigServers
- Use source field instead of dbId for storageLocation derivation
- Fix remaining hardcoded "App" in reset() leaderCheck message

* fix: persist oauthHeaders in flow state for config-source OAuth servers

The OAuth callback route has no JWT auth context and cannot resolve
config-source server configs. Previously, getOAuthHeaders would silently
return {} for config-source servers, dropping custom token exchange headers.

Now oauthHeaders are persisted in MCPOAuthFlowMetadata during flow
initiation (which has auth context), and the callback reads them from
the stored flow state with a fallback to the registry lookup for
YAML/user-DB servers.

* fix: update tests for getMCPSetupData null guard removal and ToolService mock

- MCP.spec.js: update test to expect graceful handling of null mcpConfig
  instead of a throw (getAllServerConfigs always returns an object)
- MCP.js: add defensive || {} for Object.entries(mcpConfig) in case of
  null from test mocks
- ToolService.spec.js: add missing mock for ~/server/services/MCP
  (resolveConfigServers)

* fix: address review findings — DRY, naming, logging, dead code, defensive guards

- #1: Simplify getAllServerConfigs to single getBaseServerConfigs call,
  eliminating redundant double-fetch of cacheConfigsRepo.getAll()
- #2: Add warning log when oauthHeaders absent from OAuth callback flow state
- #3: Extract resolveAllMcpConfigs to MCP.js service layer; controller
  imports shared helper instead of reimplementing
- #4: Rename _serverConfig/_provider to capturedServerConfig/capturedProvider
  in createToolInstance — these are actively used, not unused
- #5: Log rejected results from ensureConfigServers Promise.allSettled
  so cache errors are visible instead of silently dropped
- #6: Remove dead 'MCP config not found' error handlers from routes
- #7: Document circular-dependency reason for dynamic require in clearMcpConfigCache
- #8: Remove logger.error from withTimeout to prevent double-logging timeouts
- #10: Add explicit userId guard in ServerConfigsDB.upsert with clear error message
- #12: Use spread instead of mutation in addServer for immutability consistency
- Add upsert mock to ensureConfigServers.test.ts DB mock
- Update route tests for resolveAllMcpConfigs import change

* fix: restore correct merge priority, use immutable spread, fix test mock

- getAllServerConfigs: { ...configServers, ...base } so userDB wins over
  configServers, matching documented "User DB (highest)" priority
- lazyInitConfigServer: use immutable spread instead of direct mutation
  for parsedConfig.source, consistent with addServer fix
- Fix test to mock getAllServerConfigs as {} instead of null, remove
  unnecessary || {} defensive guard in getMCPSetupData

* fix: error handling, stable hashing, flatten nesting, remove dead param

- Wrap resolveConfigServers/resolveAllMcpConfigs in try/catch with
  graceful {} fallback so transient DB/cache errors don't crash tool pipeline
- Sort keys in configCacheKey JSON.stringify for deterministic hashing
  regardless of object property insertion order
- Flatten clearMcpConfigCache from 3 nested try-catch to early returns;
  document that user connections are cleaned up lazily (accepted tradeoff)
- Remove dead configServers param from getAppToolFunctions (never passed)
- Add security rationale comment for source field in redactServerSecrets

* fix: use recursive key-sorting replacer in configCacheKey to prevent cross-tenant cache collision

The array replacer in JSON.stringify acts as a property allowlist at
every nesting depth, silently dropping nested keys like headers['X-API-Key'],
oauth.client_secret, etc. Two configs with different nested values but
identical top-level structure produced the same hash, causing cross-tenant
cache hits and potential credential contamination.

Switch to a function replacer that recursively sorts keys at all depths
without dropping any properties.

Also document the known gap in getOAuthServers: config-source OAuth
servers are not covered by auto-reconnection or uninstall cleanup
because callers lack request context.

* fix: move clearMcpConfigCache to packages/api to eliminate circular dependency

The function only depends on MCPServersRegistry and MCPManager, both of
which live in packages/api. Import it directly from @librechat/api in
the CJS layer instead of using dynamic require('~/config').

* chore: imports/fields ordering

* fix: address review findings — error handling, targeted lookup, test gaps

- Narrow resolveAllMcpConfigs catch to only wrap ensureConfigServers so
  getAppConfig/getAllServerConfigs failures propagate instead of masking
  infrastructure errors as empty server lists.
- Use targeted getServerConfig in getMCPServerById instead of fetching
  all server configs for a single-server lookup.
- Forward configServers to inner createMCPTool calls so reconnect path
  works for config-source servers.
- Update getAllServerConfigs JSDoc to document disjoint-key design.
- Add OAuth callback oauthHeaders fallback tests (flow state present
  vs registry fallback).
- Add resolveConfigServers/resolveAllMcpConfigs unit tests covering
  happy path and error propagation.

* fix: add getOAuthReconnectionManager mock to OAuth callback tests

* chore: imports ordering
2026-03-28 10:36:43 -04:00
Danny Avila
fa9e1b228a
🪪 fix: MCP API Responses and OAuth Validation (#12217)
* 🔒 fix: Validate MCP Configs in Server Responses

* 🔒 fix: Enhance OAuth URL Validation in MCPOAuthHandler

- Introduced validation for OAuth URLs to ensure they do not target private or internal addresses, enhancing security against SSRF attacks.
- Updated the OAuth flow to validate both authorization and token URLs before use, ensuring compliance with security standards.
- Refactored redirect URI handling to streamline the OAuth client registration process.
- Added comprehensive error handling for invalid URLs, improving robustness in OAuth interactions.

* 🔒 feat: Implement Permission Checks for MCP Server Management

- Added permission checkers for MCP server usage and creation, enhancing access control.
- Updated routes for reinitializing MCP servers and retrieving authentication values to include these permission checks, ensuring only authorized users can access these functionalities.
- Refactored existing permission logic to improve clarity and maintainability.

* 🔒 fix: Enhance MCP Server Response Validation and Redaction

- Updated MCP route tests to use `toMatchObject` for better validation of server response structures, ensuring consistency in expected properties.
- Refactored the `redactServerSecrets` function to streamline the removal of sensitive information, ensuring that user-sourced API keys are properly redacted while retaining their source.
- Improved OAuth security tests to validate rejection of private URLs across multiple endpoints, enhancing protection against SSRF vulnerabilities.
- Added comprehensive tests for the `redactServerSecrets` function to ensure proper handling of various server configurations, reinforcing security measures.

* chore: eslint

* 🔒 fix: Enhance OAuth Server URL Validation in MCPOAuthHandler

- Added validation for discovered authorization server URLs to ensure they meet security standards.
- Improved logging to provide clearer insights when an authorization server is found from resource metadata.
- Refactored the handling of authorization server URLs to enhance robustness against potential security vulnerabilities.

* 🔒 test: Bypass SSRF validation for MCP OAuth Flow tests

- Mocked SSRF validation functions to allow tests to use real local HTTP servers, facilitating more accurate testing of the MCP OAuth flow.
- Updated test setup to ensure compatibility with the new mocking strategy, enhancing the reliability of the tests.

* 🔒 fix: Add Validation for OAuth Metadata Endpoints in MCPOAuthHandler

- Implemented checks for the presence and validity of registration and token endpoints in the OAuth metadata, enhancing security by ensuring that these URLs are properly validated before use.
- Improved error handling and logging to provide better insights during the OAuth metadata processing, reinforcing the robustness of the OAuth flow.

* 🔒 refactor: Simplify MCP Auth Values Endpoint Logic

- Removed redundant permission checks for accessing the MCP server resource in the auth-values endpoint, streamlining the request handling process.
- Consolidated error handling and response structure for improved clarity and maintainability.
- Enhanced logging for better insights during the authentication value checks, reinforcing the robustness of the endpoint.

* 🔒 test: Refactor LeaderElection Integration Tests for Improved Cleanup

- Moved Redis key cleanup to the beforeEach hook to ensure a clean state before each test.
- Enhanced afterEach logic to handle instance resignations and Redis key deletion more robustly, improving test reliability and maintainability.
2026-03-13 23:18:56 -04:00
Danny Avila
f32907cd36
🔏 fix: MCP Server URL Schema Validation (#12204)
* fix: MCP server configuration validation and schema

- Added tests to reject URLs containing environment variable references for SSE, streamable-http, and websocket types in the MCP routes.
- Introduced a new schema in the data provider to ensure user input URLs do not resolve environment variables, enhancing security against potential leaks.
- Updated existing MCP server user input schema to utilize the new validation logic, ensuring consistent handling of user-supplied URLs across the application.

* fix: MCP URL validation to reject env variable references

- Updated tests to ensure that URLs for SSE, streamable-http, and websocket types containing environment variable patterns are rejected, improving security against potential leaks.
- Refactored the MCP server user input schema to enforce stricter validation rules, preventing the resolution of environment variables in user-supplied URLs.
- Introduced new test cases for various URL types to validate the rejection logic, ensuring consistent handling across the application.

* test: Enhance MCPServerUserInputSchema tests for environment variable handling

- Introduced new test cases to validate the prevention of environment variable exfiltration through user input URLs in the MCPServerUserInputSchema.
- Updated existing tests to confirm that URLs containing environment variable patterns are correctly resolved or rejected, improving security against potential leaks.
- Refactored test structure to better organize environment variable handling scenarios, ensuring comprehensive coverage of edge cases.
2026-03-12 23:19:31 -04:00
Danny Avila
fcb344da47
🛂 fix: MCP OAuth Race Conditions, CSRF Fallback, and Token Expiry Handling (#12171)
* fix: Implement race conditions in MCP OAuth flow

- Added connection mutex to coalesce concurrent `getUserConnection` calls, preventing multiple simultaneous attempts.
- Enhanced flow state management to retry once when a flow state is missing, improving resilience against race conditions.
- Introduced `ReauthenticationRequiredError` for better error handling when access tokens are expired or missing.
- Updated tests to cover new race condition scenarios and ensure proper handling of OAuth flows.

* fix: Stale PENDING flow detection and OAuth URL re-issuance

PENDING flows in handleOAuthRequired now check createdAt age — flows
older than 2 minutes are treated as stale and replaced instead of
joined. Fixes the case where a leftover PENDING flow from a previous
session blocks new OAuth initiation.

authorizationUrl is now stored in MCPOAuthFlowMetadata so that when a
second caller joins an active PENDING flow (e.g., the SSE-emitting path
in ToolService), it can re-issue the URL to the user via oauthStart.

* fix: CSRF fallback via active PENDING flow in OAuth callback

When the OAuth callback arrives without CSRF or session cookies (common
in the chat/SSE flow where cookies can't be set on streaming responses),
fall back to validating that a PENDING flow exists for the flowId. This
is safe because the flow was created server-side after JWT authentication
and the authorization code is PKCE-protected.

* test: Extract shared OAuth test server helpers

Move MockKeyv, getFreePort, trackSockets, and createOAuthMCPServer into
a shared helpers/oauthTestServer module. Enhance the test server with
refresh token support, token rotation, metadata discovery, and dynamic
client registration endpoints. Add InMemoryTokenStore for token storage
tests.

Refactor MCPOAuthRaceCondition.test.ts to import from shared helpers.

* test: Add comprehensive MCP OAuth test modules

MCPOAuthTokenStorage — 21 tests for storeTokens/getTokens with
InMemoryTokenStore: encrypt/decrypt round-trips, expiry calculation,
refresh callback wiring, ReauthenticationRequiredError paths.

MCPOAuthFlow — 10 tests against real HTTP server: token refresh with
stored client info, refresh token rotation, metadata discovery, dynamic
client registration, full store/retrieve/expire/refresh lifecycle.

MCPOAuthConnectionEvents — 5 tests for MCPConnection OAuth event cycle
with real OAuth-gated MCP server: oauthRequired emission on 401,
oauthHandled reconnection, oauthFailed rejection, token expiry detection.

MCPOAuthTokenExpiry — 12 tests for the token expiry edge case: refresh
success/failure paths, ReauthenticationRequiredError, PENDING flow CSRF
fallback, authorizationUrl metadata storage, full re-auth cycle after
refresh failure, concurrent expired token coalescing, stale PENDING
flow detection.

* test: Enhance MCP OAuth connection tests with cooldown reset

Added a `beforeEach` hook to clear the cooldown for `MCPConnection` before each test, ensuring a clean state. Updated the race condition handling in the tests to properly clear the timeout, improving reliability in the event data retrieval process.

* refactor: PENDING flow management and state recovery in MCP OAuth

- Introduced a constant `PENDING_STALE_MS` to define the age threshold for PENDING flows, improving the handling of stale flows.
- Updated the logic in `MCPConnectionFactory` and `FlowStateManager` to check the age of PENDING flows before joining or reusing them.
- Modified the `completeFlow` method to return false when the flow state is deleted, ensuring graceful handling of race conditions.
- Enhanced tests to validate the new behavior and ensure robustness against state recovery issues.

* refactor: MCP OAuth flow management and testing

- Updated the `completeFlow` method to log warnings when a tool flow state is not found during completion, improving error handling.
- Introduced a new `normalizeExpiresAt` function to standardize expiration timestamp handling across the application.
- Refactored token expiration checks in `MCPConnectionFactory` to utilize the new normalization function, ensuring consistent behavior.
- Added a comprehensive test suite for OAuth callback CSRF fallback logic, validating the handling of PENDING flows and their staleness.
- Enhanced existing tests to cover new expiration normalization logic and ensure robust flow state management.

* test: Add CSRF fallback tests for active PENDING flows in MCP OAuth

- Introduced new tests to validate CSRF fallback behavior when a fresh PENDING flow exists without cookies, ensuring successful OAuth callback handling.
- Added scenarios to reject requests when no PENDING flow exists, when only a COMPLETED flow is present, and when a PENDING flow is stale, enhancing the robustness of flow state management.
- Improved overall test coverage for OAuth callback logic, reinforcing the handling of CSRF validation failures.

* chore: imports order

* refactor: Update UserConnectionManager to conditionally manage pending connections

- Modified the logic in `UserConnectionManager` to only set pending connections if `forceNew` is false, preventing unnecessary overwrites.
- Adjusted the cleanup process to ensure pending connections are only deleted when not forced, enhancing connection management efficiency.

* refactor: MCP OAuth flow state management

- Introduced a new method `storeStateMapping` in `MCPOAuthHandler` to securely map the OAuth state parameter to the flow ID, improving callback resolution and security against forgery.
- Updated the OAuth initiation and callback handling in `mcp.js` to utilize the new state mapping functionality, ensuring robust flow management.
- Refactored `MCPConnectionFactory` to store state mappings during flow initialization, enhancing the integrity of the OAuth process.
- Adjusted comments to clarify the purpose of state parameters in authorization URLs, reinforcing code readability.

* refactor: MCPConnection with OAuth recovery handling

- Added `oauthRecovery` flag to manage OAuth recovery state during connection attempts.
- Introduced `decrementCycleCount` method to reduce the circuit breaker's cycle count upon successful reconnection after OAuth recovery.
- Updated connection logic to reset the `oauthRecovery` flag after handling OAuth, improving state management and connection reliability.

* chore: Add debug logging for OAuth recovery cycle count decrement

- Introduced a debug log statement in the `MCPConnection` class to track the decrement of the cycle count after a successful reconnection during OAuth recovery.
- This enhancement improves observability and aids in troubleshooting connection issues related to OAuth recovery.

* test: Add OAuth recovery cycle management tests

- Introduced new tests for the OAuth recovery cycle in `MCPConnection`, validating the decrement of cycle counts after successful reconnections.
- Added scenarios to ensure that the cycle count is not decremented on OAuth failures, enhancing the robustness of connection management.
- Improved test coverage for OAuth reconnect scenarios, ensuring reliable behavior under various conditions.

* feat: Implement circuit breaker configuration in MCP

- Added circuit breaker settings to `.env.example` for max cycles, cycle window, and cooldown duration.
- Refactored `MCPConnection` to utilize the new configuration values from `mcpConfig`, enhancing circuit breaker management.
- Improved code maintainability by centralizing circuit breaker parameters in the configuration file.

* refactor: Update decrementCycleCount method for circuit breaker management

- Changed the visibility of the `decrementCycleCount` method in `MCPConnection` from private to public static, allowing it to be called with a server name parameter.
- Updated calls to `decrementCycleCount` in `MCPConnectionFactory` to use the new static method, improving clarity and consistency in circuit breaker management during connection failures and OAuth recovery.
- Enhanced the handling of circuit breaker state by ensuring the method checks for the existence of the circuit breaker before decrementing the cycle count.

* refactor: cycle count decrement on tool listing failure

- Added a call to `MCPConnection.decrementCycleCount` in the `MCPConnectionFactory` to handle cases where unauthenticated tool listing fails, improving circuit breaker management.
- This change ensures that the cycle count is decremented appropriately, maintaining the integrity of the connection recovery process.

* refactor: Update circuit breaker configuration and logic

- Enhanced circuit breaker settings in `.env.example` to include new parameters for failed rounds and backoff strategies.
- Refactored `MCPConnection` to utilize the updated configuration values from `mcpConfig`, improving circuit breaker management.
- Updated tests to reflect changes in circuit breaker logic, ensuring accurate validation of connection behavior under rapid reconnect scenarios.

* feat: Implement state mapping deletion in MCP flow management

- Added a new method `deleteStateMapping` in `MCPOAuthHandler` to remove orphaned state mappings when a flow is replaced, preventing old authorization URLs from resolving after a flow restart.
- Updated `MCPConnectionFactory` to call `deleteStateMapping` during flow cleanup, ensuring proper management of OAuth states.
- Enhanced test coverage for state mapping functionality to validate the new deletion logic.
2026-03-10 21:15:01 -04:00
Danny Avila
599f4a11f1
🛡️ fix: Secure MCP/Actions OAuth Flows, Resolve Race Condition & Tool Cache Cleanup (#11756)
* 🔧 fix: Update OAuth error message for clarity

- Changed the default error message in the OAuth error route from 'Unknown error' to 'Unknown OAuth error' to provide clearer context during authentication failures.

* 🔒 feat: Enhance OAuth flow with CSRF protection and session management

- Implemented CSRF protection for OAuth flows by introducing `generateOAuthCsrfToken`, `setOAuthCsrfCookie`, and `validateOAuthCsrf` functions.
- Added session management for OAuth with `setOAuthSession` and `validateOAuthSession` middleware.
- Updated routes to bind CSRF tokens for MCP and action OAuth flows, ensuring secure authentication.
- Enhanced tests to validate CSRF handling and session management in OAuth processes.

* 🔧 refactor: Invalidate cached tools after user plugin disconnection

- Added a call to `invalidateCachedTools` in the `updateUserPluginsController` to ensure that cached tools are refreshed when a user disconnects from an MCP server after a plugin authentication update. This change improves the accuracy of tool data for users.

* chore: imports order

* fix: domain separator regex usage in ToolService

- Moved the declaration of `domainSeparatorRegex` to avoid redundancy in the `loadActionToolsForExecution` function, improving code clarity and performance.

* chore: OAuth flow error handling and CSRF token generation

- Enhanced the OAuth callback route to validate the flow ID format, ensuring proper error handling for invalid states.
- Updated the CSRF token generation function to require a JWT secret, throwing an error if not provided, which improves security and clarity in token generation.
- Adjusted tests to reflect changes in flow ID handling and ensure robust validation across various scenarios.
2026-02-12 14:22:05 -05:00
Danny Avila
211b39f311
🔒 fix: Restrict MCP Stdio Transport via API (#11184)
- Updated MCP server configuration tests to reject stdio transport configurations, ensuring that only remote transports (SSE, HTTP, WebSocket) are allowed via the API.
- Enhanced documentation to clarify that stdio transport is excluded from user input for security, as it allows arbitrary command execution and should only be configured by administrators through YAML files.
2026-01-03 12:47:11 -05:00
Artyom Bogachenko
7844a93f8b
♻️ fix: use DOMAIN_CLIENT for MCP OAuth Redirects (#11057)
Co-authored-by: Artyom Bogachenco <a.bogachenko@easyreport.ai>
2025-12-25 12:24:01 -05:00