Commit graph

736 commits

Author SHA1 Message Date
Dustin Healy
5840e2f2ae
🪓 feat: Add Tenant-Scoped Agent Deletion (#15573)
* feat: add Agent Management delete endpoint

* fix: scope agent deletion cleanup by tenant
2026-09-04 09:51:12 -04:00
Danny Avila
78b320cbbc
🗜️ fix: Bound Conversation Page Size (#15576)
`GET /api/convos` read its page size as `parseInt(req.query.limit, 10) || 25`
and handed it to `getConvosByCursor` unclamped, where it becomes
`.limit(limit + 1)`. `?limit=-1` therefore issued `.limit(0)`, which MongoDB
reads as no limit at all, and the route returned the caller's entire
conversation list in one response; `?limit=999999999` forced the same unbounded
fetch. `attachSharedFlags` then amplified it with a second query whose `$in`
carried every returned id.

Every other list route already clamps, so the fix reuses that shape: the
projects handler's local `normalizeLimit` moves to `packages/api/src/utils/query`
and both routes share it, and `getConvosByCursor` bounds its own page size so
the invariant holds for any caller.
2026-09-03 23:52:36 -04:00
Danny Avila
32e8724193
🏗️ fix: Wait Out Concurrent Index Builds on DocumentDB (#15563)
* 🏗️ fix: Wait Out Concurrent Index Builds on DocumentDB

Mongoose starts every compiled model's automatic index build in the
background as soon as the connection opens. The durable trigger delivery
boot step then issues an explicit createIndexes() for the same collection
while that build is still running. Amazon DocumentDB allows one index build
per collection and rejects the second with code 40333, which the retry
helper did not treat as transient, so the trigger service failed to
initialize and the post-listen catch exited the process on every boot.

createIndexesWithRetry now lets the model's automatic build settle before
building explicitly, and treats the single-build rejection as retryable
with a longer backoff so a peer replica's build of the same collection is
waited out instead of crash-looping the process.

Fixes #15556

* 🏗️ fix: Poll Instead of Backing Off While a Peer Holds the Index Build

A peer replica's index build time is data-dependent, so a fixed retry
budget either expires into the same process exit this change removes or
grows arbitrarily. The single-build conflict is now handled by polling at
a fixed interval with progress logging, unbounded unless a deadline is
supplied; other errors keep the existing backoff and fail-fast behaviour.
2026-09-03 22:28:27 -04:00
Dustin Healy
2fe872d2ea
👀 feat: Add Agent Management Read Endpoints (#15561)
* feat: add Agent Management read endpoints

* fix: protect Agent management configuration reads

* fix: type Agent management list filters

* test: use typed Agent management assertions
2026-09-03 21:16:14 -04:00
Danny Avila
a6cf61abcf
🪵 feat: Add OpenTelemetry Logs Pipeline via Winston Bridge (#15565)
Exports application logs as OTel log records when OTEL_LOGS_ENABLED=true,
correlated with the active trace, and gates every signal explicitly so a
disabled signal never falls back to the SDK's default OTLP exporter.
Verified end to end over OTLP/gRPC against an in-process collector.
2026-09-03 21:16:09 -04:00
Danny Avila
458c473d9e
📧 fix: Normalize Token Emails on Write, as Every Read Already Does (#15536)
Some checks are pending
Backend Unit Tests / Build packages (push) Waiting to run
Backend Unit Tests / Codegraph select (push) Waiting to run
Backend Unit Tests / TypeScript type checks (push) Blocked by required conditions
Backend Unit Tests / Circular dependency checks (push) Waiting to run
Backend Unit Tests / Tests: api (shard 1/3) (push) Blocked by required conditions
Backend Unit Tests / Tests: api (shard 2/3) (push) Blocked by required conditions
Backend Unit Tests / Tests: api (shard 3/3) (push) Blocked by required conditions
Backend Unit Tests / Tests: data-provider (push) Blocked by required conditions
Backend Unit Tests / Tests: data-schemas (push) Blocked by required conditions
Backend Unit Tests / Tests: @librechat/api (shard 1/4) (push) Blocked by required conditions
Backend Unit Tests / Tests: @librechat/api (shard 2/4) (push) Blocked by required conditions
Backend Unit Tests / Tests: @librechat/api (shard 3/4) (push) Blocked by required conditions
Backend Unit Tests / Tests: @librechat/api (shard 4/4) (push) Blocked by required conditions
Codegraph E2E Votes / vote (full suite) (push) Waiting to run
Docker Dev Branch Images Build / publish (push) Waiting to run
* 📧 fix: Normalize Token Emails on Write, as Every Read Already Does

`findToken` and `deleteTokens` both trim and lowercase the email before querying —
the comment on `findToken` even documents it as automatic — but nothing normalized
the write. A token created with mixed case or surrounding whitespace could therefore
never be found or deleted again: an invite issued to `User@Example.com` was
unredeemable.

Fixed on the schema rather than in a caller. `createInvite` is the caller that
surfaced it (#15529), but the asymmetry belongs to `Token`, so every writer had the
same trap and any new one would inherit it. `User.email` is already declared
`lowercase: true` in this package, so this only brings `Token.email` in line with the
convention beside it — and a schema-level setter cannot be bypassed the way a helper
can.

Existing documents are not migrated. A token already stored with mixed case stays
unfindable, exactly as it is today; only new writes are corrected.

Fixes #15529

* ✏️ fix: Correct the Schema Comment and Name the No-Email Test for What It Asserts

Both from Copilot's review.

`User.email` sets `lowercase` but not `trim`, so claiming this "mirrors" it overstated
the parallel — this change trims as well, because the read side does. The comment now
states the `findToken`/`deleteTokens` contract it actually follows.

The no-email test was named for `null` while omitting the field and asserting
`undefined`. Renamed to match, and a real null case added: `TokenCreateData.email` is
`string | undefined` so null is only reachable by writing the model directly, but both
read methods branch on `email === null`, so it is worth proving the new setters leave
that value alone.
2026-09-03 07:46:41 -04:00
Danny Avila
f9f1b2fb95
✨ v0.8.8-rc2 (#14845)
Some checks failed
Backend Unit Tests / Codegraph select (push) Waiting to run
Backend Unit Tests / TypeScript type checks (push) Blocked by required conditions
Backend Unit Tests / Circular dependency checks (push) Waiting to run
Backend Unit Tests / Tests: api (shard 1/3) (push) Blocked by required conditions
Backend Unit Tests / Tests: api (shard 2/3) (push) Blocked by required conditions
Backend Unit Tests / Tests: api (shard 3/3) (push) Blocked by required conditions
Backend Unit Tests / Tests: data-provider (push) Blocked by required conditions
Backend Unit Tests / Tests: data-schemas (push) Blocked by required conditions
Backend Unit Tests / Tests: @librechat/api (shard 1/4) (push) Blocked by required conditions
Backend Unit Tests / Tests: @librechat/api (shard 2/4) (push) Blocked by required conditions
Backend Unit Tests / Tests: @librechat/api (shard 3/4) (push) Blocked by required conditions
Backend Unit Tests / Tests: @librechat/api (shard 4/4) (push) Blocked by required conditions
Codegraph E2E Votes / vote (full suite) (push) Waiting to run
Docker Dev Branch Images Build / publish (push) Waiting to run
Frontend Unit Tests / Codegraph select (push) Waiting to run
Frontend Unit Tests / Build packages (push) Waiting to run
Frontend Unit Tests / TypeScript type checks (client) (push) Blocked by required conditions
Frontend Unit Tests / Tests: @librechat/client (push) Blocked by required conditions
Frontend Unit Tests / Tests: Ubuntu (shard 1/2) (push) Blocked by required conditions
Frontend Unit Tests / Tests: Ubuntu (shard 2/2) (push) Blocked by required conditions
Frontend Unit Tests / Vite build verification (push) Blocked by required conditions
Publish `@librechat/client` to NPM / pack (push) Has been cancelled
Publish `librechat-data-provider` to NPM / pack (push) Has been cancelled
Publish `@librechat/data-schemas` to NPM / pack (push) Has been cancelled
Docker Dev Images Build / publish (push) Has been cancelled
Sync Helm Chart Tags / Ignore non-main push (push) Has been cancelled
Sync Helm Chart Tags / Sync chart tags (push) Has been cancelled
Publish `@librechat/client` to NPM / publish-npm (push) Has been cancelled
Publish `librechat-data-provider` to NPM / publish-npm (push) Has been cancelled
Publish `@librechat/data-schemas` to NPM / publish-npm (push) Has been cancelled
* chore: prepare v0.8.8-rc2 release

* test: satisfy mongo meili static checks

* test: avoid unsafe logger spy assertion

* chore: refresh v0.8.8-rc2 release

* test: restore mongo meili test isolation

* chore: normalize v0.8.8-rc2 versions

* docs: refresh v0.8.8-rc2 release surfaces
2026-09-02 22:18:48 -04:00
Danny Avila
8a46ca070c
🧊 feat: Persist Context Fading Tier Across Agent Runs (#15505)
* 🧊 feat: Persist Context Fading Tier Across Agent Runs

Carry the pruner's latched context-fading tier from `@librechat/agents`
through `contextMeta` the way `calibrationRatio` already travels, so a
historical tool result keeps the same truncated bytes from one run to the
next and Anthropic's prefix-based prompt cache survives across turns
(danny-avila/agents#497).

- Add `IAgentFadingTier` to the message and conversation `contextMeta`
  types, schemas and the data-provider zod schema
- Add `isAgentFadingTier`, `resolvePersistableFadingTier` and
  `resolveRunContextMeta` in `packages/api`; a tier is persisted once it
  carries information (masking active or a budget below the window), and
  a latched tier is kept even at a neutral calibration ratio
- Seed `createRun({ fadingTier })` from the parent response's contextMeta
  regardless of encoding, since caps are character-based; the field rides
  the existing forward-compatible `runConfig` variable and older SDK
  versions ignore it, as does `Run.getFadingTier` being absent
- Accept a valid `fading` field in event actor context and reject a
  malformed one, matching the calibration checks
- Capture run context meta through one module-level helper for both
  completion paths instead of two inline copies

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WNbgMeR2aZDyhsqAy4L8gF

* 🧊 fix: Seed Fading Tier Across HITL Resume, Validate at the DB Layer

Address review findings on fading tier persistence:

- Leave the "is this tier worth persisting" decision to the SDK, whose
  `Run.getFadingTier()` now returns only informative tiers; comparing
  against the client's window misclassified untouched conversations
  whenever a reserve ratio was configured
- Carry `contextMeta` through the HITL pause projection (staged approval,
  job metadata, resume state) and seed the rebuilt client from it, so a
  paused turn resumes with the same tier and calibration instead of
  re-deriving a shallower one and rewriting the prefix
- Share run seeding between `chatCompletion` and `resumeCompletion`, and
  make capture tolerate partial runs and bare resume contexts
- Validate `contextMeta.fading` in `commitAgentEventActorState` with a
  shared `isAgentFadingTier` guard in data-schemas, so a malformed tier is
  rejected at the DB layer instead of silently cold-starting the actor
- Reuse `IAgentEventActorContextMeta` for `IMessage.contextMeta` instead of
  extending an inline duplicate

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WNbgMeR2aZDyhsqAy4L8gF

* 🧊 test: Cover Resume Seeding From Paused Context Meta

Assert the resume controller seeds the rebuilt client from the context meta
captured at the pause before rebuilding the run, and move the capture
helper's JSDoc back above its function.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WNbgMeR2aZDyhsqAy4L8gF

* 🧹 style: Sort Imports in Fading Tier Plumbing

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WNbgMeR2aZDyhsqAy4L8gF

* 🧊 fix: Persist Fading Tier Through Redis Jobs and Branches

Address Codex and Copilot review findings on fading tier persistence:

- Deserialize `contextMeta` in `RedisJobStore`, whose explicit read mapper
  rebuilt every other pause field but this one, so Redis-backed
  deployments resumed with no tier to seed after every HITL pause
- Copy server-private `contextMeta` onto the assistant message created by
  `POST /api/messages/branch`, so the next turn seeds its pruner the same
  way it would from the parallel source response
- Constrain the `fading` subdocuments in both Mongoose schemas (version
  enum, positive budget, required fields) to match the zod schema
- Throw a distinct "fading tier is invalid" error from both context meta
  validators instead of reporting a calibration failure

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WNbgMeR2aZDyhsqAy4L8gF

* 🧊 fix: Carry Context Meta Through Stop and Re-Pause

Addresses the second Codex pass on the fading-tier persistence:

- A re-pause whose resumed segment has nothing persistable now clears the
  job's `contextMeta` in the same transition instead of leaving the first
  pause's calibration and tier for the next resume to seed from.
- The client publishes the run's live calibration and fading tier onto the
  job after each pre-invoke context snapshot (deduplicated on value), and
  the Stop path copies `jobData.contextMeta` onto the stopped response, so
  a follow-up from a stopped turn keeps the prompt prefix stable.
- Mid-run captures (HITL pause, Stop) read the graph's live state; `Run`
  refreshes its own getters only after the stream settles, so the pause
  used to persist the seeds it started from.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WNbgMeR2aZDyhsqAy4L8gF

* 🧊 fix: Fence Context Meta Publication Ahead of Stop

A Stop handled by another replica reads the job before the owner is
signaled, so a fire-and-forget publish could lose the race, and a run
stopped before its first context snapshot never published the tier it
inherited. The client now publishes the inherited seed before the run
streams or resumes, and the context-usage handler awaits each
snapshot's publish so the write lands before the model call it
describes begins. Failures still only log; the pre-run publish is
optional-called so bare resume contexts keep working.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WNbgMeR2aZDyhsqAy4L8gF

* 🧊 fix: Read Back Context Meta After the Abort Claim

The run owner publishes its context meta ahead of each model call and
awaits the write, so a publish landing between the abort's initial job
read and its terminal claim describes the call whose partial output the
abort snapshot carries. abortJob now reads the same-epoch job back after
the claim and content re-read and takes its context meta, so the stopped
response persists the tier that produced its bytes. The refresh is
best-effort and logs on failure like the content refresh beside it.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WNbgMeR2aZDyhsqAy4L8gF

* 🧊 fix: Share In-Flight Context Meta Writes and Clear Neutral Live State

Two snapshot callbacks in a parallel-agent run can capture the same tier
before the first write settles; the publisher now caches the in-flight
promise per serialized value so the second caller awaits the same
durable write instead of treating an uncommitted value as published.
A live snapshot with nothing left to persist after an earlier publish now
writes a neutral record (ratio 1, no tier), since a running job's fields
cannot be deleted through the metadata writer, so a Stop no longer
persists an earlier non-neutral state onto the response. The pre-run
seed publish is unchanged and is told apart from live snapshots by a
flag from the context-usage sink.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WNbgMeR2aZDyhsqAy4L8gF

* 🧊 chore: Drop Unused Rest Parameter in Context Meta Test

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WNbgMeR2aZDyhsqAy4L8gF

* 🧊 feat: Persist Per-Agent Fading Tiers as Compact Context Meta

Reconciles the host with @librechat/agents at 9aacad1a, where graph
history stays canonical and each Run derives a provider-only projection
from the latched tiers. LibreChat therefore persists only compact state:
the calibration ratio, the default agent's tier, and now the per-agent
tiers from Run.getFadingTiers(), stored as validated entries so agent IDs
never become MongoDB field names and restored onto a null-prototype
record for RunConfig.fadingTiers. Truncated messages, canonical tool
content, projection provenance and per-message truncation state are
never persisted; the SDK's latched flag is stripped on capture.

The per-agent map travels everywhere the single tier already did:
response contextMeta, event-actor checkpoint state, HITL pause and
resume job metadata in both stores, the Stop path, and branch creation.
A shared Mongoose definition keeps the message and conversation schemas
identical, the zod schema validates entries, and commitAgentEventActorState
rejects malformed maps. Comments no longer describe graph messages as
truncated; only the provider projection is.

Tests cover: tool results and inputs reach the persisted content parts
at full size; contextMeta carries exactly the compact tier and
calibration fields; a persisted snapshot round-trips into the seeded
RunConfig unchanged and prototype-safe; per-agent tiers survive pause,
Redis job round-trips, and event-actor context validation.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WNbgMeR2aZDyhsqAy4L8gF

* 🧊 fix: Restore Event Actor Types and Publish Seeds Before Abortable Setup

The previous commit's doc rewording in the conversation types dropped the
event-actor interfaces that precede the fading tier, which broke the
declaration bundle in CI; they are restored unchanged.

The inherited context meta is now published onto the job as soon as the
parent's state is loaded and awaited at the top of chatCompletion and
resumeCompletion, ahead of run creation and the other abortable setup
stages, so a Stop during setup still persists the parent's tiers onto the
stopped response. The partial response saved when every subscriber
disconnects now carries the job's context meta like the Stop and pause
paths do. The branch route keeps the source's context meta in the saved
message but strips it from the HTTP response, matching client reads.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WNbgMeR2aZDyhsqAy4L8gF

* 🧊 fix: Serialize Distinct Context Meta Publications

Overlapping snapshots from parallel agents could carry different tiers;
each started its own write and both job stores keep whichever write
finished last, so an older snapshot could overwrite a newer one. Each
distinct publication is now issued after the previous one settles, in
call order, so the newest snapshot is what a Stop reads back. Equal
values still share one write.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WNbgMeR2aZDyhsqAy4L8gF

* 🧊 fix: Project Root Message Reads and Keep Inherited Meta on Setup Failure

The root GET /api/messages paths returned raw documents: the
single-message query read called getMessages without a projection and the
cursor page performed an unprojected find. Both now apply
CLIENT_MESSAGE_SELECT, with getMessagesByCursor accepting a select option,
so the persisted fading tiers never reach the client through the
paginated endpoint; the search-hydration read projects the same way.

A turn that loads a parent's context meta but fails before its run exists
used to replace the inherited meta with the capture of a missing run, so
the persisted error response lost the last valid tiers. Both finalizers
now keep the inherited meta when no run was created while a created run's
neutral state may still clear it.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WNbgMeR2aZDyhsqAy4L8gF

* 🧊 refactor: Move Context Meta Publication Into packages/api

The publication coordinator (ordering, deduplication, in-flight sharing,
failure state) now lives in packages/api as createContextMetaPublisher,
with selectRunContextMetaToPublish deciding what a publication carries;
the legacy client keeps only the wiring to the job metadata writer. A
transiently failing write is retried with a short backoff before the
publication is reported and forgotten, so a snapshot's model call no
longer proceeds on the first rejection while the job holds the previous
record. The SDK swallows handler errors, so retrying is the only way to
strengthen the fence without failing the turn.

A terminal save now supplies contextMeta: null when the finished run has
nothing to carry, and saveMessage unsets a previously stored value in
that case, so a disconnect snapshot's record cannot outlive the run that
completed neutrally.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WNbgMeR2aZDyhsqAy4L8gF

* 🧊 fix: Widen the saveMessage Method Type for the Null Unset

The exported method interface still typed contextMeta as the message
field, so the spec exercising the null unset failed typechecking; it now
mirrors the implementation signature.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WNbgMeR2aZDyhsqAy4L8gF

* 🧊 fix: Publish Every Agent's Snapshot, Sanitize Search Hits, Unbound Agent IDs

The context-usage handler now awaits the run's context meta publish for
every ON_CONTEXT_USAGE event, hidden sequential agents included, while
the visible-snapshot bookkeeping and the client emission keep their gate;
a Stop before the next visible snapshot finds the tier a hidden agent's
model call latched. Search results are built from the hydrated hit minus
its server-private contextMeta, since Meilisearch hydration projects every
schema field. The per-agent tier entry guard no longer caps the agent ID
length: ephemeral agent IDs encode endpoint, model and sender without a
bound, and dropping such an entry would let that agent alone re-derive
its tier on the next turn. Also formats the publication spec so the
static checks pass.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WNbgMeR2aZDyhsqAy4L8gF

* 🧊 fix: Unset Context Meta Atomically, Keep Publication State After a Failed Write

- `saveMessage` folds the `contextMeta` unset into the same update that
  persists the terminal response, on both the plain and the provenance-merge
  paths, so a failure between two writes can no longer leave a completed row
  carrying a disconnect snapshot's state.
- `createContextMetaPublisher` tracks whether any record has committed
  separately from the retryable latest publication, so `hasPublished` stays
  true after a later publication exhausts its retries and a following neutral
  snapshot still overwrites the earlier record on the job.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WNbgMeR2aZDyhsqAy4L8gF

* 🧊 fix: Carry Context Meta Through the Resumable Stop Route and Neutral Resumed Completions

- The resumable Stop route copies the refreshed `jobData.contextMeta` onto the
  terminal assistant message it persists, as the abort middleware already
  does, so a stopped resumable response keeps the tiers that produced its
  partial output.
- A resumed run that completes with neutral context state now saves
  `contextMeta: null`, which unsets the paused segment's record instead of
  letting an omitted field keep it on the completed row.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WNbgMeR2aZDyhsqAy4L8gF

* 🧊 fix: Trust Only Server-Authored Context Meta, Unset It on a Neutral Stop, Read It From the Job on Disconnect

- `POST /api/messages/:conversationId` and conversation imports drop any
  client-supplied `contextMeta`, and the run seeds its calibration and fading
  tiers only from a server-authored parent response, so a forged tier can never
  shape a provider projection.
- Both Stop paths save `contextMeta: jobData.contextMeta ?? null`, so a job that
  re-paused with neutral state unsets what an earlier pause stored on the row
  instead of leaving it behind.
- The disconnect partial save reads the same-epoch job record for the run's
  published context meta, since the client-facing resume snapshot never carries
  server-private state.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WNbgMeR2aZDyhsqAy4L8gF

* 🧪 test: Mock the Job Store in the Resume Metadata Spec

The disconnect partial save now reads the same-epoch job record through
`GenerationJobManager.getJobStore()`, so the resume metadata spec's manager
mock provides it as the other disconnect spec already does.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WNbgMeR2aZDyhsqAy4L8gF

* 🧊 fix: Keep Context Meta Off the POST Message Response

`POST /api/messages/:conversationId` returned the saved row verbatim, so a
client write against an existing server-authored message echoed the row's
stored context meta. The route now projects the row through the same client
projection the branch route uses, which drops the server-private field.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WNbgMeR2aZDyhsqAy4L8gF

* 📦 chore: Bump @librechat/agents to 3.7.16 for Persisted Fading Tiers

3.7.16 is the first release that carries danny-avila/agents#497, so
`Run.getFadingTier()` / `getFadingTiers()` now report the latched tier
this branch persists and `RunConfig.fadingTier` / `fadingTiers` seed the
next run from it. The dependency set is unchanged from 3.7.15; only the
package entry moves.

Adds an SDK-backed round trip to the fading spec: a first pruner latches
an informative tier, `resolveRunContextMeta` reduces it to the persisted
shape, and a second pruner seeded from that shape reproduces the first
run's projected tool-exchange bytes under instruction and calibration
drift, while an unseeded pruner under the same drift relaxes its tier and
rewrites them.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WNbgMeR2aZDyhsqAy4L8gF

---------

Co-authored-by: Claude <noreply@anthropic.com>
2026-09-02 20:29:43 -04:00
Danny Avila
790f2b3153
⚡ feat: Add Gemini 3.8 Flash Support (#15519)
Some checks failed
Backend Unit Tests / Build packages (push) Waiting to run
Backend Unit Tests / Codegraph select (push) Waiting to run
Backend Unit Tests / TypeScript type checks (push) Blocked by required conditions
Backend Unit Tests / Circular dependency checks (push) Waiting to run
Backend Unit Tests / Tests: api (shard 1/3) (push) Blocked by required conditions
Backend Unit Tests / Tests: api (shard 2/3) (push) Blocked by required conditions
Backend Unit Tests / Tests: api (shard 3/3) (push) Blocked by required conditions
Backend Unit Tests / Tests: data-provider (push) Blocked by required conditions
Backend Unit Tests / Tests: data-schemas (push) Blocked by required conditions
Backend Unit Tests / Tests: @librechat/api (shard 1/4) (push) Blocked by required conditions
Backend Unit Tests / Tests: @librechat/api (shard 2/4) (push) Blocked by required conditions
Backend Unit Tests / Tests: @librechat/api (shard 3/4) (push) Blocked by required conditions
Backend Unit Tests / Tests: @librechat/api (shard 4/4) (push) Blocked by required conditions
Codegraph E2E Votes / vote (full suite) (push) Waiting to run
Docker Dev Branch Images Build / publish (push) Waiting to run
Frontend Unit Tests / Codegraph select (push) Waiting to run
Frontend Unit Tests / Build packages (push) Waiting to run
Frontend Unit Tests / TypeScript type checks (client) (push) Blocked by required conditions
Frontend Unit Tests / Tests: @librechat/client (push) Blocked by required conditions
Frontend Unit Tests / Tests: Ubuntu (shard 1/2) (push) Blocked by required conditions
Frontend Unit Tests / Tests: Ubuntu (shard 2/2) (push) Blocked by required conditions
Frontend Unit Tests / Vite build verification (push) Blocked by required conditions
Sync Locize Translations & Create Translation PR / Sync Translation Keys with Locize (push) Has been cancelled
Sync Locize Translations & Create Translation PR / Create Translation PR on Version Published (push) Has been cancelled
* ⚡ feat: Add Gemini 3.8 Flash Support

Adds first-class support for Google's Gemini 3.8 Flash (`gemini-3.8-flash`,
GA 2026-09-02) for both the Gemini API (AI Studio) and Google Cloud Gemini
Enterprise Agent Platform, following the Gemini 3.7 Flash integration (#14818).

The Flash-family handler that PR generalized already carries the shape 3.8
needs, so registering the model is a one-line rule rather than new behavior:
it inherits the strip of deprecated sampling params (temperature/topP/topK),
the rejected penalty params, and `thinkingBudget`, defaults to `medium`
thinking, and — like 3.7 — errors on `minimal`, so an explicit `minimal` is
substituted with the nearest supported level, `low`.

- Context window (1,048,576) in googleModels; API + cache pricing in tx.ts
- Model dropdown (config.ts) and GOOGLE_MODELS examples for both integrations
- Register the model in the Flash-family thinking-rule table
- Apply the same introductory pricing as 3.6/3.7 Flash ($0.75 in / $3.75 out /
  $0.075 cached, per 1M), reverting to $1.50 / $7.50 / $0.15 on 2027-01-01;
  the existing comments at both call sites now name 3.8 alongside 3.6/3.7
- Tests mirroring the 3.7 Flash coverage: thinking default, legacy-param strip,
  explicit level pass-through, `minimal` substitution, versioned aliases,
  context window, model-key mapping, and rates

Refs #15516

Ref: https://ai.google.dev/gemini-api/docs/models/gemini-3.8-flash
Ref: https://ai.google.dev/gemini-api/docs/pricing

* 🔧 chore: Update @librechat/agents dependency to version 3.7.15 in package-lock.json and related package.json files
2026-09-02 15:06:40 -04:00
Danny Avila
d5b2a8580e
📖 feat: Add Claude Fable 5.1 Support (#15512)
Some checks are pending
Docker Dev Images Build / publish (push) Waiting to run
Sync Helm Chart Tags / Ignore non-main push (push) Waiting to run
Sync Helm Chart Tags / Sync chart tags (push) Waiting to run
Claude Fable 5.1 (`claude-fable-5-1`, GA 2026-09-01) succeeds Claude Fable 5
at the same input/output prices, with a 1M context window and 128K max output.
The Mythos-class helpers added for Fable 5 already match it — `isMythosClassModel`
tests `claude-(?:fable|mythos)[-.]?\d`, so adaptive thinking, summarized-display
opt-in, sampling omission, 1M context, the 128K output clamp, prompt caching,
the Vertex multi-region gate, and the Bedrock PDF exemption all cover 5.1
without a code change. What was missing was registration and pricing.

Cache reads are the one rate that differs, and the substring matcher would have
silently gotten it wrong: `claude-fable-5-1` falls through to the `claude-fable-5`
key, which prices cache hits at the usual 0.1x base input ($1/MTok). Fable 5.1
and Mythos 5.1 price them at 0.025x ($0.25/MTok), so every cache read would have
billed 4x — on exactly the long agentic sessions this model is built for.

- Register `claude-fable-5-1` in the shared Anthropic list and
  `global.anthropic.claude-fable-5-1` in the Bedrock list
- Add 1M context / 128K output entries for `claude-fable-5-1` and
  `claude-mythos-5-1`
- Add $10/$50 rates and `{ write: 12.5, read: 0.25 }` cache rates for both
- Update the `.env.example` and Vertex `librechat.example.yaml` examples
- Test that 5.1 resolves to its own key rather than collapsing onto 5.0, across
  token maps, pricing, cache rates, and Mythos-class detection

The three documented breaking changes from Fable 5 need no handling here:
LibreChat never sends a forced `tool_choice` on the Anthropic path (the agents
SDK omits the param when unset, so the API default `auto` applies); thinking
blocks an earlier model can't read are dropped server-side, unbilled; and
history is rebuilt from stored content parts rather than replayed signed
thinking blocks, so the prefix-binding check has nothing to reject.

Refs #15506
2026-09-02 10:45:54 -04:00
Danny Avila
1c7a172321
🕹️ feat: Configure BYOM Tool Permissions (#15504)
* feat: add configurable BYOM permissions

* test: cover BYOM permission settings live

* fix: harden BYOM permission settings

* perf: reuse principals for environment discovery

* chore: normalize BYOM settings style

* fix: Align BYOM Discovery and Admission

* fix: Harden BYOM Permission Boundaries

* fix: Align BYOM Skill Approval Routing

* fix: Preserve Lazy BYOM Approval Capabilities

* perf: Parallelize Cached Environment Reads
2026-09-02 10:38:00 -04:00
csdivad
8a40e9f675
🔄 fix: Reindex MeiliSearch Documents After Indexed-Projection Changes (#15484)
* 🔄 fix: Reindex MeiliSearch Documents After Indexed-Projection Changes

- Introduce _meiliIndexSchemaVersion schema field and MEILI_INDEX_SCHEMA_VERSION constant
- Stamp schema version on single-document and batch index writes
- Query unindexed and stale-version documents in syncWithMeili
- Count stale documents in pendingIndexing in getSyncProgress
- Add spec coverage for reindexing stale-version documents

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* 🔄 test: use the registered Meili conversation model

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* 🔄 style: satisfy Prettier in Meili indexing progress counts

---------

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Danny Avila <danny@librechat.ai>
2026-09-01 21:19:22 -04:00
csdivad
ce704f5c11
🔍 fix: Include Message-Matched Conversations in Sidebar Search (#15483)
* 🔍 fix: Include Message-Matched Conversations in Sidebar Search

- Union conversation and message index matches in getConvosByCursor
- Type searchMessages with SearchParams and SchemaWithMeiliMethods
- Inject searchMessages into createConversationMethods
- Bound sidebar Meili queries with 1000 limit and attributesToRetrieve
- Fall back to title-only matches when message search fails or is uninjected
- Escape filter user values in search queries

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* fix(search): preserve original conversation keys in sidebar hits

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* 🔍 fix: reuse the loaded conversation model

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* 🔍 fix: type the loaded conversation search model

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

---------

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-09-01 21:17:11 -04:00
Danny Avila
164f146bdf
🛞 feat: Manage Personal Code Environment Lifecycles (#15369)
* feat: add self-service code worker pairing

* fix: enforce code worker principal boundaries

* fix: separate pairing control planes from execution

* fix: preserve principal worker routing

* fix: pin artifact traffic to code workers

* fix: authenticate code worker selection

* test: mock worker-aware artifact headers

* fix: harden self-service code worker routing

* fix: fail closed for principal code pairing

* feat: manage personal code environments

* fix: harden personal code environment lifecycle

* fix: satisfy backend lifecycle types

* fix: harden principal environment lifecycle and quotas

* fix: fence environment lifecycle races

* fix: reconcile code environment stack

* style: format code execution headers

* fix: reconcile code environment lifecycle

* fix: close self-service pairing gaps

* fix: close code environment reconciliation gaps

* fix: close code environment UI review findings

* fix: satisfy code environment type checks

* fix: route code priming to selected worker

* fix: harden code environment lifecycle fences

* fix: type lease renewal interval

* fix: harden environment lifecycle recovery

* fix: close lifecycle recovery races

* fix: type compensated agent timestamps

* fix: fence lifecycle cleanup by tenant and revision

* fix: scope environment revision fencing
2026-09-01 21:10:05 -04:00
csdivad
0f9667b391
🔎 fix: Respect Search Page Limits in MeiliSearch Queries (#15482)
* fix(api): align message search with page size

Pass a positive, capped page size to MeiliSearch and use the same conversation limit so requested message search pages are not silently truncated.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* fix(data-schemas): expand shared-link search results

Request the MeiliSearch result ceiling using only conversationId so cursor pagination is not limited to the default 20 hits or burdened with unused fields.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

---------

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-09-01 21:05:25 -04:00
Danny Avila
51edb2a660
🛫 feat: Enable Self-Service Code Worker Pairing (#15367)
* feat: add self-service code worker pairing

* fix: enforce code worker principal boundaries

* fix: separate pairing control planes from execution

* fix: preserve principal worker routing

* fix: pin artifact traffic to code workers

* fix: authenticate code worker selection

* test: mock worker-aware artifact headers

* fix: harden self-service code worker routing

* fix: fail closed for principal code pairing

* style: format code execution headers

* fix: close self-service pairing gaps

* fix: satisfy code environment type checks
2026-09-01 18:46:33 -04:00
Danny Avila
aa7539909b
🪶 feat: Enable Standalone Skill Authoring (#15480)
* feat: enable standalone skill authoring

* test: align skill e2e flow with enable toggle

* fix: preserve legacy skill state on version restore
2026-09-01 18:39:05 -04:00
Danny Avila
67e2107d74
🔐 fix: Keep OpenID Metadata Out of Logs and Traces (#15470)
Some checks are pending
Backend Unit Tests / Build packages (push) Waiting to run
Backend Unit Tests / Codegraph select (push) Waiting to run
Backend Unit Tests / TypeScript type checks (push) Blocked by required conditions
Backend Unit Tests / Circular dependency checks (push) Waiting to run
Backend Unit Tests / Tests: api (shard 1/3) (push) Blocked by required conditions
Backend Unit Tests / Tests: api (shard 2/3) (push) Blocked by required conditions
Backend Unit Tests / Tests: api (shard 3/3) (push) Blocked by required conditions
Backend Unit Tests / Tests: data-provider (push) Blocked by required conditions
Backend Unit Tests / Tests: data-schemas (push) Blocked by required conditions
Backend Unit Tests / Tests: @librechat/api (shard 1/4) (push) Blocked by required conditions
Backend Unit Tests / Tests: @librechat/api (shard 2/4) (push) Blocked by required conditions
Backend Unit Tests / Tests: @librechat/api (shard 3/4) (push) Blocked by required conditions
Backend Unit Tests / Tests: @librechat/api (shard 4/4) (push) Blocked by required conditions
Codegraph E2E Votes / vote (full suite) (push) Waiting to run
Docker Dev Branch Images Build / publish (push) Waiting to run
Frontend Unit Tests / Codegraph select (push) Waiting to run
Frontend Unit Tests / Build packages (push) Waiting to run
Frontend Unit Tests / TypeScript type checks (client) (push) Blocked by required conditions
Frontend Unit Tests / Tests: @librechat/client (push) Blocked by required conditions
Frontend Unit Tests / Tests: Ubuntu (shard 1/2) (push) Blocked by required conditions
Frontend Unit Tests / Tests: Ubuntu (shard 2/2) (push) Blocked by required conditions
Frontend Unit Tests / Vite build verification (push) Blocked by required conditions
* 🔒 fix: Keep Secrets Out of Debug Logs and Spans

* 🧯 fix: Derive Safe Exception Types

* 🛡️ fix: Close Diagnostic Redaction Gaps

* 🧼 fix: Redact Escaped JSON Secrets
2026-09-01 13:22:12 -04:00
Danny Avila
f0f82ec5fa
🧷 fix: Fence Server-Admitted Queue Boundaries (#15393)
* fix: fence server-admitted queue boundaries

* fix: preserve effective queue admission boundaries

* fix: fence queue recovery predecessor

* fix: unify queued boundary reconciliation

* fix: make queued turn receipt settlement monotonic

* fix: narrow queued receipt projection status

* fix: close queued turn settlement races

* fix: make queued admission lineage clock independent

* fix: centralize queued turn admission ownership

* test: type queued admission outcomes

* fix: make queued admission identity durable

* fix: carry queued admission identity end to end

* style: sort queued turn test imports

* fix: unify indeterminate admission ownership

* fix: migrate queued admission fence state

* fix: migrate every queued admission owner

* fix: surface queued admission reconciliation

* fix: preserve queued receipt evidence

* test: type queued receipt regression

* fix: stabilize queued receipt authority

* fix: supersede initial receipt fetches
2026-08-31 12:31:07 -04:00
Danny Avila
81338cef95
🍃 test: Sweep Every DB Schema Method Against a Live Engine (#15392)
* 🍃 test: Sweep Every data-schemas Method Against a Live Engine

Adds `misc/documentdb/sweep.documentdb.spec.ts`: drives all 510 exported
data-schemas methods against a real engine and records, per method, how many
driver queries it issued and whether the engine rejected any of them.

Every DocumentDB incompatibility found so far — pipeline updates, `$$REMOVE`,
`$facet`, the transaction-probe false negative, the Mongoose-compiled mixed
projection — was invisible until someone thought to look for its class. The
sweep inverts that: the engine adjudicates whatever each method emits, known
class or not. Query shapes are parsed before they are matched, so an empty
database adjudicates most entry paths.

Mechanics: arguments are synthesized from each function's parameter names,
then repaired adaptively from the validation errors they provoke (ObjectId and
number casts, required schema paths, object-parameter shapes) until the method
reaches the database; the driver prototype is instrumented to count calls and
capture server errors — including on cursor consumption, where parse errors
actually surface — and duplicate-key violations are classified as benign. A
method that never issues a query is reported un-adjudicated instead of
silently green: coverage is visible, not assumed.

Two modes with identical mechanics so matrices diff cleanly: in-memory MongoDB
(`SWEEP_BASELINE=true`, free, CI-able) and live DocumentDB (`DOCUMENTDB_URI`).
`SWEEP_REPORT_PATH` writes JSON for machine diffing;
`DOCUMENTDB_STRICT=true` turns rejections into failures.

First full run against live DocumentDB 5.0.0: 363 of 510 methods issued at
least one query, zero engine rejections, zero divergences from the MongoDB
baseline. The jest config gains the main config's transform-ignore patterns so
the full method bundle (which pulls ESM markdown tooling) loads.

* 🍃 fix: Adjudicate Index DDL, Real ACL Cascades, and a Run-Scoped Sweep Database

Review round 1, all four findings real:

- The driver instrumentation now wraps `createIndex`, `createIndexes`,
  `dropIndex`, `dropIndexes`, and `listIndexes` (cursor-wrapped), so an engine
  rejecting index DDL — which `recordAuditEntry` issues before any listed
  operation and then swallows — is adjudicated instead of surfacing as a
  false `no-queries` under STRICT.
- The `removeAllPermissions` no-op override is gone: `createMethods` supplies
  a real implementation delegating to `deleteAclEntries`, so driven deletion
  cascades exercise their ACL operations (not-driven drops 147 → 142).
- Each race timer is cleared when its attempt settles, instead of hundreds of
  referenced timers living until the shared deadline and the last one holding
  the process open after the report.
- The live run connects to a run-scoped database
  (`librechat_sweep_<runId>`), so its `dropDatabase` teardown can never erase
  fixtures out from under the sibling live suites sharing the URI, nor any
  retained state in the URI's own database.

* 🍃 fix: Attribute Straggler Queries by Async Context and Adjudicate Transaction Commits

Review round 2, all three findings real:

- Per-method attribution now rides AsyncLocalStorage instead of a module
  global: a method that outlives its timeout keeps recording queries and
  engine errors under ITS OWN row rather than corrupting whichever row the
  sweep advanced to, and a final reclassification pass upgrades any row whose
  engine error arrived after its outcome was assigned.
- The session prototype's `commitTransaction`/`abortTransaction`/
  `withTransaction` are instrumented: transaction-lifecycle rejections surface
  from the session, not a collection operation, and several methods convert
  them into domain errors — previously a real commit rejection classified as
  a mere `threw` and STRICT passed.
- The compatibility doc's published numbers now come from the
  corrected-harness baseline (368 driven / 142 un-driven) and state explicitly
  that the recorded live-run matrix predates the instrumentation widening, so
  the next live run is authoritative for the newly adjudicated surfaces.

* 🍃 fix: Settle Stragglers Before STRICT and Commit a Real Transaction on a Replica-Set Baseline

Review round 3, all four findings real:

- STRICT is now enforced against the FINALIZED rows: `afterAll` waits
  (bounded) for every timed-out invocation to settle, reclassifies rows whose
  engine errors arrived late, and asserts zero rejections — previously a
  straggler's rejection could arrive after its per-test assertion had already
  passed against the `timeout` outcome.
- The baseline runs on a single-node replica set with the live branch's exact
  connection options (`autoIndex`/`autoCreate` off, run-scoped `dbName`):
  a standalone baseline would reject driven transactions for topology reasons,
  and asymmetric model auto-creation would attribute DDL in one mode and not
  the other — both manufacturing false matrix divergences.
- `resolveMCPAuthorityProof` gets a seeded case (fixture ported from the
  compat suite, executed in tenant context via the new `within` wrapper on
  overrides): generic synthesis died on target validation before a session
  ever started, so the session instrumentation adjudicated nothing. It now
  drives 22 operations through the production path, transaction commit
  included, on both engines.

* 🍃 docs: Sync the Published Baseline Counts With the Seeded Harness

* 🍃 fix: Materialize Transaction-Read Collections and Settle Stragglers Before Teardown

Review round 4, three findings real (the fourth — stale doc totals — was
already fixed on the pushed head):

- Straggler settlement moved BEFORE engine teardown: dropping the database or
  closing the client would abort outstanding work with closed-client errors
  that are not engine verdicts, leaving a timed-out row unadjudicated while
  STRICT passes.
- The authority fixtures (sweep AND compat) materialize PluginAuth and Token
  before the transaction: `loadCurrentProof` reads both inside it, and
  DocumentDB rejects any in-transaction statement touching a non-existent
  collection — the rejection surfaces as `proof_unavailable` before the commit
  is reached. This is the mechanism behind the compat authority test failing
  on every live cluster run while passing against MongoDB; with the
  collections materialized, the next live run adjudicates the transaction
  itself instead of the missing namespace.
- Engine errors are recognized by `instanceof MongoServerError` instead of a
  `name` equality, so server-originated subclasses (MongoWriteConcernError,
  MongoBulkWriteError) can no longer slip through the session wrapper as
  domain throws.

* 🍃 fix: Materialize Group and Config for the Authority Transaction and Exclude Constructor Exports

Review round 5, both findings real. The proof transaction also reads Group and
Config; the sweep fixture materializes both (the compat fixture already
creates a Config record and now materializes Group). SessionError — an error
class exported alongside the session methods — is excluded from the method
map, correcting the totals to 509 methods / 369 driven / 140 un-driven; the
published baseline is synced in the same commit.

* 🍃 fix: Build a Fresh Method Bundle per Case and Emit a Normalized Matrix

Review round 6, both findings real:

- Every row now invokes a freshly constructed method bundle. Factory-local
  caches leaked across alphabetically ordered rows — the session methods cache
  `sessionIndexesPromise`, so `ensureSessionIndexes` returned that settled
  promise and was misreported as `no-queries`. It now issues its 3 index
  operations, making the matrix order-independent (370 driven / 139 un-driven).

- The report separates run metadata from a canonical `rows` payload with
  run-specific values normalized out (run id, generated ObjectIds and UUIDs),
  so the two engines' matrices diff cleanly instead of differing on every row
  that embeds a synthesized value in its error text.

* 🍃 fix: Materialize Authority Snapshot Namespaces Before Its Transaction

Amazon DocumentDB rejects any statement inside a transaction that touches a
collection which does not exist, and `asMCPError` converts that server
rejection into `proof_unavailable` — so on a deployment where no PluginAuth
or Token row has ever been written, every MCP authority proof fails with an
error naming nothing about the real cause. This is the mechanism behind the
live-harness authority probe failing on every real cluster run since July
while passing against MongoDB.

`loadAuthoritativeSnapshot` now materializes the nine namespaces its snapshot
reads before opening the transaction, once per process. MongoDB tolerates the
in-transaction read either way, so nothing changes there; a create failure is
logged at debug and left to surface from the transaction itself.

The regression test drops the credential namespaces to reproduce the state and
asserts the resolve recreates them and succeeds; it fails without the fix.

* 🍃 docs: Record the Authoritative Hardened Live Run

The hardened harness ran against a live DocumentDB 5.0.0 cluster: 370 of 509
methods drove at least one query, zero engine rejections, and the matrix is
byte-identical to the MongoDB baseline — every outcome and query count
matches. This run adjudicated index DDL, transaction commits and aborts, ACL
cascades, and the authority snapshot transaction, none of which the first
pre-hardening run could see.

The same run turned the compatibility suite's MCP authority probe green for
the first time on any real cluster, confirming the namespace-materialization
diagnosis for the `proof_unavailable` failures seen since July.

* 🍃 fix: Retry Snapshot Namespace Creation After a Transient Failure

The catch suppressed every `createCollection` error, so a transient failure
settled the once-per-process memo as successful: creation was skipped for the
rest of the process, the transaction kept failing on the missing collection,
and every later proof returned `proof_unavailable` even after the condition
cleared. The existing reset-on-reject path could never fire because the inner
promise never rejected.

Only `NamespaceExists` (including a concurrent creator winning the race) is
suppressed now; every other failure propagates, clears the memo, and lets the
next request retry. The regression test rejects the first creation with a
transient error, asserts the proof fails, then asserts the following request
recreates the namespace and succeeds — it fails against the swallow-all form.

* 🍃 fix: Throttle Snapshot Namespace Preflight Retries With a Bounded Cooldown

Clearing the memo on every rejection made a persistent failure — an
authorization error, say — turn each authority request into another serial
collection preflight: a DDL retry storm precisely while the database is
unhealthy, since the single-flight promise only coalesces requests while an
attempt is pending.

A failure is now recorded with its timestamp and replayed without touching the
database until a bounded cooldown elapses (30s, overridable through the
existing hooks for tests). Transient failures stay retryable, just not on
every request. Both sides of the boundary are covered: one test asserts a
persistent failure issues no further creations inside the cooldown, and the
existing retry test runs with the cooldown disabled to assert recovery still
works. The throttle test fails with the cooldown gate removed.

* 🍃 fix: Instrument Db-Level Namespace DDL in the Sweep

The authority path's namespace preflight calls `Model.createCollection()`,
which delegates to `Db.createCollection` — not to any `Collection` operation
the sweep wrapped. A DocumentDB rejection of that DDL would therefore be
laundered by `asMCPError` into `proof_unavailable`, leaving the row
classified as a mere `threw` while `DOCUMENTDB_STRICT=true` passed: the exact
false-green class this suite exists to close, reintroduced by the preflight
itself.

`Db.createCollection`/`dropCollection`/`renameCollection` are now
instrumented, and `NamespaceExists` (48) joins the benign server codes since
an idempotent preflight against an existing collection is the expected
outcome, not an incompatibility. The authority row's adjudicated query count
rises from 22 to 31, which is those nine preflight operations becoming
visible.
2026-08-31 09:54:32 -04:00
Danny Avila
14d4f2789d
🍃 fix: Send a Pure Inclusion Projection From the Legacy Receipt Sweep (#15390)
Some checks are pending
Docker Dev Images Build / build (Dockerfile, librechat-dev, node) (push) Waiting to run
Docker Dev Images Build / build (Dockerfile.multi, librechat-dev-api, api-build) (push) Waiting to run
Sync Helm Chart Tags / Ignore non-main push (push) Waiting to run
Sync Helm Chart Tags / Sync chart tags (push) Waiting to run
* 🍃 fix: Send a Pure Inclusion Projection From the Legacy Receipt Sweep

The legacy actor-receipt sweep selected its candidates with
`.select('_id +agentEventActorReconciliations')`. `_id` alone does not make a
Mongoose projection inclusive and a `+` token only un-hides its field, so this
compiles to `{ _id: 1 }` plus a `: 0` exclusion for every other
`select: false` sibling — a mixed projection MongoDB tolerates via the `_id`
exception but Amazon DocumentDB rejects (`Projections cannot have a mix of
inclusion and exclusion`). The sweep failed on every 30-second maintenance
pass, receipts never expired, and the rejection took the whole Promise.all
down with it, so the sequenced lane reclamation never ran either.

- The candidate read uses the object form `{ _id: 1,
  agentEventActorReconciliations: 1 }` — a pure inclusion that still returns
  the hidden field and fetches nothing extra. A regression test spies on the
  driver call and asserts the compiled projection contains no exclusions.
- Every maintenance step now fails alone: a rejection is logged with its step
  name and counted as zero progress instead of aborting the pass, and lane
  reclamation runs regardless. Covered by a test rejecting one step and
  asserting reclamation still runs.
- The static guard gains a detector for the exact broken shape —
  `select('_id +field')` — with fixtures for both directions. The shape is
  narrow on purpose: probing real Mongoose shows a bare non-`_id` token makes
  the projection inclusive and turns `+` tokens into plain `field: 1`
  inclusions, so the repo's nine other `+`-selects are all legal and stay
  unflagged. The live audit suite gains a raw mixed-projection probe and
  drives the sweep itself as production shape number eight.

* 🍃 fix: Gate Lane Reclamation on Batch Recovery and Type the Projection Assertion

Review round 1, both findings real:

- Batch-receipt recovery is a prerequisite of lane reclamation, not an
  independent step: reclamation consumes the lane-cleanup markers
  (`fulfillLaneCleanupRequest` unsets the delivery-side marker, and
  `reclaimLaneIfInactive` clears `cleanupRequestedAt` when the lane is still
  retained), so running it against a half-recovered batch erases a request no
  later successful recovery can re-arm, retaining the lane permanently. The
  blanket isolation reintroduced exactly that; reclamation now waits for a
  batch-recovery pass that did not fail, while the genuinely independent steps
  keep failing alone. Covered by a test rejecting batch recovery and asserting
  reclamation is withheld.

- The captured-projection assertion drops its `Record<string, unknown>` cast
  and asserts deep equality with the exact expected pure-inclusion projection,
  which is both typed and strictly stronger.
2026-08-30 22:10:44 -04:00
Danny Avila
c93bd7ec9d
🪟 fix: Keep the Subagent Panel Steady and Honest Across Event Deliveries (#15387)
* 🪟 fix: Keep the Subagent Panel Steady and Honest Across Event Deliveries

* 🧮 fix: Budget the Activity Projection at the Database Seam and Scope Retained Views

* 🧮 fix: Charge Label Associations to the Budget and Scope History Flags

* 🧮 fix: Gate Mixed Activity Metadata and Keep the Budgeted Suffix Contiguous

* 🧵 fix: Render Only the Active Thread's Retained Turns
2026-08-30 21:58:44 -04:00
Danny Avila
c7f0a8f247
📬 feat: Move Agent Queued Turns Server-Side (#15370)
* feat: move agent queued turns server-side

* test: update queued turn composition coverage

* fix: harden queued-turn lifecycle ownership

* fix: centralize queued-turn lifecycle ownership

* fix: close queued-turn lifecycle gaps

* fix: unify queued turn lifecycle transitions

* fix: fence queued-turn deletion admission

* fix: preserve queued-turn lifecycle ownership

* fix: close queued-turn reconciliation races

* fix: reconcile quarantined queued-turn admissions

* fix: annotate admission evidence default

* fix: deepen queued turn lifecycle admission

* fix: fence queued turn provider admission
2026-08-30 21:53:58 -04:00
Danny Avila
fcae1025c0
🧳 feat: Register Principal-Owned Code Environments (#15365)
* feat: add principal-owned code environments

* fix: address code environment CI coverage

* fix: expose principal code environments in endpoint config

* fix: reject missing code environment bodies

* fix: harden code environment lifecycle

* fix: revalidate principal code environments

* fix: synchronize code environment authorization

* fix: bind code environments to current principals

* fix: fail closed on code ACL cache errors

* fix: prevent code environment override shadowing

* fix: preserve principal environment defaults

* fix: suppress revoked code environment aliases

* fix: fail closed on environment augmentation

* fix: narrow code environment defaults

* fix: isolate code environment fallback

* style: sort code config imports
2026-08-30 19:33:19 -04:00
Danny Avila
9684b2478f
🍃 fix: Restore Amazon DocumentDB Compatibility Across Trigger and Subagent Writes (#15375)
* 🍃 fix: Restore Amazon DocumentDB Compatibility Across Trigger and Subagent Writes

The README advertises Amazon DocumentDB 5.0+ support, but six sites added
between 2026-07-29 and 2026-08-30 reintroduced constructs the engine rejects,
and the transaction probe silently reported a false negative. All seven are
adjudicated against a live DocumentDB 5.0.0 cluster, not against AWS's
omission-based documentation.

Rejected by every engine version, now rewritten with supported operators:

- `claimNextAgentTriggerDelivery` used a pipeline-form update. It reads the
  globally oldest candidate under the same sort, then claims that exact `_id`
  and lifecycle state with a compare-and-swap. Ownership is still granted only
  by the fenced write. An empty queue costs the one read it costs today; a
  successful claim costs one additional round trip.
- `renewAgentTriggerDeliveryProducerLease` used a `$cond` to keep the later
  deadline. `$max` is the exact classic-operator form and evaluates no
  expression.
- `claimBackgroundToolResults` and `releaseBackgroundToolResultClaims` rewrote
  the entire `content` array through `$map`. The filtered positional operator
  addresses the matching parts directly, so each write touches only those parts
  and still needs no read-modify-write.
- The bounded subagent thread-view projections used `$$REMOVE` to omit keys.
  They now emit `null` and are pruned back to absent inside the pass that
  already materializes each record, preserving the absent-vs-present contract.
- The selected-task read used `$facet`. It is now two concurrent reads, which
  also removes the 16 MiB single-document risk the surrounding code already
  avoids for the same reason.

`supportsTransactions` probed a canary collection that does not exist, and
DocumentDB rejects a transaction touching a non-existent collection. The probe
returned `false` on an engine that fully supports transactions, so every caller
silently took the non-transactional path. It now materializes the canary first,
once per process.

Adds `documentdb.spec.ts`, a static guard over the whole class, because no
runtime test can catch it: every suite runs `mongodb-memory-server`, which is
real MongoDB and accepts all of these constructs. Adds
`misc/documentdb/audit.documentdb.spec.ts` to re-adjudicate against a real
cluster, and records the live verdicts and connection requirements in the
compatibility assessment.

* 🍃 fix: Address Review — CAS Exhaustion, AST Guard, Single-Snapshot Selected Read

Addresses all three review findings, and the strengthened guard immediately
found a seventh incompatible site the original character-level scan missed.

- Exhausting the claim's compare-and-swap budget now throws instead of
  returning null. Every lost round read a claimable row, so the queue is not
  empty; null is the engine's confirmed-empty signal and advances its idle
  backoff, which would leave queued work waiting. A throw lands on the engine's
  retryable claim-failure path, which holds the polling cadence. Covered by a
  new test asserting the bounded budget and that the loser owns nothing.

- The static guard now walks the TypeScript AST: it catches pipelines bound to
  a variable first, cast with `as`, chosen by a conditional, or carried in a
  `bulkWrite` operation's `update` property (including shorthand). Every shape
  is proven against fixtures in the suite itself, so the detectors can never
  silently go inert. The stronger scan flagged `updateToolCallResult`, whose
  pipeline lived in a `stages` variable: the backgrounded tool-call settle path
  was still broken on DocumentDB. Rewritten with the filtered positional
  operator — subfield sets preserve an existing `resultClaim` without per-part
  branching, an explicit `$unset` keeps the old whole-object replacement's
  wakeup-disarm semantics, and the attachments merge becomes `$pull` plus
  `$push` (two atomic writes that converge under the retry the claim fencing
  already provides). New tests lock the disarm and supplied-claim semantics,
  and the live audit suite now drives this method too.

- The selected-task read is a single aggregation again: each selected row
  yields both its bounded message fields and its source payload from the same
  document version, restoring the `$facet` snapshot guarantee the concurrent
  split lost, and returning the endpoint to three reads.

* 🍃 fix: Close Settle Atomicity Windows and Spread Claim Contention

Self-review pass (eight independent review angles over the branch diff).

Correctness:

- The attachments merge is a single fenced write again: read the array, merge
  with the exact semantics the old `$filter`/`$concatArrays` had, and write it
  back with the settle patch in one update guarded on the array being unchanged
  — the same compare-and-swap this file already uses for control receipts. The
  `$pull`-then-`$push` split could strip attachments on a failure between the
  writes (the throw path retires the delivery, so nothing healed it) and
  duplicated them under concurrent re-application.
- A supplied `resultClaim` is stamped BEFORE the settle write instead of after
  it. The claim-last order left a terminal, claimable, unclaimed part visible
  to the ~1s wakeup poll, which could acquire and deliver a result a manual
  poll had already delivered; claim-first leaves only inert intermediate states
  (a claim on a part that is not yet terminal admits no consumer). The stamp's
  guard is `resultClaim: null`, restoring the old `$ifNull` treatment of a
  stored null.
- `releaseBackgroundToolResultClaims` filters on `content: { $type: 'array' }`:
  the filtered positional operator errors on a row without the array where the
  old pipeline no-op'd, and that throw surfaced as retryable release failures
  that left claims stamped forever.
- The claim loop reads a BATCH of candidates per round, so the engine's
  parallel claimers spread across the batch instead of convoying on one
  head-of-queue row; exhaustion now throws the typed
  `AgentTriggerClaimContentionError`. The dead-capability adoption fence also
  pins `availableAt`, which the write copies.
- The transaction probe probes first and creates the canary only after a
  failure — plain MongoDB never pays the DDL or gains the collection — logs a
  failed create instead of swallowing it, and concurrent first probes share one
  in-flight promise.

Structure and guard:

- One `partScope(prefix)` builder yields the `$elemMatch` document filter and
  the arrayFilters element filter from a single predicate, deleting the
  aggregation `$expr` twin, and the agent-ownership rule shared by settling and
  claiming lives in one `agentOwnershipFilter` helper.
- The static guard scans all three backend workspaces (`packages/data-schemas`,
  `packages/api`, `api`), parses each file once, and shares the tree between
  both detectors.
- The audit suite asserts each probe's returned verdict, the compat suite's run
  recipe carries the connection parameters the audit header documents, and the
  `$facet` pipeline assertion covers every captured aggregation in one loop.

* 🍃 fix: Keep Attachment Contention Retryable and the Selected Snapshot Authoritative

Second codex round, both findings real:

- Exhausting the attachment-merge fence now returns an unmatched result with a
  warn log instead of throwing. The settle retry loop treats unmatched as
  retry-then-heal-later and degrades to false; a throw escaped it onto
  ambiguous-failure handling that can retire the completion, losing a healthy
  completed tool result to mere contention. Locked by a test asserting the
  bounded rounds resolve to `{ matched: false }` rather than a rejection.

- When the selected task's rows also appear in the normal page window, the
  dedup pass kept the page copy — read by an independent concurrent aggregate —
  and discarded the single-snapshot version, letting a page row claim a
  transcript that `selectedSources` does not carry. Duplicates are now replaced
  in place with the snapshot version (page order kept), so the selected
  message/source pair stays per-document coherent end to end.

* 🍃 fix: Treat a Stored-Null Claim Stamp as Unclaimed in the Claim Fence

Third codex round. The subfield-preserving settle write keeps a persisted
`resultClaim: null` that the old whole-object rewrite dropped, and the claim
path's outer `$elemMatch` fence still required the field to be absent via
`$exists: false`. The in-memory claimable scan, the claim arrayFilters, and the
settle stamp all already treat a null stamp as unclaimed, so a part carrying
one became terminal but permanently unclaimable: every manual poll and wakeup
read `ready` in memory, missed the fence, and returned `not_ready` forever.

The fence now matches missing-or-null like every other layer. The regression
test persists an actual null stamp — asserted present before claiming, so the
test cannot pass vacuously — and claims it.
2026-08-30 18:33:25 -04:00
Danny Avila
a9ccac8656
🧲 feat: Enable Secure Attached Environment Pairing (#15355)
* feat: add secure code environment pairing

* fix: satisfy code environment type checks

* fix: secure code environment administration

* fix: isolate code pairing control plane

* fix: validate code pairing control responses

* fix: secure code pairing transport

* fix: validate code pairing wire format

* fix: harden pairing secret lookup
2026-08-30 17:12:20 -04:00
Danny Avila
7533d138fa
🧬 perf: Evolve Compaction Guidance on Warm Turns (#15371)
* perf: evolve compaction guidance on warm turns

* style: sort compaction adapter imports
2026-08-30 17:11:20 -04:00
Danny Avila
c1cb591d49
🗿 feat: Add Attached Stateful Code Environments (#15352)
* feat: add attached stateful code environments

* fix: include code environment in lazy agent type

* fix: harden stateful environment routing

* fix: complete code environment route isolation

* fix: declare code environment map type

* fix: preserve configured code execution routes

* fix: harden stateful environment updates

* test: assert route-scoped sandbox readiness

* fix: harden stateful environment lifecycle

* fix: isolate migrated code sessions

* test: assert route-qualified code sessions
2026-08-30 15:38:45 -04:00
Danny Avila
dc77b78d3e
🔇 refactor: Quiet Framework Logging in Unit Tests (#15363)
* 🔇 refactor: Quiet Framework Logging in Unit Tests

Unit test output was overwhelmingly framework log lines rather than test
results. A `packages/api` run printed 9,076 lines for 434 suites; 1,596 of
them were the identical `at Console.log (winston/transports/console.js:87)`
frame that Jest staples onto every winston write.

Three causes:

- Each winston logger built its Console transport with a hard-coded
  `level: 'info'`. Winston resolves a transport's explicit level ahead of
  its parent's, so the logger's own `NODE_ENV`-derived `warn` never applied
  — in tests or in production. `CONSOLE_LOG_LEVEL` now drives it, defaulting
  to the previous `info` so deployments are unaffected. `meiliLogger` had
  the same hard-coded level and is routed through the same helper.
- `LOG_TO_FILE` defaults on, so every Jest worker opened DailyRotateFile
  transports. A run left `packages/api/logs/` holding a 107KB error log, a
  gzipped rotation, audit JSON, and — from fake-timer suites — files dated
  1969 and 2023.
- Client suites carried leftover debug `console.log` in shipped source
  (`Test mode. Skipping silent refresh.` alone accounted for 109 blocks)
  plus three test-harness defects whose React error dumps buried everything
  else: a markdown suite rendering Mermaid with no Router, a `~/Providers`
  mock missing `useSearchContext` (the component's own catch swallowed the
  throw, so the sources path was dead while the suite passed), and
  `getBoundingClientRect` stubs without `left`, which made a computed
  `right` NaN.

A shared `config/jest.setup.logging.cjs` sets the two env vars for both
backend workspaces. It sets env only and requires nothing: eagerly requiring
the logger froze `CREDS_KEY` into `encryptV3` before specs could set it and
broke five suites. `TEST_VERBOSE_LOGS=true` restores the logs for debugging.

Output per full run:

  packages/api           9,076 -> 770 lines   (winston lines 1,596 -> 0)
  packages/data-schemas  2,521 -> 946 lines   (winston lines   290 -> 0)
  client                10,384 -> 3,153 lines (console blocks  285 -> 74)

Test counts are unchanged: client 476 suites / 5,827 tests green, and the
Ariakit `act(...)` warnings the selector suites emitted are gone rather than
suppressed.

* 🔧 fix: Settle Shutdown Timers Before Counting Them

`destroy()` finishes cancelling its fenced retirement timers on the tick
after it resolves. The count only returned to the baseline because a console
write from the logger happened to yield first — silencing that logging left
four timers still pending at the assertion, deterministically failing the
suite in CI's sharded, coverage-enabled run.

Advance the fake clock by zero before counting, so the assertion proves the
timers were cancelled rather than that something incidental yielded.

* 🛡️ fix: Validate CONSOLE_LOG_LEVEL Instead of Trusting It

Winston resolves a level name against the logger's level map, so a name
outside it resolves to `undefined` and the transport drops every message.
Passing `CONSOLE_LOG_LEVEL` straight through meant a typo (`warning`) or a
stray space silently muted all console output — a misconfiguration that reads
as a dead deployment.

Validate against the level set, normalize case and whitespace, and fall back
to the default with a warning on the way past, since the logger cannot report
its own misconfiguration.

The level map itself was duplicated verbatim in both loggers; it now lives in
`utils` beside the validation that depends on it.

* 🎚️ fix: Let CONSOLE_LOG_LEVEL Outrank DEBUG_CONSOLE

`DEBUG_CONSOLE=true` forced the console transport to `debug`, so on those
deployments every non-`silent` value of `CONSOLE_LOG_LEVEL` was discarded —
`CONSOLE_LOG_LEVEL=error` still emitted debug and info, defeating the control
it advertises.

`DEBUG_CONSOLE` now only moves the *default* level and keeps choosing the debug
format, so an explicit level stays in charge of verbosity. With no explicit
level the resolved default is still `debug`, unchanged for existing setups.

Covers the wiring with a spec that re-imports the logger per environment,
rather than only testing the resolver in isolation.
2026-08-30 15:26:30 -04:00
Danny Avila
2ae6c8aea9
🛎️ feat: Wake Agents on Background Tool Completion (#15350)
* feat: wake agents for background tool completion

* fix: isolate background completion contracts

* fix: anchor background completion identity

* fix: close background completion delivery gaps

* fix: preserve manual background polling

* fix: preserve legacy tool group identity

* fix: close background wakeup identity gaps

* test: type background wakeup enqueue mock

* fix: preserve background completion identity

* test: type activity phase fixture

* test: mock phase media query

* fix: arbitrate background result ownership

* fix: retain tool step routing helper

* test: expand phase groups before identity checks

* fix: preserve background completion ownership

* fix: bound durable background results

* fix: type wakeup input budget export

* chore: sort background handler imports

* fix: persist timed-out background completions

* fix: reconcile background completion ownership

* fix: type background completion capabilities

* fix: harden background completion terminalization

* test: type missing completion evidence

* fix: require evidence for completion retirement

* chore: satisfy background completion static checks

* test: cover automatic background completion wakeups

* test: assert background wakeup agent identity

* fix: wake capability-fenced trigger deliveries

* test: keep capability shield fixture public

* test: assert capability worker wakeup

* fix: close background completion ownership gaps

* test: satisfy completion lease static checks

* fix: preserve artifact completion wakeups

* fix: close background delivery recovery gaps

* fix: simplify background receipt guidance

* fix: recover dead background completion batches

* fix: fence background completion recovery

* fix: recheck background recovery ownership

* fix: fence unpublished background continuations

* test: await background message hydration
2026-08-30 08:56:41 -04:00
Danny Avila
e3ccaba5af
🛄 feat: Restore Compaction Guidance Across Continuations (#15356)
* 🧭 feat: preserve compaction guidance across continuations

* 🔧 fix: narrow persisted compaction fields
2026-08-30 08:13:24 -04:00
Danny Avila
70f735336d
🛎️ fix: Enroll Remote Agent Runs in the Generation Lifecycle (#15349)
* fix: enroll remote agent runs in generation lifecycle

* fix: close remote lifecycle ownership gaps

* fix: close remote conversation drain races

* fix: fence remote runs during conversation deletion

* fix: reconcile remote deletion and settlement races

* fix: complete owner deletion recovery

* fix: preserve remote cleanup receipts

* fix: consume deletion receipts before cleanup

* fix: expose idempotent deletion option

* chore: sort remote lifecycle imports
2026-08-30 07:14:13 -04:00
Danny Avila
fa913148fb
🔒 fix: Refresh MCP OBO Tokens From the Live OpenID Session (#15334)
* 🧊 fix: Inline-refresh OpenID session tokens at MCP OBO call time

Resolves the walk-away failure mode where MCP tool calls using OBO auth
fail with "No valid OpenID access token is available for OBO exchange"
after a user idles past their access-token lifetime. The strategy-time
snapshot on `user.federatedTokens` could expire mid-stream before
`resolveOboToken` ran, while `req.session.openidTokens` carried a still-
valid (or refreshable) token that nothing read.

- New OpenIDSessionRefresh service: per-user single-flighted closure that
  reads `req.session.openidTokens` at OBO time and inline-refreshes via
  `openid-client.refreshTokenGrant` when expired (30s skew), persisting
  via `req.session.save()`. No cookie writes (headers already flushed).
- `resolveOboToken` gains a required UpstreamTokenProvider parameter
  (typed as `() => Promise<OIDCTokens | null>`, reusing the shared shape
  from @librechat/data-schemas). Compile-time guarantee that every call
  site is updated.
- New `session_refresh_failed` OboTokenResolutionReason distinguishes
  "session expired and IdP rejected refresh" from "no upstream token
  ever existed."
- `req` threaded through createMCPTool/createMCPTools/createToolInstance
  to construct the closure with captured request, plus fail-closed
  guards in MCPConnectionFactory.getOboTokens and MCPManager.callTool
  when the closure isn't plumbed.
- Startup warning in MCPServersInitializer when OBO is configured but
  OPENID_REUSE_TOKENS is unset (the strategy populating
  user.federatedTokens is only registered under reuse, so OBO would
  fail every call without it).

Tests: 16 new in OpenIDSessionRefresh.spec.js; obo.spec.ts extended
for the new param + error reason; wiring smoke tests in MCPManager,
MCPConnectionFactory, MCPServersInitializer, and MCP.spec.js.

* 🛡️ fix: Harden OBO inline-refresh against token type and session edge cases

- Token-preference asymmetry: live-token reuse and expires_at derivation
  now strictly gate on the access_token, not the id_token. Added a
  required `tokenPreference` parameter on isLiveSessionTokenStillValid,
  buildOIDCTokensFromSession, and createOpenIDSessionTokenProvider
  so every call site is explicit. Dropped the bogus id_token-exp
  fallback in performIdpRefresh — id_token TTL is governed by IdP
  session policy and would mark a short-lived access_token reusable
  past its real lifetime.
- Missing req in /reinitialize route: the manual reconnect
  endpoint now forwards req into reinitMCPServer, so OBO servers can
  build a session-aware upstream-token closure instead of failing with
  missing_upstream_token.
- Single-flight key collisions: composed key as
  tenantId:openidIssuer:openidId:sessionId via getSingleFlightKey.
  Concurrent calls in the same session still coalesce; separate sessions
  never share an in-flight refresh, preventing refresh-token rotation
  from breaking sibling sessions and preventing cross-tenant token
  crossover when distinct users share an IdP sub.
- Opaque access token reuse): persist accessTokenExpiresAt
  (unix seconds, from tokenset.expires_in) on each refresh AND on initial
  login / SPA refresh in setOpenIDAuthTokens. New getAccessTokenExp
  helper falls back to it when the access token isn't a JWT, avoiding
  redundant inline refreshes for Microsoft Graph and Auth0 default
  audiences.
- Log hygiene: the single-flight key (containing sessionId,
  openidId, openidIssuer, tenantId) is now SHA-256-hashed in the
  "Joining in-flight refresh" debug log. Preserves cross-line correlation
  via a 12-char prefix without leaking credential or PII material.

Documented req.session.openidTokens shape contract via JSDoc typedef so
the new accessTokenExpiresAt field has a discoverable home alongside the
existing accessToken/idToken/refreshToken/expiresAt/lastRefreshedAt.

Tests: OpenIDSessionRefresh.spec.js up to 30 passing (added coverage for
opaque-token reuse, JWT-access-token-exp fallback, no-id_token-fallback
regression, cross-session no-coalesce, persistence on refresh, and a
guard against stale accessTokenExpiresAt carryover). AuthService.spec.js
adds two cases covering accessTokenExpiresAt persistence on login.
mcp.spec.js (route) gains a regression test asserting req flows into
reinitMCPServer.

* 🔍 fix: Detect OBO-only MCP admin config overrides

Admin Config overlays for YAML-defined MCP servers compare only
ADMIN_CONFIGURABLE_FIELDS to decide whether to lazy-init a config-tier override.
The OBO config field was added after that fingerprint list, so an override that
only added or changed `obo` was treated as unchanged YAML and skipped.

Include `obo` in the admin-configurable field list and add a regression test for
an OBO-only override.

* 🔊 fix: Mock MCP OAuth timeout in SDK integration test

MCPConnectionFactory.attemptToConnect reads mcpConfig.OAUTH_HANDLING_TIMEOUT
when building the OAuth connection timeout. The SDK OAuth integration test
mocked mcpConfig without that field, which made the timeout calculation produce
NaN and caused the test to fail before the OAuth refresh/start path completed.

Add OAUTH_HANDLING_TIMEOUT to the test mock.

* ♻️ refactor: Pass OBO upstream-token closure into MCP instead of req

Build the OpenID upstream-token provider at the request boundary and thread
only the closure through MCP handling, so the MCP service layer no longer
receives the raw Express request. The closure still reads/refreshes the live
session at tool-call time, preserving the walk-away recovery.

- Drop `req`/`capturedReq` from createMCPTools, createMCPTool, reconnectServer,
  createToolInstance, and reinitMCPServer; forward `upstreamTokenProvider`
  instead. Closure is constructed in loadTools, loadToolDefinitionsWrapper, and
  the reinitialize route, where req/res are in scope.
- OBO: fall back to user.federatedTokens when the provider yields no live
  session, so OIDC remote-agent calls (verified bearer, no session) still work.
- Inline refresh: mirror a rotated refresh token to the refreshToken cookie via
  a shared setRefreshTokenCookie helper, guarded by !res.headersSent (no-op on
  the streaming path; session copy stays authoritative).
- Single-flight: hydrate a joining request's own session from the resolved
  tokens so a later OBO call doesn't replay a rotated-away refresh token.

Addresses owner feedback and three review findings.

* 🔒 fix: Recover OIDC refresh-token rotation after SSE OBO refresh

When an inline OBO refresh rotates the OpenID refresh token after SSE headers
have already been sent, the browser refreshToken cookie cannot be updated. Store
a short-lived encrypted bridge from the stale cookie token to the rotated token
so /api/auth/refresh can recover after express-session loss.

Use the signed openid_user_id cookie to load user context for bridge validation,
retry only on invalid_grant, and delete the bridge only after the bridged refresh
succeeds.

* 🔨 fix: hydrate joined OIDC refresh sessions with stable refresh tokens

Update single-flight OIDC refresh joiners whenever refreshed access token
state changes, even if the IdP keeps the refresh token unchanged.

This prevents joined requests from retaining stale accessToken or
accessTokenExpiresAt values and redundantly refreshing later in the same run.

* 🌉 Persist OIDC refresh-token recovery bridges in MongoDB

Store SSE OBO refresh-token recovery bridges in MongoDB instead of
process-local memory so /api/auth/refresh can recover after worker
restarts or cross-worker routing.

Derive bridge expiry from REFRESH_TOKEN_EXPIRY so the recovery window
matches the stale refreshToken cookie it repairs, and delete bridges
after successful recovery.

* 🤝 Coordinate OIDC inline refreshes across workers

Add a short-lived Mongo-backed refresh-flight record so concurrent
OBO refreshes for the same OpenID session do not redeem the same
rotating refresh token on different workers.

The winning worker performs the IdP refresh and stores an encrypted
result; joiners wait for that result, hydrate their request session,
and return without calling the IdP.

* ⚓ Keep OpenID marker cookies aligned on inline refresh

Refresh token_provider and openid_user_id with the same expiry as the
rotated refreshToken cookie when an inline OBO refresh can still write
headers.

Share the marker-cookie writer with the normal OpenID auth refresh path
so the fallback /api/auth/refresh branch continues to recognize valid
OpenID refresh tokens after session expiry.

* 🔑 fix: include refresh token in OIDC local refresh flight key

Key the process-local OIDC refresh coalescing by the current session
refresh token, matching the Mongo-backed flight key. This prevents a
request with a newly rotated token from joining an older pending refresh
and inheriting its failure/result.

* 🌉 fix: store OIDC refresh bridge without cookie response

Treat missing or non-cookie responses like headers-sent streaming
responses during inline OIDC refresh. When the IdP rotates the refresh
token and cookies cannot be written, persist a recovery bridge so a later
/auth/refresh can recover after session expiry.

* 🫙 fix: preserve stale OIDC cookie bridge key

Track the refresh token last written to the browser cookie separately
from the current session refresh token. When inline OIDC refreshes rotate
tokens without a writable response, keep bridging from the browser-stale
token directly to the latest session token.

* 🙌 fix: keep OIDC bridge recovery success on cleanup failure

Make refresh-token bridge cleanup best-effort after a bridged OIDC
refresh succeeds. A transient delete failure now logs a warning but does
not convert the already-refreshed session and cookies into a 403 response.

* 📦 test: Exclude RefreshTokenBridge from tenant-isolation coverage

Add RefreshTokenBridge to the tenant-isolation coverage allowlist because
refresh bridge lookups run during unauthenticated OpenID refresh recovery.
The controller first recovers user context from the signed OpenID marker
cookie, then the bridge methods apply explicit user and tenant filters.

Ambient tenant isolation would bind this recovery path to request-local
tenant context that is not available at the point the stale cookie is being
resolved

* ⚡ Fix OpenID refresh flight retry and marker hydration

Allow failed OpenID refresh flights to be reclaimed immediately instead of pinning transient errors.

Preserve the browser refresh-token marker when joined refreshes hydrate session tokens from a shared flight result.

Stabilize AuthService tests by isolating mocked module imports from prior suites.

* 🛠️ fix: centralize OBO identity scoping

Add shared auth identity helpers for app user ids, OpenID subjects,
tenant ids, and normalized OpenID issuers.

Thread a non-placeholder-visible OBO identity context from the real
request user through MCP connection, tool-call, reinit, and refresh
paths. Keep tenantId and openidIssuer out of createSafeUser so MCP
user placeholders do not expose those fields.

Scope OBO token cache and in-flight exchange keys by tenant, issuer,
OpenID subject, scopes, and a SHA-256 hash of the upstream assertion.
This prevents cross-tenant/cross-issuer collisions and avoids reusing
tokens minted from stale rotated assertions.

Use the shared identity helpers for OpenID refresh-flight keys and
refresh-token bridge recovery records so related OBO refresh paths share
the same identity normalization rules.

The helper is intended for auth-boundary and credential-cache code, not
as a blanket replacement for ordinary app user id ownership checks.

* 🛠️ fix: preserve OIDC refresh-token sync on save failures

Sync OpenID refresh-token cookie/bridge state before persisting the
session so a transient session-store failure cannot lose an IdP-rotated
refresh token.

Also trigger sync when the session refresh token differs from the
browser refresh-token marker, not only when the current grant rotates
the token. This lets later writable refreshes repair stale browser
cookies left behind by SSE refreshes.

Route refresh bridge identity through the shared identity helper with
the threaded OBO identity context, falling back to request/user context
when needed.

Add regression coverage for session-save failures, stale browser cookie
repair, non-writable bridge storage, and shared-helper identity fallback.

* 🛠️ fix: keep OIDC refresh bridge during recovery grace

After successful bridged refresh recovery, re-store the stale-cookie bridge
with a short grace TTL instead of deleting it immediately. This lets parallel
/api/auth/refresh requests that already sent the stale browser cookie recover
before they can observe the first response's Set-Cookie.

Retarget the bridge to the refresh token returned by the bridged retry so
B-to-C refresh-token rotation remains recoverable. The grace TTL is parsed with
math() and defaults to 60s, which shrinks the replay window from the original
REFRESH_TOKEN_EXPIRY bridge lifetime to the short recovery grace period.

Remove the now-unused explicit bridge delete path from the service and
data-schemas method surface. Add regression coverage for grace re-store,
identity symmetry, retry failure behavior, and same-key upsert replacement.

* 🛠️ fix: fail closed on OBO MCP user identity mismatch

Add an OBO-specific guard before MCP tool execution that requires the
effective invocation user and captured request user to both have ids and
to match. This prevents OBO tool calls from falling back to a separate
configurable.user_id identity after request-bound OBO context has already
been captured.

Keep the existing user id fallback behavior for non-OBO MCP calls.

Tests cover mismatched OBO users, missing user ids, and the matching-user
path ignoring a conflicting configurable.user_id.

* 🛠️ fix: Guard OpenID bridge retry user identity

Extract the shared OpenID refresh/user-resolution flow in AuthController
so the normal refresh path and bridge-recovery retry use the same grant,
claims, issuer, user lookup, and diagnostic logging code.

Preserve the existing path-specific behavior: the normal path still owns
migration updates and 401 login redirects, while the bridge retry still
falls through to the existing 403 invalid-token response.

Add a bridge-recovery guard that rejects retry results whose resolved
user id differs from the signed openid_user_id cookie before issuing
tokens or re-storing the grace bridge. Cover both the successful
matching-user recovery and the mismatched-user rejection.

* 🛠️ fix: type-safety polish on OBO data layer

Replace refresh token bridge query/update Record<string, unknown> usage
with typed Mongoose FilterQuery and UpdateQuery definitions.

Harden OpenID marker cookie JWT expiry handling by converting refresh
expiry milliseconds to integer seconds and rejecting invalid or
non-positive durations.

Add focused CSRF tests for fractional refresh expiry values and invalid
expiry configuration.

* 🛠️ fix: Bind OpenID session tokens to authenticated identity

Stamp OpenID session token state with the LibreChat user id, OpenID subject,
tenant id, and normalized issuer when tokens are stored.

Fail closed before OBO inline token reuse/refresh when the session token
identity does not match the current authenticated identity, preventing a stale
or mixed Express session from supplying another user's upstream assertion.

Also validate the normal /api/auth/refresh session-token reuse shortcut against
the signed marker-cookie user before returning cached session tokens.

Note: sessions created before this change carry no identity stamp and are
treated as a mismatch. This is self-healing — the reuse path forces a full IdP
refresh (which re-stamps the session) and the OBO path throws, surfacing as a
one-time re-authentication for active OBO users at deploy time. The session
re-stamps within one session lifetime (SESSION_EXPIRY, default 15 min).

* 🛠️ fix: Recover OpenID refresh token drift

Prefer the browser refresh-token cookie when it differs from the
server-side OpenID session state, and force a real IdP refresh in that
case instead of reusing stale session tokens.

Store a short-lived refresh-token bridge when inline OBO refresh writes
a rotated browser cookie but session persistence fails, so follow-up
refreshes can still recover from the old token.

Keep the bridge grace TTL centralized in RefreshTokenBridge so both
recovery paths use the same env-backed value.

Note: drift is measured against the last-synced browserRefreshToken
marker, so the SSE path (intentionally stale cookie, authoritative
session) does not false-positive. Sessions predating the marker have no
browserRefreshToken; for those, drift falls back to comparing the cookie
against the session refresh token and prefers the cookie on difference.
This is the same self-healing pre-change-session window as the identity
binding fix and re-syncs within one session lifetime.

Tests cover cookie/session drift selection, reusable-session bypass on
drift, bridge storage after session-save failure, and the shared bridge
constant wiring.

* 🛠️ fix: Harden OBO token caching and expiry handling

Reject malformed OBO grant responses before writing them to the exchanged-token cache so a missing access_token cannot poison the cache.

Store absolute expires_at values with cached OBO tokens and ignore legacy cache entries without usable expiry metadata. This keeps cached-token freshness based on the token’s real
remaining lifetime instead of reusing the original relative expires_in on cache hits.

Move OBO expiry normalization and skew helpers into packages/api and use them from both the JS exchange service and the TS MCP resolver. Apply a 30-second safety margin with a one-
second floor for short-lived tokens, covered by direct helper tests and caller-level regression tests.

Tests:
- packages/api: npm run build
- packages/api: npx jest src/mcp/oauth/expiry.spec.ts src/mcp/oauth/obo.spec.ts
- api: npx jest server/services/OboTokenService.spec.js

* 🛠️ fix: Harden OBO refresh-token bridge lookup and indexing

Reuse getValidOpenIDReuseUserId for the bridge-recovery user lookup in
refreshController instead of re-verifying openid_user_id inline. The shared
helper enforces the JWT_REFRESH_SECRET presence check and a strict
typeof payload.id === 'string' guard, rejecting tokens whose id claim is
present but not a string (e.g. a numeric id) that the inline check accepted.

Fail closed on issuer mismatch in getRefreshTokenBridge. Both the stored and
the expected issuer are now normalized and compared for equality, so a bridge
is recovered only when both sides agree (both absent, or both present and
equal after normalization). Previously the check was skipped whenever the
stored issuer was absent, allowing recovery across mismatched issuer context.

Drop the unused {oldRefreshTokenHash, userId, tenantId, openidIssuer} index
and the openidIssuer field on RefreshTokenBridgeQuery. The data-layer filter
only queries the 3-field {oldRefreshTokenHash, userId, tenantId} index; the
issuer is verified in application code, not the query. Hoist the repeated
model accessor into getRefreshTokenBridgeModel.

Note: issuer is now load-bearing for recovery. A bridge stored with an issuer
recovers only when the lookup supplies a matching issuer; the recovery lookup
reads user.openidIssuer via AUTH_REFRESH_USER_PROJECTION (an exclusion
projection that retains the field). If a user's persisted openidIssuer is
empty while the stored bridge has one, recovery fails closed (falls through to
normal re-authentication) until the bridge TTLs out — no security regression.

Tests cover invalid signed-cookie payloads bypassing the bridge, both
asymmetric issuer-presence cases, issuer normalization before comparison, and
an index-alignment assertion guarding against re-adding the dropped index.

* 🛠️ fix: Degrade OBO discovery on token resolution failures

Catch expected OboTokenResolutionError failures during MCP tool discovery and
fall back to unauthenticated tool listing instead of aborting discovery. This
keeps discovery aligned with the existing unauthenticated listing behavior while
preserving unexpected errors as real failures.

Also correct OBO tool-call freshness comment and tighten the OBO trust-check
permissions type to the existing role permission shape.

Tests:
- npx jest src/mcp/__tests__/MCPConnectionFactory.test.ts --runInBand --coverage=false
- npx jest src/mcp/oauth/obo.spec.ts --runInBand --coverage=false

* 🛠️ fix: tighten OBO tool-call errors, bridge logging, and flight typing

Move resolveToolCallUserId inside the tool-call try/catch so an OBO
identity mismatch surfaces with serverName/toolName context and the
standard tool-call-failed message instead of an opaque bare Error.

Raise the refresh-token bridge lookup failure log from debug to warn so
transient infrastructure failures on the unauthenticated /api/auth/refresh
path are observable, and guard the message access against non-Error values.

Replace the unknown+cast in isDuplicateKeyError with a hasErrorCode type
predicate so the duplicate-key check reads error.code without an assertion.

Preserve real math/isEnabled in the MCPConnectionFactory test mock (mock
only processMCPEnv) so mcpConfig timeouts no longer resolve to NaN, fixing
the TimeoutNaNWarning that masked slow OAuth retry behavior.

* 🧪 fix: Restore the Flight Uniqueness Index and Buffer the Graph Cache TTL

Two CI failures on the merge, both in suites this environment cannot run
(their MongoDB binary download is blocked).

`GraphApiService.spec.js` still asserted the unbuffered TTL. Graph tokens
route through the same `getTokenCacheTtlMs` as the OBO and openidStrategy
caches, so the entry now expires 30s before the credential does.

`openidRefreshFlight.spec.ts` dropped the database between tests, which takes
the indexes with it, and Mongoose builds them only once when the model is
compiled. Whether the unique `key` index survived into a test was a race with
that one-time build. Without it a second `create` inserts instead of raising a
duplicate-key error, so every worker believes it won the flight — the
mutual exclusion the file exists to prove. Indexes are now rebuilt after each
drop, which also makes the reclaim and complete cases reach those paths for
the right reason.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SxKWxwqxAGckYpRsYTqx3F

* fix: address OBO review findings

* 🔐 fix: Install Bridge Indexes and Carry OBO Through Assistant Recovery

Two findings from the Codex pass on d08c82f0d.

The refresh-token bridge relied on Mongoose auto-indexing for both of its
indexes, and `MONGO_AUTO_INDEX=false` is a supported deployment setting. A
bridge holds an encrypted refresh token and the TTL index is the only thing
that ever deletes one, so under that setting they would accumulate for the
life of the collection while concurrent upserts lost the compound uniqueness
the filter assumes. Installed before the first write, matching the flight
methods and the session and schedule methods before them.

`recoverServerTools`, the assistant create/update path that reruns
`reinitMCPServer` when a referenced server's catalog and connection snapshot
are both missing, was the last reinit site not carrying the upstream-token
closure. For an OBO server the factory rejects the connection outright, so the
assistant write failed with unavailable MCP definitions. It now builds the
provider at that request boundary like the other entry points; assistant
writes have no `res`, so a rotation there falls back to the recovery bridge.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SxKWxwqxAGckYpRsYTqx3F

* fix: harden OBO refresh coordination

* ⏳ fix: Keep Elapsed Expiries Elapsed and Revoke the Superseded Session

Two of the five findings from the Codex pass on fcdc15885 — the two that are
defects in code this branch introduced rather than design questions about the
bridge.

`getSkewedTokenExpiresAtMs` floored every result at a second in the future,
including an expiry the provider had already declared elapsed. An exchange
answering `expires_in: 0` or a past `expires_at` was handed to the MCP
connection stamped valid for another second, which only moves the failure
downstream. The floor now applies to a lifetime that is still live, which is
what it was for; an elapsed one stays elapsed so the caller rejects it. Same
for the cache TTL, which falls back to the elapsed-credential floor.

Bridge recovery left the stale token's durable Session behind. Only the token
it recovered through was passed as `existingRefreshToken`, so that one's
session was replaced while the token the browser actually presented kept its
record until its original expiry. That record, with the marker cookie still
bound to it, is what authorizes local image access for OpenID users — so a
copy of the stale cookie outlived the rotation it had lost. Revoked
explicitly on successful recovery.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SxKWxwqxAGckYpRsYTqx3F

* fix: close OBO refresh review findings

* 🎟️ fix: Carry the Bridged Token Through a Non-Rotating Recovery

Bridge recovery passes the browser's stale token as `existingRefreshToken` so
the durable Session naming it is the record replaced. That also makes the
stale token the fallback the installed session and the refresh cookie use when
a tokenset carries no `refresh_token` of its own, which holds only while the
recovery grant rotates.

An IdP that answers that grant without rotating sends the browser back to the
very token the bridge exists to retire: `storeOpenIDSession` installs and
deletes the same stale record in one call, and the cookie is rewritten to a
token the IdP already rejected — a sign-out on the next refresh. The grace
bridge one line above already guards this with `|| bridgedRefreshToken`; the
resolved tokenset now does the same, so leader and followers alike publish the
recovered token.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SxKWxwqxAGckYpRsYTqx3F

* 🚫 fix: Reject an OBO Exchange That Returns an Expired Credential

Preserving an elapsed expiry through the skew helper only helps if something
acts on it, and nothing did: `MCPManager.callTool` checks the access token and
nothing else before setting the Authorization header, so a credential the IdP
declared spent still went downstream to fail there. It is rejected at the
exchange now, where the reason is known, and retryably — the exchange itself
worked, so a fresh grant can succeed.

Completes the elapsed-expiry change in cb26a6f7d, which made the stamp honest
without giving anyone a reason to look at it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SxKWxwqxAGckYpRsYTqx3F

* fix: close OpenID refresh review findings

* fix: coordinate OpenID refresh entry points

* chore: sort OpenID flight imports

* fix: fence OpenID refreshes during logout

* fix: close OpenID logout publication races

* fix: narrow completed refresh flight

* test: cover bridge cleanup failure after ownership loss

The compensating delete in storeRefreshTokenBridgeWithLease swallows its own
failure so the lease error stays the one the caller sees. Nothing asserted
that, so removing the inner catch left every suite green while callers began
receiving the cleanup error instead.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SxKWxwqxAGckYpRsYTqx3F

* fix: compensate OpenID bridges only on proven ownership loss

The post-write lease assertion deletes the bridge it just published when it
throws, but it threw for two different reasons: a coordination record that is
no longer ours, and a coordination read that simply failed. Treating the second
as the first destroys the only mapping from the token the browser still holds
to the one the IdP already rotated to, so a transient Mongo error on the
headers-already-sent path signed the user out.

Tag the ownership error where the lease raises it and compensate only for that,
preserving the bridge whenever ownership is merely undetermined. A preserved
bridge stays behind the logout revocation fence, so the safe default costs
nothing.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SxKWxwqxAGckYpRsYTqx3F

* fix: reject spent OpenID refresh results

Two ways a refresh could report success while handing back a credential
nothing can use.

An inline refresh carries the previous id_token forward when the IdP omits
one on rotation, so tokenset.id_token is not necessarily freshly issued.
setOpenIDAuthTokens applied its freshness guard only to the session copy and
took tokenset.id_token unconditionally, so /refresh returned an expired
bearer even though the grant produced a usable access token. Skip it only
when it is provably expired: an id_token whose expiry cannot be read stays
preferred, since access_token may be opaque or scoped to another audience.

normalizeExpiresIn preserves a zero or negative lifetime rather than
discarding it, so a grant declaring an already-spent access token still
published, rotating the refresh token and returning a token every freshness
check rejects. Each OBO call then repeated the grant. Reject an elapsed
lifetime before publishing; an unknown lifetime still publishes.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SxKWxwqxAGckYpRsYTqx3F

* fix: harden OpenID refresh publication

* fix: keep identity and results intact through OpenID refresh cleanup

Two follow-ons from the last round's fixes.

Stripping an expired carried-forward id_token from the refresh result removed
the only identity material a rotation without id_token leaves behind. The
result is rebuilt by buildOIDCTokensFromSession, so it carries no provider
claims() either, and getTokenClaims accepts only those two — bridge recovery
failed with "no usable identity claims" before setOpenIDAuthTokens could hand
back the fresh access token. The stripped token now travels in a
non-enumerable marker, alongside the existing browser and predecessor markers,
which identity resolution reads and the authentication response never sees.

The lease drained a pending renewal by awaiting it in finally, so a transient
coordination failure there threw from finally and replaced the operation's
result. The refresh had already settled and published, so the caller saw a
failure on credentials that had rotated. Proven ownership loss is recorded on
ownershipLost and checked before the return, so the drain has nothing to add
but noise; it now absorbs and logs.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SxKWxwqxAGckYpRsYTqx3F

* refactor: fence OpenID recovery publication

* fix: close OpenID publication transaction

* refactor: make OpenID publication transactional

* fix: satisfy OpenID publication type checks

* fix: fence OpenID session publication

* fix: authorize OpenID refresh publication

* fix: bind OpenID replay generations

* fix: fence OpenID response generations

* fix: authorize OpenID token delivery

* fix: linearize OpenID publication delivery

* chore: sort OpenID refresh flight imports

---------

Co-authored-by: J.C. Bartle <jcbartle@users.noreply.github.com>
Co-authored-by: jbartle <jbartle@rand.org>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: jcbartle <7274202+jcbartle@users.noreply.github.com>
2026-08-29 23:55:58 -04:00
Danny Avila
77c2a51cf3
🪧 fix: Advertise Detached Event Actor Support from the Generation Store (#15322)
Some checks failed
Backend Unit Tests / Build packages (push) Waiting to run
Backend Unit Tests / Codegraph select (push) Waiting to run
Backend Unit Tests / TypeScript type checks (push) Blocked by required conditions
Backend Unit Tests / Circular dependency checks (push) Waiting to run
Backend Unit Tests / Tests: api (shard 1/3) (push) Blocked by required conditions
Backend Unit Tests / Tests: api (shard 2/3) (push) Blocked by required conditions
Backend Unit Tests / Tests: api (shard 3/3) (push) Blocked by required conditions
Backend Unit Tests / Tests: data-provider (push) Blocked by required conditions
Backend Unit Tests / Tests: data-schemas (push) Blocked by required conditions
Backend Unit Tests / Tests: @librechat/api (shard 1/4) (push) Blocked by required conditions
Backend Unit Tests / Tests: @librechat/api (shard 2/4) (push) Blocked by required conditions
Backend Unit Tests / Tests: @librechat/api (shard 3/4) (push) Blocked by required conditions
Backend Unit Tests / Tests: @librechat/api (shard 4/4) (push) Blocked by required conditions
Codegraph E2E Votes / vote (full suite) (push) Waiting to run
Docker Dev Branch Images Build / build (Dockerfile, lc-dev, node) (push) Waiting to run
Docker Dev Branch Images Build / build (Dockerfile.multi, lc-dev-api, api-build) (push) Waiting to run
Frontend Unit Tests / Codegraph select (push) Has been cancelled
Frontend Unit Tests / Build packages (push) Has been cancelled
Frontend Unit Tests / TypeScript type checks (client) (push) Has been cancelled
Frontend Unit Tests / Tests: @librechat/client (push) Has been cancelled
Frontend Unit Tests / Tests: Ubuntu (shard 1/2) (push) Has been cancelled
Frontend Unit Tests / Tests: Ubuntu (shard 2/2) (push) Has been cancelled
Frontend Unit Tests / Vite build verification (push) Has been cancelled
* fix: activate detached event actions automatically

* fix: shield detached terminal generations

* perf: share capability availability index

* style: flatten capability status selection

* fix: preserve mixed-version lifecycle shells

* fix: complete detached action store adapters

* fix: close mixed-version capability races

* fix: honor legacy capability success
2026-08-28 16:33:23 -04:00
Danny Avila
487984193a
🪂 feat: Make Event Actor Detached Actions Durable (#15307)
Some checks are pending
Docker Dev Images Build / build (Dockerfile, librechat-dev, node) (push) Waiting to run
Docker Dev Images Build / build (Dockerfile.multi, librechat-dev-api, api-build) (push) Waiting to run
Sync Helm Chart Tags / Ignore non-main push (push) Waiting to run
Sync Helm Chart Tags / Sync chart tags (push) Waiting to run
* feat: make Event Actor detached actions durable

* fix: align detached actor checks with merged base

* fix: close detached actor lifecycle gaps

* fix: break detached action outcome cycle

* fix: deepen detached action ownership

* fix: fence detached launch handoffs

* fix: preserve detached rollout fencing

* test: type detached recovery failure

* fix: close detached ownership handoffs

* fix: stage detached action activation

* fix: persist detached terminal retries

* fix: fence detached durability ownership

* fix: fence trigger lane publication

* fix: fence detached transition generations

* fix: retire failed detached predecessors

* fix: close detached rollout consumers

* test: type detached retry turns

* fix: close detached ownership gaps

* fix: fence detached action resumes

* fix: fence prior-head event actor resumes

* fix: preserve legacy event actor resumes

* fix: align detached resume boundaries
2026-08-28 14:21:04 -04:00
Danny Avila
04a7577821
🧵 fix: Retry Shared Links After Message Persistence Gaps (#15306)
Some checks failed
Backend Unit Tests / Build packages (push) Waiting to run
Backend Unit Tests / Codegraph select (push) Waiting to run
Backend Unit Tests / TypeScript type checks (push) Blocked by required conditions
Backend Unit Tests / Circular dependency checks (push) Waiting to run
Backend Unit Tests / Tests: api (shard 1/3) (push) Blocked by required conditions
Backend Unit Tests / Tests: api (shard 2/3) (push) Blocked by required conditions
Backend Unit Tests / Tests: api (shard 3/3) (push) Blocked by required conditions
Backend Unit Tests / Tests: data-provider (push) Blocked by required conditions
Backend Unit Tests / Tests: data-schemas (push) Blocked by required conditions
Backend Unit Tests / Tests: @librechat/api (shard 1/4) (push) Blocked by required conditions
Backend Unit Tests / Tests: @librechat/api (shard 2/4) (push) Blocked by required conditions
Backend Unit Tests / Tests: @librechat/api (shard 3/4) (push) Blocked by required conditions
Backend Unit Tests / Tests: @librechat/api (shard 4/4) (push) Blocked by required conditions
Codegraph E2E Votes / vote (full suite) (push) Waiting to run
Docker Dev Branch Images Build / build (Dockerfile, lc-dev, node) (push) Waiting to run
Docker Dev Branch Images Build / build (Dockerfile.multi, lc-dev-api, api-build) (push) Waiting to run
Frontend Unit Tests / Codegraph select (push) Waiting to run
Frontend Unit Tests / Build packages (push) Waiting to run
Frontend Unit Tests / TypeScript type checks (client) (push) Blocked by required conditions
Frontend Unit Tests / Tests: @librechat/client (push) Blocked by required conditions
Frontend Unit Tests / Tests: Ubuntu (shard 1/2) (push) Blocked by required conditions
Frontend Unit Tests / Tests: Ubuntu (shard 2/2) (push) Blocked by required conditions
Frontend Unit Tests / Vite build verification (push) Blocked by required conditions
Sync Locize Translations & Create Translation PR / Sync Translation Keys with Locize (push) Has been cancelled
Sync Locize Translations & Create Translation PR / Create Translation PR on Version Published (push) Has been cancelled
* fix: Retry shared links after message persistence gaps

* chore: Sort share test imports

* fix: Address share review feedback

* fix: Normalize hydrated share target
2026-08-28 08:05:52 -04:00
Danny Avila
c06b09c945
🤖 feat: make Event Actor HITL durable (#15305)
* 🤖 feat: make event actor HITL durable

* 🤖 fix: break event actor outcome cycle

* 🤖 fix: close durable actor terminal races

* fix: harden durable event actor recovery proofs

* fix: close event actor resume publication races
2026-08-28 08:05:11 -04:00
Danny Avila
31325b4f61
🎰 fix: Merge Message Provenance With Compare-and-Swap Retries (#15304)
Some checks are pending
Backend Unit Tests / Build packages (push) Waiting to run
Backend Unit Tests / Codegraph select (push) Waiting to run
Backend Unit Tests / TypeScript type checks (push) Blocked by required conditions
Backend Unit Tests / Circular dependency checks (push) Waiting to run
Backend Unit Tests / Tests: api (shard 1/3) (push) Blocked by required conditions
Backend Unit Tests / Tests: api (shard 2/3) (push) Blocked by required conditions
Backend Unit Tests / Tests: api (shard 3/3) (push) Blocked by required conditions
Backend Unit Tests / Tests: data-provider (push) Blocked by required conditions
Backend Unit Tests / Tests: data-schemas (push) Blocked by required conditions
Backend Unit Tests / Tests: @librechat/api (shard 1/4) (push) Blocked by required conditions
Backend Unit Tests / Tests: @librechat/api (shard 2/4) (push) Blocked by required conditions
Backend Unit Tests / Tests: @librechat/api (shard 3/4) (push) Blocked by required conditions
Backend Unit Tests / Tests: @librechat/api (shard 4/4) (push) Blocked by required conditions
Codegraph E2E Votes / vote (full suite) (push) Waiting to run
Docker Dev Branch Images Build / build (Dockerfile, lc-dev, node) (push) Waiting to run
Docker Dev Branch Images Build / build (Dockerfile.multi, lc-dev-api, api-build) (push) Waiting to run
Frontend Unit Tests / Codegraph select (push) Waiting to run
Frontend Unit Tests / Build packages (push) Waiting to run
Frontend Unit Tests / TypeScript type checks (client) (push) Blocked by required conditions
Frontend Unit Tests / Tests: @librechat/client (push) Blocked by required conditions
Frontend Unit Tests / Tests: Ubuntu (shard 1/2) (push) Blocked by required conditions
Frontend Unit Tests / Tests: Ubuntu (shard 2/2) (push) Blocked by required conditions
Frontend Unit Tests / Vite build verification (push) Blocked by required conditions
* fix: preserve provenance with classic updates

* test: reject provenance update arrays
2026-08-28 00:24:07 -04:00
Danny Avila
6fba51a35a
🫆 feat: Fingerprint Agent Context for Zero-Read Warm Continuation (#15299)
* feat: add context-fingerprinted warm continuation

* fix: preserve manual skills across warm turns

* perf: keep ordinary agent initialization neutral

* fix: fingerprint complete agent topology

* fix: close warm context replay gaps

* fix: preserve lazy subagent memory scope

* perf: fingerprint only model-bound memory

* fix: project lazy memory capability

* style: sort warm continuation imports

* fix: make warm continuation reconstructable

* fix: preserve lazy skill isolation

* fix: retain scoped skill semantics
2026-08-27 23:01:14 -04:00
Danny Avila
03fa98eb37
🕸️ feat: Complete Keenable Web Search Provider (#15288)
* feat: wire Keenable web-search provider into config, schema, and UI

Keenable landed as a search provider in @librechat/agents (#285, shipped in
3.2.58+), but LibreChat did not yet expose it. This adds the config/schema/UI
glue so it can be selected, mirroring the existing Tavily provider.

- data-provider: add `keenable` to SearchProvider type + SearchProviders enum,
  keenableApiKey/keenableApiUrl schema fields, and a keenableSearchOptions block
  (maxResults, site, attributionTitle, timeout).
- data-schemas: register keenable in webSearchAuth.providers and default the
  key/URL placeholders in loadWebSearchConfig.
- api/web: pass keenableSearchOptions through to the provider and handle
  Keenable's keyless model. Unlike other providers it authenticates with no key
  (the public endpoint), picking up an optional key/URL when set; the URL
  override is SSRF-preflighted like other user-provided URLs.
- client: add Keenable to the provider dropdown with an optional API-key input.
- docs: document KEENABLE_API_KEY/KEENABLE_API_URL in .env.example and a
  webSearch example in librechat.example.yaml.
- tests: keyless + keyed auth resolution, config defaults, and schema parsing.

* fix: ESLint no-unused-vars and clarify Keenable yaml example

- Remove the now-unused RerankerTypes import in data-schemas web.ts (the lint
  job runs with --max-warnings 0 on changed files, so this latent warning failed
  CI once the file was touched).
- Note in the librechat.example.yaml Keenable stanza that a scraper (and
  reranker) is still required for web search to load, and include a Firecrawl
  scraper in the example.

* chore: fix import order drift (sort-imports)

* feat: add Keenable as a keyless scraper and select it without a pinned provider

The Keenable scraper landed in @librechat/agents#337, so wire the scraper
category the same way the search provider already is: `scraperProvider:
keenable` reads pages through Keenable's public fetch endpoint with no key
(a key only lifts rate limits, and the endpoint is overridden with
KEENABLE_FETCH_URL). Paired with `rerankerType: none` this makes a fully
keyless web-search stack possible for the first time.

Also closes the Codex finding on this PR: because none of Keenable's auth
fields are required, the generic auth loop skips it whenever it isn't pinned,
so a key submitted through the API-key dialog (which cannot pin a provider)
left the providers category unauthenticated. Keenable is now selected in that
case, gated on one of its values actually being present so installs that
configured nothing keep their current behavior. The scraper gets the same
fallback, additionally gated on Keenable being the resolved search provider,
so it never silently scrapes for another provider.

* fix: select the Keenable scraper from a supplied key, not only for Keenable search

The API-key dialog submits credentials and cannot pin a provider, so choosing
Keenable as the scraper while search stays on Serper/SearXNG/Tavily had no
effect: the unpinned-scraper fallback required Keenable to also be the resolved
search provider.

A supplied Keenable value now triggers it as well, which is the only signal the
dialog can send. The fallback still runs only when no keyed scraper
authenticated, and with neither trigger the category stays unauthenticated, so
a deployment that never configured Keenable is unaffected.

Note the fully keyless choice still cannot be expressed through the dialog:
Keenable's key is optional, so picking it with no key submits nothing at all.
librechat.example.yaml now documents pinning scraperProvider: keenable for that
case.

* style: Sort Keenable imports

* fix: Harden Keenable auth resolution

* fix: Preserve Keenable selection intent

* fix: Fail closed on invalid web search auth

* fix: close keenable auth gaps

* style: sort web auth imports

* fix: preserve web search selection integrity

* fix: isolate web search auth ordering

* fix: silence expected credential misses

* chore: bump agents sdk

* fix: preserve web search preference ownership

* fix: forward cleared Keenable endpoint

* style: sort web search hook imports

* fix: require intent for credential clears

---------

Co-authored-by: Ilya Bogin <ilya.bogin@keenable.ai>
2026-08-27 14:29:20 -04:00
Danny Avila
de59da9636
🎟️ refactor: Require Credentials for Local Image Access by Default (#15252)
* 🔐 fix: Protect Local Image Access by Default

* 🔐 fix: Scope Image Authorization to Active Sessions

* 🧹 style: Format Image Authorization Checks

* 🛡️ fix: Harden Image Avatar Authorization

* 🧭 style: Sort Image Authorization Imports

* 🔐 fix: Close Image Authorization Review Gaps

* 🧭 fix: Normalize Stored Avatar Base Paths

* 🏢 fix: Resolve Tenant Assistant Image Policy

* 🛂 fix: Enforce Effective Image Access Policy

* 🧹 style: Flatten Assistant Config Selection

* 🧷 fix: Preserve Image Access Compatibility

* 🪪 fix: Make Image Sessions Revocable

* 🏗️ fix: Move Image Session Policy Into API
2026-08-27 09:55:27 -04:00
Danny Avila
41ba808e6f
🧅 feat: Reveal Subagent Thread History and Event Detail on Demand (#15283)
* ✨ feat: Refine Subagent Thread Activity UX

* test: align subagent completion lifecycle

* chore: satisfy subagent panel static checks

* fix: harden subagent activity pagination

* fix: preserve paged subagent history

* fix: surface unrecoverable activity boundaries

* fix: rebase paged activity after live advances

* fix: bound child activity history state

* fix: preserve rebased child history order

* fix: retain unavailable child history boundary

* fix: fence child history cursor generations

* fix: preserve subagent history continuity

* fix: retain live turns during history rebase

* fix: close subagent history edge cases

* fix: harden subagent history phase transitions
2026-08-27 08:56:06 -04:00
Danny Avila
0d0d7d05bb
🧬 fix: Harden SAML Identity Binding (#15264) 2026-08-27 06:45:41 -04:00
Danny Avila
f0eda61638
🧵 feat: Unify Subagent Child Threads (#15261)
* feat: unify subagent child thread rendering

* chore: sort subagent UI imports

* fix: preserve bounded subagent thread context

* fix: render persisted child activity in unified timeline

* fix: bound durable subagent activity projections

* perf: cap subagent activity source scans

* chore: sort subagent thread imports

* fix: align completed child messages

* fix: bound selected subagent activity reads

* fix: bound child activity storage reads

* fix: preserve child receipt truncation state

* test: preserve projected receipt truncation

* test: align child completion e2e

* test: isolate Stable Diffusion logger mock
2026-08-27 06:04:13 -04:00
Danny Avila
44edcbe014
🧾 feat: Store Durable Event Actor Receipts (#15265)
* 🧾 feat: Store durable event actor receipts

* 📊 fix: Scope actor delivery metrics

* 🧱 fix: Close durable receipt recovery gaps

* fix: serialize actor action admission

* fix: retire terminal batch members

* fix: close actor terminal recovery races

* fix: reclaim lanes after terminal receipts

* fix: close actor receipt recovery gaps

* fix: preserve ambiguous legacy handling

* fix: recover legacy actor outcomes

* fix: admit leased actor actions

* fix: serialize actor admission recovery

* fix: close actor recovery races

* fix: token-fence actor admissions

* fix: preserve mixed-version actor fences

* fix: harden actor rollout and metrics

* fix: gate durable actor receipt rollout

* fix: enforce base actor receipt rollout

* fix: align experimental actor rollout
2026-08-27 06:00:43 -04:00
Danny Avila
52fa88fc54
📬 feat: Serialize Bound Actor Event Mailboxes (#15260)
* feat: serialize bound actor event mailboxes

* test: harden actor mailbox batch ordering

* fix: treat batch roots as mailbox authority

* test: pin mailbox replay semantics

* fix: preserve mailbox authority through settlement
2026-08-26 10:02:47 -04:00
Danny Avila
68fc46a055
🪢 feat: Resume Bound Event Actors from Checkpoint Forks (#15227)
* feat: resume event actors from checkpoint forks

* fix: fence event actor checkpoint uncertainty

* fix: satisfy event actor type contracts

* fix: make actor reconciliation recoverable

* fix: preserve event actor lifecycle transitions

* fix: fence event actor lifecycle outcomes

* fix: enforce event actor lifecycle ownership

* fix: retain event actor settlement proof

* 🔒 fix: Retain Event Actor Receipts Through Repair and Bound Their Journal

Repair and compensation deleted the reconciliation row they resolved, which
was the only durable proof that the invocation had already applied an external
action. A delayed duplicate owner could then reacquire the same invocation id
and repeat that action. Both resolutions now retire their receipt to `settled`
and record how it settled, so the same-id tombstone survives; `history_repaired`
and `action_compensated` still force a cold rebuild. Compensation undoes the
effect without re-authorizing the delivery, so a legitimate retry must arrive
under a new invocation id. A retried repair converges on its own receipt.

Bound the journal so a long-lived actor cannot grow its conversation document
without limit: a new fence is admitted only when no active lifecycle row
exists, so a capped push can evict nothing but the oldest settled receipts.

Stop shipping the unbounded source payload on every bound-child continuation.
It rode the delivery body regardless of the feature flag while the sibling
`fire` body deliberately sends event identity alone, so a large webhook payload
could push a previously working delivery past the chat route's body limit. The
actor binds an invocation from identity and never builds the prompt from it.

Blank the positional token map on warm continuations. It is derived from the
full DB history, while a warm run executes on checkpoint-restored state, so its
indices address different messages and the pruner never recounts them —
misattributing cached counts to the wrong messages in both directions.

Keep the replaced-claim exit on its cleanup path when preserving reconciliation
fails: the committing CAS already left a blocking row, so the failed status
upgrade costs provenance, not safety.

* 🧪 test: Pin the Warm Continuation's Map/Summary Asymmetry

Give the warm-continuation client test a populated token map and a real
cross-run summary so its assertions bite: the positional map must arrive
blank (checkpoint-restored state no longer matches DB-derived indices, and
the pruner never recounts a populated entry) while `initialSummary` must
pass through unchanged — it rides the system tail and summarizes
pre-boundary turns that were excluded from the very history the committed
checkpoint was built from, so blanking it would silently drop context no
warm run can recover.

* ⚖️ fix: Honor Compensation in Settlement and Age-Bound the Receipt Journal

A compensated receipt still tombstones its invocation id, but its external
effect was explicitly undone — the terminal handler nonetheless replayed
every settled lifecycle's stored action as authoritative and settled the
public outcome as applied, telling an action-aware source the operation
stands and suppressing the new-invocation retry compensation requires. The
handler now settles a compensated invocation as failed with an explicit
compensation error, overriding even fresh applied run evidence from a
replayed generation; verified and repaired receipts continue to replay
applied.

Receipt eviction is now primarily age-based: a stale same-id owner is
bounded by time, not by how many newer invocations settle, so the previous
count-only slice let a high-rate actor evict a tombstone while its delayed
duplicate owner could still wake and repeat the action. Admission prunes
only settled receipts older than a retention window that dwarfs every
generation, job, and delivery-retry lifetime, and the count cap is demoted
to a raised document-size backstop.

* 🔀 fix: Serialize Compensation Against Settlement and Never Evict Live Receipts

The terminal handler read its lifecycle snapshot, verified history, and then
settled the public outcome — so a compensation resolving the same receipt
during that window lost: the handler settled applied from its stale snapshot
and no retry could ever change the replay-identity-locked outcome. The
receipt's status CAS is now the serialization point: verification resolves
the receipt BEFORE settling, whichever transition wins determines the public
outcome, and a crash between resolve and settle converges through the
retained receipt's replay. The verified-replay probe requires the receipt's
own resolution, so a compensated receipt can never satisfy a verification
retry. This inverts the settle-before-receipt ordering deliberately: that
ordering guarded proof that resolution used to delete, and the receipt now
retains its full action proof through resolution.

The document-size cap is no longer an eviction quota. A receipt inside its
retention window is never discarded: when the journal holds a full cap of
unexpired receipts, new invocations are refused fail-closed until receipts
age out, making duplicate protection and document integrity simultaneous
invariants instead of a rate-dependent trade.

* 🎓 fix: Keep Skill-Bearing Event Actors on the Legacy Path

Skill primes are spliced into the message list directly ahead of the newest
message, and a warm continuation forwards only that newest message — so a
checkpoint-restored actor would keep serving the prime bodies baked in at
its last cold start and never observe an edited or newly attached skill.
Until the actor head carries a context fingerprint that forces a cold
rebuild when the agent's skill context changes, agents with always-apply or
manual skill primes stay on the legacy path, which re-primes fresh bodies
every turn: correct on every event, just never warm.

* 📜 fix: Gate Fork Mode on the Skills Capability, Not Just Request-Time Primes

History-derived re-priming was a third path into the same staleness class:
an actor that previously invoked a skill carries no request-time prime
arrays, yet primeInvokedSkills re-resolves that skill's current body from
history each turn and the warm slice drops the reconstruction — leaving the
checkpoint's old body active after edits. The fork gate now keys on the
priming hook itself (present exactly when the skills capability is enabled)
alongside the request-time arrays, so every skill-body path routes to the
legacy rebuild until #15235's context fingerprint restores warm
continuation for skill-bearing actors.

* 🧾 fix: Capture Applied-Action Proof at Tool Execution, Not After sendMessage

The executor read applied-action evidence from the run-step collection the
instant sendMessage resolved, but that collection is populated
asynchronously — an applied invocation could classify as actionless
(runSteps still empty while the tool result already streamed), discarding
its fork and stranding the actor cold while the terminal handler later
settled the same delivery as applied from the persisted evidence.

Authoritative proof is now recorded in graph context the moment the
expected tool executes: the request-owned recorder observes the tool-end
chain (which ToolNode dispatches synchronously with both input and output)
and applies the same fences as run-step evidence — exact tool name with the
MCP-suffixed form, the declared argument subset against the execution
input, an error-free result, and the background non-execution receipt
exclusion. readAppliedAction consults the receipt first; run-step
inspection remains the fallback for paths that bypass the tool-end chain.

Regression coverage reproduces the observed ordering: the real executor
commits the head from the receipt while run steps are empty, warm-continues
the next event, and never re-executes the action; recorder fences and the
receipt-first controller wiring are covered separately.

* 🎯 fix: Supply Execution Arguments to the Tool End Callback

The live Vertex + MCP canary exposed a contract mismatch the synthetic
fixtures hid: the ON_TOOL_EXECUTE execution path invoked its tool end
callback with output only, while the action recorder must verify the
declared argument subset against the execution input. The receipt never
qualified, every turn fell back to cold history rebuilds, and the
tournament advanced with zero actor heads and zero retained checkpoints
while looking successful.

The execution handler owns both halves at the same moment, so the fix is
at the source rather than a correlation store: ToolEndCallbackData gains
the executed call's input and every callback site passes tc.args. A
handler-level regression drives the real createToolExecuteHandler and
asserts the callback receives both fields; recorder regressions pin the
production shapes — an output-only tool end must starve an
argument-fenced receipt rather than trust an unfenced match, and still
qualifies a name-only expected action.

* 🕵️ fix: Mark Background Deliveries So They Cannot Impersonate Applied Actions

The background-claim callback reports the ORIGINAL tool's name for artifact
attribution on the poll turn that harvests a completed task. A name-only
expected action could therefore be impersonated by work some earlier turn
dispatched: the recorder would attribute that delivery to the current
invocation and commit a head whose state never contained the invocation's
own action. The run-step evidence path never had this hole — it sees the
poll tool's name — so the recorder must match its provenance discipline.

Delivery callbacks now carry an explicit backgroundDelivery marker set at
the one site that rewrites the name, and the recorder ignores marked
deliveries outright. Regressions pin both halves of the contract: the
delivery callback must carry the marker with the poll call's arguments,
and a marked delivery can never qualify even a name-only expected action.

* 🧿 fix: Version Invalidations, Keep Evidence Ahead of Output Policy, Gate Detachable Actions

Three closeout-round findings, each converted into an invariant.

Every legacy-path invalidation now advances a durable epoch — including for
headless and already cold-marked actors, where the marker alone leaves no
CAS-visible trace — and the actor-head CAS requires the epoch observed at
preparation. A concurrently prepared fork whose history predates an
intervening legacy turn can no longer commit past it; the commit reports an
ordinary conflict and journals.

Execution identity is now emitted before post-execution output policy: when
a side-effecting tool succeeds but its returned content is withheld by the
output filter, the callback delivers an outputFiltered receipt with blank
content — the recorder accepts it as proof (rejecting model-detached calls
it cannot distinguish through the blank shape), the artifact path never
sees it, and an applied action is no longer reclassified actionless and
re-executed on retry.

Background-capable expected actions stay off the fork path: dispatch
returns a launch handle every evidence fence correctly rejects, and the
completion is provenance-marked as another turn's work, so a fork would
settle actionless before the external effect lands with nothing to stop a
retry from dispatching it again. The gate mirrors the MCP-suffix name
matching of the evidence path.

* 🚧 fix: Seal the Whole Legacy Turn Behind a Second Epoch Advance

The epoch fenced only the legacy turn's start: a fork preparing after the
begin invalidation but before the turn's message persistence observed the
new epoch and cold marker, rebuilt from history that did not yet contain
the turn, and committed cleanly because nothing advanced the epoch again —
making the incomplete rebuild authoritative and clearing the marker.

Every legacy event turn now seals its invalidation at terminal persistence
with a second epoch advance, on the success, replaced-claim, and error
exits alike. Sealing deliberately carries no quiescence requirement — it
must succeed while a fork fence is active, because that is exactly the
mid-turn race it defeats — and a fork that already committed against the
begin epoch is healed the same way: the seal re-marks the head cold, so
the next event rebuilds with complete history. Seal failure never diverts
the turn's own exit; the begin bump still fences everything prepared
before the turn.

* 🔗 fix: Replace the Best-Effort Epoch Bump With a Durable Legacy-Turn Fence

The second epoch advance could not make a legacy turn atomic, and three
findings shared that root cause: two conditional updates left a headless
gap a fork could create the head inside; the error exit sealed before
saveErrorTurn made the error history durable; and any crash or failure
between persistence and sealing left an incomplete fork authoritative,
because the seal was best-effort and its failure was swallowed.

A legacy turn now carries one durable fence. A token is written before
execution by a single update-pipeline write — no two-write gap, and the
cold marker is applied only where a head exists via $cond/$$REMOVE. While
the token is present no fork may prepare (the adapter refuses) or commit
(the CAS requires its absence), because the turn's messages are not yet
durable. One atomic write clears the exact token and advances the epoch
once history is persisted — after saveErrorTurn on the error exit — and
success, replacement, and error exits all route through it.

Failure is now fail-closed rather than silent: a failed seal leaves the
token set, which keeps blocking forks and is logged as such, and a fence
abandoned by a crashed turn is reclaimed only once stale, advancing the
epoch and marking any head cold so the next event rebuilds from whatever
history actually survived.

* fix: serialize legacy event actor turns

* fix: close legacy actor fence ownership gaps

* fix: preserve legacy actor persistence fences
2026-08-26 08:39:00 -04:00
Ravi Kumar L
56f0cde9a5
🩻 feat(langfuse): add tenant export telemetry (#15247) 2026-08-26 07:38:30 -04:00
Marco Beretta
0383030817
⚡ perf: Avoid Parallel Full and Paged Prompt Loading on Startup (#15031)
Co-authored-by: Danny Avila <danny@librechat.ai>
2026-08-25 21:21:17 -04:00
Marco Beretta
57f3cb750b
🐇 perf: Bound MCP Agent Access Scan by MCP Referencing Agents and Share the Read-Through Cache (#15028)
* perf: bound MCP agent access lookups to MCP-referencing agents and share the registry read-through cache via Redis

Resolving MCP server access started from the full set of agents a user
can VIEW: the ACL distinct query materialized every accessible agent id
(400+ in affected deployments) and shipped the whole list back as an
$in filter, even when almost none of those agents reference MCP
servers. The calculation now starts from the agents whose
mcpServerNames are non-empty (covered by the existing
mcpServerNames_1_tenantId_1 index) and bounds the ACL query to those
candidate ids, so cost scales with agents that actually use MCP
servers. When no agent references any server, the agent-side ACL query
is skipped entirely. The same inversion applies to single-server
access checks, which now short-circuit on servers no agent references.

The registry's per-user read-through cache was process-local Keyv, so
every container recomputed access and no invalidation crossed
instances. It is now backed by standardCache (Redis when configured,
in-memory otherwise) with generation-based invalidation that orphans
entries in O(1) instead of scanning the keyspace, plus a process-local
memo to keep repeated per-request reads off the network.

Closes #14016

* style: fix import order in registry read-through cache files

* fix: harden MCP registry cache sharing per review

Encrypt the shared read-through store with encryptV2 so decrypted
oauth/apiKey credentials never leave the process in plaintext, and fail
open to a cache miss when an entry cannot be decoded.

Treat MCP_REGISTRY_CACHE_TTL=0 as a full cache bypass: entries derive
from ACL access, and without a TTL there is no bound on how long a
revoked user could keep receiving a stale map, in Redis or anywhere.

Move the invalidation generation tag to its own no-TTL store so it can
never expire before entries written under it, sweep expired memo
entries at most once per TTL window so one-time users cannot accumulate,
and settle the candidates and principals reads through one Promise.all
so a principals rejection can never surface as unhandled. The public
direct-server ACL read now starts alongside the candidate scan instead
of after it.

* fix: encrypt and harden the per-server MCP read-through cache

Route the per-server read-through cache through the same encrypted,
ttl-gated wrapper as the aggregate map, so decrypted credentials never
reach Redis in either cache and MCP_REGISTRY_CACHE_TTL=0 disables both
rather than storing ACL-derived entries forever. A null envelope keeps
cached-absent lookups distinguishable from misses, preserving the
existing negative-caching contract.

Shared-store failures now degrade instead of rejecting: entry and
generation reads fail open to a miss, writes fall back to the
process-local memo, so a Redis outage costs a recompute rather than
failing MCP requests. get and set recheck the generation after the
store round trip so an invalidation landing mid-flight cannot have its
pre-mutation value memoized for a full TTL window.

* fix: tenant-scope shared MCP cache keys and close read races

Shared read-through entries are now keyed with scopedCacheKey, which
appends the active ALS tenant, because the DB reads behind a miss are
tenant-filtered: without it a public or per-user lookup in one tenant
could satisfy another tenant's lookup from Redis. The single-flight map
partitions the same way.

The direct-server ACL read is chained off the principals promise so a
rejection handler is attached the moment it is created on both the user
and public paths, and all three initial reads settle through one
Promise.all. The generation recheck in the aggregate cache moved after
decode, so an invalidation landing during decryption still cannot pin
its pre-mutation value. The per-server cache exposes a single decoded
getEntry whose hit flag preserves negative caching, and an undecodable
entry deletes itself instead of shadowing the server until expiry.

* fix: miss on unreadable cache generation and parallelize the direct fetch

An unreadable generation tag now surfaces as a miss on get and a
memo-only write on set instead of falling back to generation zero,
which is a real first generation: a transient read failure could
otherwise revive an unexpired pre-invalidation entry for another TTL
window. The direct-server fetch, which only needs the ids resolved by
the first settlement, now runs concurrently with the agent ACL query
instead of after it.

* fix: fence cache fills across invalidations and scope them per tenant

A fill is now fenced by the generation observed at its miss: when a
registry mutation invalidates the cache while the caller is still
reading Mongo/YAML, the completed value is dropped instead of written
under the new generation, so a removed server or revoked ACL cannot be
resurrected for another TTL window by any replica.

The generation tag itself is tenant-scoped, matching the tenant-scoped
entries: a mutation in one tenant now evicts only that tenant's entries
instead of forcing every tenant back through the ACL and config reads.
Genuinely global events (operator config changes, lifecycle resets) use
the new invalidateAllGlobal, which trades a rare namespace scan for
cross-tenant eviction.

* fix: fence error-induced misses and decouple the agent ACL path

Store-read and decode failures now record their miss generation like an
ordinary miss, so a fill that lands after Redis recovers is still fenced
when an invalidation completed during the compute. The agent-side ACL
lookup chains from its actual inputs (candidates and principals) and the
direct-server fetch follows the ids immediately, so neither read waits
behind the other's settlement barrier.

* fix: bound fill fences by settlement and scope eviction per tenant

Miss fences no longer expire with the entry TTL: a fence lives until its
fill settles or a newer miss overwrites it, so a compute slower than one
TTL window is still fenced when an invalidation completed mid-flight.
The per-server cache gained the same protection through invalidation
versions: a fill finishing after its key was deleted, or after a
namespace clear, is dropped instead of repopulating the pre-mutation
value for every replica.

CACHE-tier mutations (YAML/App repository writes, reinspection, stubs)
now invalidate globally, since those entries are shared across tenants:
the per-server namespace clears and the aggregate map takes its global
path, while DB-tier mutations keep the tenant-scoped eviction. Tenant
invalidation also stops clearing the whole process memo: entries are
tagged with the tenant they were memoized under and only the acting
tenant's are evicted.

* fix: fence each cache fill by its own miss and pass the reinspection tier through

A miss now hands back a fill token capturing the generation (or, for
the per-server cache, the invalidation versions) it observed, and the
matching set fences on that token instead of a shared per-key marker.
Concurrent fills for the same key straddling an invalidation therefore
keep distinct fences: the pre-mutation fill is dropped even when a
newer miss for the same key is current, while the newer fill lands.
The shared marker maps are gone, along with their expiry and
overwrite semantics.

reinspectServer also threads its storageLocation into the invalidation
so a DB-backed reinspection takes the tenant-scoped path instead of the
global CACHE-tier eviction.

* fix: address PR review bot findings

chatgpt-codex-connector:
- share generation fences across cache replicas
- fence aggregate single-flight work by generation
- use an indexed MCP agent candidate predicate
2026-08-25 20:10:32 -04:00