Commit graph

13 commits

Author SHA1 Message Date
Danny Avila
08f9d7f7e0
🧵 feat: Stream a Background Subagent Child Through Its Own Generation Job (#18169)
* 🫀 feat: Let a Job Owner Keep a Silent Run Alive

A generation job's idle clock advances only with emitted chunks, so a run that
works without output for 20 minutes is reaped while it is still alive.
touchJob refreshes the idle clock and the storage lifetimes an append would
have extended, fenced on the generation epoch and limited to the replica that
owns the runtime.

* 🧶 refactor: Share One Fold From Child Frames to Content Parts

readSubagentContent folded public child frames into content parts inline.
The fold is now a synchronous function a second caller can use, so a frame
can be folded and published without yielding in between.

* 🏷️ feat: Mark the Generation Job of a Background Subagent Task

Job metadata gains subagentTaskId and subagentParentConversationId, set only
by the subagent task store. A marked job is a child-thread job, so it stays
out of the active chat list, and isBackgroundChildJob tells it apart from an
event-bound child, whose job the chat controller owns.

* 🧵 feat: Run a Background Child as a Resumable Generation Job

createChildGenerationHost opens a generation job keyed by the child thread id
and feeds it the child's public step frames. Text deltas that queue behind a
slow write merge instead of dropping, the terminal state is claimed before the
caller persists and the final event is published only afterwards, and no
stream failure can fail the child: the run and its result carry on without a
live view.

* 🪢 feat: Stream a Background Child Through Its Own Generation Job

The subagent task store takes a childGenerations dependency. Once a child's
thread and input row are durable it opens a job keyed by the thread id, feeds
it each public frame, refreshes its idle clock on every lease renewal, and
settles it around its own persistence, so a reader sees the final event only
after the result row exists. A cancel, a failure and a job aborted from
outside all end both the child and its stream.

endpoints.agents.subagentActivity.generationJobs turns it off. A stream that
cannot be opened or written never fails the child.

* 🔌 feat: Give the Server's Task Store Its Child Stream

The host wiring builds the child generation host from the generation job
manager and hands it to the subagent task store, so a background child's
stream is reachable in the running server.

* 🚧 fix: Refuse Chat Abort and Steer on a Background Child's Job

The chat abort route saves a user row and a partial response and upserts the
conversation, and the steer route queues text only the chat controller drains.
Both authorized on the job's owner alone, so either could act on a background
child's job and write a chat-shaped history into a view-only thread, or accept
a steer nothing would ever apply. Both now answer 409 for a job the subagent
task store owns; the child is stopped and steered through its control
endpoint.

* 🧹 fix: Evict a Finished Job Before a Live One at Capacity

The in-memory job store made room by deleting its oldest job whatever its
status, so a full store could end a run somebody was still waiting on. It now
takes the oldest finished job first. Also covers a job manager that shuts down
while a background child runs: the child still persists its cancellation.

* 🛰️ test: Read a Background Child's Stream From Another Replica

Against real Redis: a second replica resumes with everything the child has
streamed, receives only newer deltas live, then the final event; a child that
streams past the manager's 5,000-event early buffer before anyone attaches
still resumes whole on both replicas; and the owner's touch keeps a silent
child's job alive.

* 🧽 chore: Sort Imports in the Child Stream Files

* 🧪 test: Expect the Child Stream Lever in the Activity Defaults

* 📜 feat: Store What a Background Child Streamed as Standard Content (#18173)

A child's result row carried its text and a bounded activity projection, not
the content parts every agent message stores, so a reader had to rebuild the
child's output from the projection. The row now also carries what the child
streamed, on completion, failure and cancellation, in the standard shape.

endpoints.agents.subagentActivity.generationContentBytes bounds it (4 MiB by
default, 0 stores none). Content over the bound is left off, so the row always
fits and keeps its text and projection as before.

* 🛟 fix: Keep a Stalled or Failing Child Stream From Touching Its Child

Four ways the stream could still reach the run it only observes. A frame that
could not be copied threw into the child's own progress report. A job store
that never answered held the child's start behind job creation and its result
behind the terminal claim. And a stream that had stopped being written was
never ended, so its job read as a live child for twenty minutes after the
child finished.

Frames are now copied defensively, opening and settling are bounded, a job
that arrives after its child stopped waiting is ended, and a detached stream
is still touched and still settled with its full final event.

* 🧯 fix: Let a Child Stream Observe Its Run Without Ever Stopping It

Review round 1.

A failed Redis append retires the job's runtime by aborting its controller,
and the child stream forwarded every abort of that controller to the task
store as a cancellation. A stream-store outage therefore stopped the child's
model and tool work. The stream no longer signals the child at all: deletion,
shutdown and the control endpoint already stop a child through the task
store, and an aborted, reaped or retired job now only ends the live view.

The in-memory store's capacity eviction preferred any terminal job, including
one that still owed persistence, a host action, a schedule reconciliation or
a provider drain. Those are no longer treated as finished.

The chat abort route's refusal for a background child's job moves out of the
CJS route into createBackgroundChildAbortGuard, registered ahead of it.

* ♻️ fix: Replace a Dead Background Child's Job on Its Thread

A background child's job outlives an owner that crashed, or a settlement that
could not reach the store, for up to twenty minutes. A continuation of the
thread in that window was refused its own stream, because the dead job still
read as live. The caller of open holds the thread's lease, so a background
child's job still marked live there has lost its owner: it is replaced and its
readers are told to refetch. A live job that is not a background child's, such
as an event-bound child paused for approval, is still never replaced.

* 🧱 fix: Fail the Abort Guard Closed and Budget Content Against the Whole Row

Review round 2.

The abort guard passed the request on when its job lookup failed, so a
retried lookup in the route could run the chat-shaped abort on a child the
guard never saw. A failed lookup now ends the request at the error boundary.

A child's streamed content was checked against its own limit only. A result
row also carries the text and the transcript, so content that passed the
limit could push the document past MongoDB's 16 MiB and turn a successful
child into a failed one. The content is now the optional field: it is left
off whenever the row's large fields together would exceed the budget.

Also accounts for the guard's job read in the two abort route specs that
sequence it, and gives the limiter spec's api mock the guard.

* 🪶 refactor: Settle a Child Stream Once and Fold a Frame It Cannot Copy

The task store settles a child's stream on its outcome path and again in its
cleanup. The second call made another terminal claim that could only lose;
the stream now remembers that it claimed and only persists. A frame that
cannot be copied is still folded from the original, so the stored content
stays whole when the stream detaches over it.

* 🧪 test: Cover the Child Stream's Failure Branches and the Content Fold

The change-risk report showed the stream's fail-open branches untested: a
terminal claim that lands after its wait, a claim or a final publish that
throws, the settle timeout, the backlog limit, and a rejected touch. Each now
has a case. The shared content fold gets direct cases for a closed step, a
commentary phase and an applied steer.
2026-11-15 08:35:08 -05:00
Dustin Healy
7dd4fafdf3
📬 feat: Persist Channel Events and Delivery Receipts (#17595)
* 📬 feat: Persist Channel Events and Delivery Receipts

* fix: enforce channel run identity and fenced recovery

* fix: fence and erase channel state during account deletion

* refactor: share channel owner admission with private persistence

* refactor: name channel account-erasure boundary explicitly

* fix: prove channel identity and delivery authority at storage boundaries

* refactor: enforce channel delivery protocol and terminal lifecycle

* fix: enforce channel tenant and operation boundaries

* fix: initialize channel config from its composition owner

* test: verify config installs channel storage reader

* test: complete config initializer contract in model mocks

* test: include channel config reader in migration fixture

* test: await SSE subscription before advancing keepalive time

* test: Observe and Drain Concurrent Fixture Work
2026-11-10 21:03:57 -08:00
Marco Beretta
79b836d79f
📃 fix: Enforce Terms Acceptance on the Server and Mark Backfilled Consent (#17063)
* fix: Enforce Terms Acceptance on the Server When modalAcceptance Is On

* fix: Mark Backfilled Terms Acceptance Timestamps

* test: Cover Server-Enforced Terms Acceptance in the Mock Harness

* fix: Gate Management Routes and Skill Sync and Resolve Terms Policy Fail-Closed

* fix: Evict Cached Auth Documents When Terms Acceptance Is Reset

* fix: Guard Skill Sync Behind Terms Acceptance for Every Skill List Route

* fix: Move Terms Skill-Sync Guard Into the API Package and Evict All Users on Reset

* fix: Batch Terms Reset Cache Eviction and Sign the Spec User In Fresh

* fix: Let Stop Through the Terms Gate, Refresh Startup Config, and Wait Out Failed Eviction

* refactor: Narrow the Terms Exemptions and Share One Refusal Path Across Clients

* refactor: Compose the JWT Terms Gate Through an API Package Helper

* style: Sort Imports for the Shared Terms Refusal Helper

* test: Expose the Terms Gate Helper in the JWT Auth Spec Mock

* test: Cover the Terms Refusal Code on Pre-Stream Upload Errors
2026-10-14 17:45:28 +02:00
Marco Beretta
4afd5e17f8
💓 feat: Keep Quiet Chat Streams Alive Behind Idle-Timeout Proxies (#16904)
Some checks are pending
Backend Unit Tests / Build packages (push) Waiting to run
Backend Unit Tests / Codegraph select (push) Waiting to run
Backend Unit Tests / TypeScript type checks (push) Blocked by required conditions
Backend Unit Tests / Circular dependency checks (push) Waiting to run
Backend Unit Tests / Tests: api (shard 1/3) (push) Blocked by required conditions
Backend Unit Tests / Tests: api (shard 2/3) (push) Blocked by required conditions
Backend Unit Tests / Tests: api (shard 3/3) (push) Blocked by required conditions
Backend Unit Tests / Tests: data-provider (push) Blocked by required conditions
Backend Unit Tests / Tests: data-schemas (push) Blocked by required conditions
Backend Unit Tests / Tests: @librechat/api (shard 1/4) (push) Blocked by required conditions
Backend Unit Tests / Tests: @librechat/api (shard 2/4) (push) Blocked by required conditions
Backend Unit Tests / Tests: @librechat/api (shard 3/4) (push) Blocked by required conditions
Backend Unit Tests / Tests: @librechat/api (shard 4/4) (push) Blocked by required conditions
Codegraph E2E Votes / vote (full suite) (push) Waiting to run
Docker Dev Branch Images Build / publish (push) Waiting to run
Frontend Unit Tests / Client build recovery regression (push) Waiting to run
Frontend Unit Tests / Codegraph select (push) Waiting to run
Frontend Unit Tests / Build packages (push) Waiting to run
Frontend Unit Tests / TypeScript type checks (client) (push) Blocked by required conditions
Frontend Unit Tests / Tests: @librechat/client (push) Blocked by required conditions
Frontend Unit Tests / Tests: Ubuntu (shard 1/2) (push) Blocked by required conditions
Frontend Unit Tests / Tests: Ubuntu (shard 2/2) (push) Blocked by required conditions
Frontend Unit Tests / Vite build verification (push) Blocked by required conditions
2026-10-09 10:29:45 +02:00
lia-by-librechat[bot]
73d704656e
📔 feat: Preserve Owner Text During PII Redaction (#16311)
* 📔 feat: Preserve Owner Text During PII Redaction

* 🧵 fix: Keep Ordinary Chat Views Outside Private Text Provider

* 🧷 fix: Preserve Owner Text Across Copy and Edit Paths

* 🔏 fix: Protect PII on Denial and Abort Paths

* fix: preserve protected text through denial, stop, and owner reload

* test: cover preliminary private metadata in disconnect fixture

* test: carry preliminary metadata in resume controller fixture

* test: preserve Stop prerequisite identity in tenant fixture

* fix: keep protected owner text out of search and exports

* fix: preserve trusted PII placeholders through provider inspection

* fix: coordinate protected turns across Stop and native sharing

* fix: Preserve Private Text Admission and Query Lifecycles

* fix: Track Private Query Effect Dependencies

* fix: Fence Protected Admission and Screenshot Targets

* fix: Share Protected Admission and Preserve Native Token Trust

* test: Wait for Private Message Concurrency Indexes

* fix: Preserve Protected Admission Across Recovery and Titles

* fix: Coordinate Protected Root Admission and Early Stop

---------

Co-authored-by: Lia <lia@librechat.ai>
2026-10-02 11:43:10 -04:00
Marco Beretta
5b84c4b57f ⏳ feat: Add Ephemeral Retention Mode for Forced Temporary Chats (#13811)
* feat: add ephemeral retention mode for forced temporary chats

Adds RetentionMode.EPHEMERAL, which forces every newly saved chat
temporary: hidden from history and search, given the temporary-chat
expiration, and with the toggle locked on.

Forces isTemporary and applies the temporary expiration in the normal
conversation and message save methods, routes Assistants/Threads
messages through the normal message save, applies the all-data
retention policy to file uploads and shared links, and stamps imports,
forks, and duplicates. Locks the frontend toggle and keyboard shortcut
behind an administrator-enforced badge.

The default temporary mode and the existing all mode are unchanged.

* fix: enforce ephemeral retention when resuming paused chats

* fix: preserve forced retention across pauses and assistant turns

* test: Use Real Module IDs for Installed Package Mocks

* test: Wait for Resumed Retention Persistence

* test: Control Mongo Index Setup in OpenID Fixture

* test: Close Mobile Drawer Before Retention Resume Scenario

* fix: Store Responses API Messages Under the Retention Write Context

Input and output messages were saved with the raw request, so under
ephemeral retention they missed the forced-temporary classification the
conversation received. Route both through getConversationWriteContext
and announce the stored reply as saved.

* fix: Align Resumed Responses With the Parent Retention Deadline

A resumed response was saved with a fresh forced-temporary window while
the re-stamp kept the conversation's earlier deadline, so a chat that
paused again could leave its response stored after the conversation
expired. Stamp the saved response with the parent's deadline.

* fix: Clear Released Bookmark Tags When Forcing a Chat Temporary

stampForcedRetention released a converted chat's bookmark counts but
kept its tags, so deleting that chat before TTL removal decremented the
same tags again and reduced counts owned by other visible chats. Clear
the tags in the conditional conversion update that releases them.

* fix: Store No Bookmark Tags on Forced-Temporary Imports

An ephemeral import, fork or duplicate skipped its tag count increment
but kept the source tags, so deleting the copy before TTL removal
decremented counts owned by other visible chats. Store the same tags the
import counts, which is none for a forced-temporary copy.

* test: Provide the Forced-Temporary Retention Helper to the Header Stacking Mock

* test: Reuse One Admin Login per Worker in Ephemeral Retention Scenarios

* fix: Keep Legacy Temporary Chats Temporary When Forking Under All-Data Retention

* fix: Convert a Resumed Chat Under Ephemeral Retention Before It Can Re-Pause Empty

* fix: Treat a Reloaded Pre-Policy Run as Temporary Once Retention Is Forced

* fix: Stamp Messages with the Deadline a Concurrent Retention Stamp Stored

* fix: Wait for the Startup Config Before Rebuilding a Resumed Submission

* test: Wait for the Resumed Response Row in the Ephemeral Resume Scenario

* test: Provide the Real Command Executor Stamp to the Callbacks Mock

(cherry picked from commit 02a70e9c58)

Original-PR: https://github.com/LibreChat-AI/LibreChat/pull/13811
2026-10-01 03:46:19 +00:00
lia-by-librechat[bot]
55f3187296 🥢 fix: Fit MCP App Snapshots Within Message Documents (#16299)
* 🥢 fix: Fit MCP App Snapshots Within Message Documents

* 🥢 fix: Resolve Message Model Only at App Write Time

* 🥢 style: Sort MCP App Snapshot Persistence Imports

* 🥢 fix: Centralize MCP App Message Admission Across Writers

* 🥢 fix: Inject the App Budget Reader Without a Config Cycle

* test: Cover App Budget Initialization Across Mocked Hosts

---------

Co-authored-by: Lia <lia@librechat.ai>
(cherry picked from commit 411bb5444f)

Original-PR: https://github.com/LibreChat-AI/LibreChat/pull/16299
2026-10-01 03:46:17 +00:00
Dustin Healy
aec2121923
🔐 feat: Add M2M Authentication for Agent Management (#15555)
* feat: add M2M authentication for Agent Management

* fix: Harden Agent Management authentication

* fix: Preserve OIDC authentication safeguards

* fix: Normalize management principal IDs

* fix: Bound OIDC signing key lookups

* fix: Allow non-browser Agent Management clients
2026-09-03 20:07:00 -04:00
Danny Avila
e7f1838515
⚡ feat: Reliable Interrupt & Steer Escalation and Recovery (#14558)
Some checks are pending
Docker Dev Branch Images Build / build (Dockerfile, lc-dev, node) (push) Waiting to run
Docker Dev Branch Images Build / build (Dockerfile.multi, lc-dev-api, api-build) (push) Waiting to run
GitNexus Index / index (push) Waiting to run
GitNexus Index / post-index (push) Blocked by required conditions
* feat: surface interrupt-steer escalation on waiting messages

The interrupt & steer feature shipped reachable only through the
composer chord, the send-button hovercard, and the composer button; a
message already waiting (queued for after the run, or steered and
parked at the next tool boundary) had no path to it. Both waiting
surfaces now carry one:

- Queued rows get an icon-only ZapOff escalation button beside the
  existing Steer primary. It routes through sendQueuedNow, which now
  takes a preempt option on its live-run path. The tooltip teaches the
  composer chord, derived through resolveComposerKeyDown so a rebound
  or yielded chord is never advertised.
- In-flight steer bubbles get an "Interrupt now" overflow entry with
  the same race rules as Edit: reclaim first, and only a `reclaimed`
  outcome resubmits (via retrySteer with preempt, swapping the chip
  for an interrupting one). `applied` and run-ended-mid-reclaim
  outcomes stop at the existing informational toasts, so the words can
  never land twice. Not offered on a steer already preempting.
- Every during-run overflow menu gains an "Always interrupt instead"
  toggle for steerInterruptsByDefault, next to the existing steer/queue
  default toggle. MenuEntry supports disabled for the new entries.

Only one interrupt can be unresolved at a time: while one preempt is
pending (or the run is paused on approval, where the server 409s),
every escalation control disables instead of racing the same seal.

Ten new tests across both surfaces; 381 green in the affected suites.

* fix: lock escalation across its reclaim window, keep the paused control visible, label as steer

Codex round 1, all three findings.

P2, escalation race. The single-interrupt invariant had a window between
clicking "Interrupt now" and the reclaim resolving, where no preempt
chip existed for the chip-derived gate to see: two bubbles escalated
back-to-back could both resubmit. A shared escalating flag (Jotai,
per-conversation) now covers the window and disables every escalation
control on both surfaces, and a fresh recheck before resubmitting
catches an interrupt armed elsewhere meanwhile (composer chord, queued
row); those words re-home to the queue with an informational toast
instead of breaking the invariant.

P2, unreachable paused state. canSteer is defined as
hasRealConvoId && !pausedOnApproval, so gating the button on canSteer
removed it exactly when it was meant to render disabled; the test only
passed on an impossible stub combination. The render gate is now
duringRunActive && (canSteer || pausedOnApproval), and the test uses the
real invariant.

P2, label semantics. "Interrupt & send now" borrowed the name of the
hard-abort action; this one preserves the partial answer and steers.
Renamed to "Interrupt & steer now" (com_ui_interrupt_steer_now).

Both behavior fixes counterfactually verified; 384 tests green across
the affected suites.

* fix: disable bubble escalation while the run cannot accept a steer

Codex round 2, one P2. Answer mode (ask_user_question) sets
duringRunActive false while pausedOnApproval stays false, since that
flag only detects approval-bearing tool calls. The bubble's escalation
entry stayed enabled there, so clicking it cancelled a healthy waiting
steer and the preempt resubmission bounced off RUN_PAUSED, degrading
the words to the queue. The entry now also disables on
!duringRunActive, matching the queued-row control's gate.

Counterfactually verified: reverting the gate fails the new
answer-mode test.

* fix: recheck live run state after the reclaim, not just at the click

Codex round 3, one P2, and it is the round-1 recheck principle applied
one level deeper: the entry-time disable cannot see a run that pauses
(tool approval, answer mode) while the reclaim round-trip is in flight,
and the .then closure held the render's stale steering controls, so the
resubmit would fire into a RUN_PAUSED rejection after the reclaim had
already surrendered the steer's boundary slot.

The escalation continuation now reads the LIVE controls through a
latest-ref: if the run can no longer accept a steer, the words re-home
to the queue with an informational toast instead of resubmitting, and
the resubmit itself also goes through the live controls.

Counterfactually verified: reading the stale closure instead of the ref
fails the new mid-reclaim pause test.

* refactor: make escalation one atomic server-side arm, in place

Codex round 4: four P2s, every one an interleaving of the same window —
escalation as reclaim-then-repost is a compound, non-atomic operation
whose continuation must revalidate the world (FIFO position lost, ref
assigned too late, no run fence, competing bubble actions). Rounds 1-3
patched that window with a lock and rechecks; round 4 shows the window
itself is the defect, so this removes it instead of guarding it again.

Escalation is now POST /chat/steer/arm: the server flips preempt on the
EXISTING queued item in one atomic store op (new IJobStore.armSteer; a
decode-patch-encode LSET Lua on Redis, an in-place mutation in memory),
fenced to the validated generation and refused once the queue closes.
The handler mirrors the steer POST's preempt contract exactly: durable
flag gated on the owner's recorded capability, volatile requestPreempt
fire-and-forget because the durable flag is the truth resume/handover
re-arm from.

By construction this resolves all four findings: FIFO survives (the
item never moves; the whole queue still drains in instruction order at
the seal), there is no continuation to hold stale controls, the store
op is fenced to the original run, and a competing Edit/Queue/Cancel
either beats the arm (armed:false, chip untouched) or operates on the
armed item, whose cancel already disarms.

The client escalation entry becomes one mutation: armed:true relabels
the chip in place (same steerId, same position), PREEMPT_UNSUPPORTED
and lost races toast honestly, and the round 1-3 machinery — the
escalating lock atom, the latest-ref, the post-reclaim rechecks and
their two toast strings — is deleted rather than extended.

Verified: 7 new handler tests on the real in-memory manager (including
FIFO preservation and the stale-generation fence), 2 Redis integration
tests against real Redis (in-place arm keeps order and every field;
missing/stale/closed all refuse), client suites 396 green.

* fix: decide capability inside the atomic arm, neutralize the lost-race toast

Codex round 5, both findings, both edges of the new arm design rather
than its mechanism.

P2, capability TOCTOU. A HITL resume on a rolling deploy rewrites
preemptCapable for the SAME generation, so the handler's read could go
stale between validation and the flag flip, arming a steer the live
owner cannot seal. armSteer now returns armed | missing | incapable,
with the owner's live capability part of the same atomic predicate as
the generation fence (HGET preemptCapable inside the Lua; the flat job
field, not a metadata blob — the in-memory store reads the same field).
The handler's pre-check is deleted rather than kept alongside; the
store predicate is the single source. New handler test rewrites the
capability after queueing and expects PREEMPT_UNSUPPORTED with the item
left unflagged; the Redis guards test now asserts the incapable refusal
against real Redis.

P2, ambiguous toast. armed:false covers injected, cancelled, re-homed,
and run-over alike, so telling the user the message "already reached
the agent" claimed one specific outcome. The lost-race branch now uses
a neutral message (com_ui_steer_arm_lost_race) and defers to the events
for what actually happened.

* fix: flip the escalation lock synchronously before the arm request

Codex round 6, one P2. Round 4 deleted the escalating flag along with
the reclaim continuation it guarded, but that left the one-interrupt
gate blind during the arm request's own round trip: the chip-derived
check cannot see an arm until its response relabels the chip, so on a
slow connection two bubbles could both arm before either response
landed. Double-arm is harmless server-side now (the run seals once and
drains the whole queue in order), but every escalation control
advertises "one interrupt at a time" by disabling, and the controls
must tell the truth.

The per-conversation escalating flag returns as a pure UX gate: set
synchronously at click, before the mutation, cleared on settlement, and
folded into interruptPending on both surfaces. Unlike its round 1-3
ancestor there is no continuation behind it to guard and no recheck to
pair with it.

Counterfactually verified: without the synchronous set, the two-bubble
race test arms twice. 207 tests green across the Chat Input suites.

* test(e2e): cover escalation of waiting messages through the real seal

Three mock-harness tests on E2E_SLOW_REPLY, a 160-chunk stream with no
tool boundary, so an in-thread steer part can ONLY come from a genuine
mid-stream seal — which makes each test a behavioral proof rather than
a UI check:

- Queued row escalation: the ZapOff button turns a waiting queued
  message into a preempt-armed steer (202 echoes preempt: true) that
  seals and injects, where the sibling steering.spec test proves the
  unescalated path waits for run end instead.
- Bubble in-place arm: an ordinary steer (202 with no preempt echo)
  waits as a bubble, POST /chat/steer/arm answers armed: true, the
  bubble relabels in place (same single bubble, same text, escalation
  no longer offered on reopen), and the armed steer seals mid-stream.
- Always-interrupt toggle: flipped from a waiting row's overflow menu,
  plain Enter now produces a preempt: true steer that seals in the SAME
  run, and the menu offers the way back. An afterEach clears the
  localStorage preference so a mid-test failure cannot leak
  preempt-by-default into the rest of the serial suite.

All three verified locally through the full harness (real backend, mock
LLM, seeded DB): 3 passed in 27s.

* feat: dedicated escalation arrow + shortcut, menu split into actions and preferences

The escalation was still half-hidden: the bubble only offered it inside
the overflow menu, and the tooltip taught the composer chord, which does
a different thing (interrupts with typed text, not this chip). Three
changes make it a first-class command:

- A shared EscalateNowButton (circular arrow, ghost-bordered like the
  composer's interrupt control) is always visible on BOTH surfaces:
  beside each queued row's Steer primary and on every waiting steer
  bubble next to its menu. It disappears once a steer is interrupting.
- A dedicated registry shortcut, escalateSteer (Cmd/Ctrl+Shift+.),
  editing-allowed and rebindable like every other action. Deliberately
  NOT an Enter chord: the composer owns every Enter chord, and the
  yield design rests on no default binding using Enter besides submit.
  Its handler clicks the newest enabled arrow control (bubbles beat
  queued rows), so the shortcut can never diverge from the button, and
  the arrow's tooltip teaches THIS command via the registry display.
- The overflow menus separate one-off actions from sticky behavior
  changes: Edit, Cancel, Queue, then a smaller "Preferences" section
  holding the queueing and always-interrupt toggles, each with the
  standard InfoHoverCard reusing the Settings panel's descriptions.
  "Interrupt & steer now" leaves the menu entirely.

386 client tests green, including a menu-structure test locking the
order and the absence of the escalation entry; bubble escalation tests
drive the visible arrow. The e2e spec's bubble test now clicks the
arrow, and a fourth test drives the dedicated shortcut end to end
through a real mid-stream seal.

* style: bind the escalation arrow to its message (variant A anatomy)

Two same-weight circles in a row read as one control group, leaving the
arrow's ownership ambiguous, and a floating arrow stops meaning anything
once several messages stack. The shared control now carries variant A's
anatomy: a thin divider binds a small SOLID arrow (filled, inverted) to
the message region on its left, and the menu ellipsis stays a bare
glyph, so the two affordances can no longer blur together — and the
divider+arrow pairing repeats cleanly per chip at N messages.

* chore: drop the unused within import CI lint caught

* fix: advertise the escalation shortcut only while the control is live

Codex on the e2e head, one P2: the tooltip appended the chord hint even
while the button was disabled, advertising a shortcut that does nothing
during an approval pause. The flagged control (InterruptNowButton) was
since replaced by the shared EscalateNowButton, which inherited the
pattern; the successor now omits the chord whenever the control is
disabled, matching the rule the during-run hovercard already follows.

* fix: harden steer escalation lifecycle and recovery

* test(e2e): disambiguate accessible steer preferences

* test: align abort persistence coverage with prerequisites

* chore(i18n): remove obsolete steer race message

* chore: normalize imports across steering changes

* test: exercise stream integration on Redis Cluster

* test: scope HITL checkpoints to generation

* test: fix cluster cleanup and locale policy

* fix: keep escalation visible during ask pauses

* fix: fence recovery downgrade and stale predecessors

* fix: require generation owner abort acknowledgement

* fix: validate delayed preempt arms

* test: align final escalation fixtures

* fix: preserve in-memory predecessor abort handoff

* fix: restore controls for recovered queued messages

* test: cover recovered queue controls

* fix: close final steering review gaps
2026-07-31 20:07:56 -04:00
Danny Avila
73699b5c25
⚡ perf: Reduce Agent Chat Startup Latency (#14423)
* perf: reduce agent chat startup latency

* test: align Redis stream readiness assertions

* perf: overlap remaining agent startup work

* perf: persist initial agent job metadata atomically

* test: add agent startup latency benchmark

* fix: harden resumable agent stream lifecycle

* fix: isolate replacement stream lifecycles

* fix: preserve terminal stream epochs
2026-07-25 07:58:20 -04:00
Danny Avila
9bb351ad9c
🧭 feat: Mid-Run Steering and Queued Messages for Agent Runs (#14220)
* 🧭 feat: Mid-Run Steering and Queued Messages for Agent Runs

Steering: submit a message while a run is generating; the server queues
it in the job store (cross-instance) and a run-scoped PostToolBatch hook
injects it into graph state at the next tool-batch boundary, records an
inline 'steer' content part on the response (replayed as a user message
on later turns), and streams on_steer_applied to the client.

Queuing: messages composed during a run auto-send as normal follow-up
turns after clean completion (one per final event, FIFO); user aborts
leave them as chips unless armed by interrupt-and-send.

Requires hook injectedMessages support in @librechat/agents
(danny-avila/agents#299); hard-gated via a capability probe so older
SDKs 501 the steer route instead of draining and dropping messages.

* 🧵 fix: Harden Steering Against Finalization Races and Route Guard Gaps

Addresses local Codex review findings on the steering feature:

- Close-and-drain the steer queue atomically at finalization (final event,
  abort) so a steer POST racing teardown is rejected instead of 202-ACKed
  and then silently cleared; the closed flag lives on the job hash and is
  reset when a replacement job reuses the stream id.
- Clear inherited steer queues on createJob — a job replacement must not
  drain the replaced run's messages.
- Keep steers queued across a HITL pause instead of draining them into
  ephemeral client state: resumeState re-seeds chips on reload and the
  resumed run injects them at its first tool boundary (steers key TTL now
  extends to the approval window; on_steers_pending event removed).
- Queue the NO_ACTIVE_RUN steer fallback while the final SSE is still
  settling — a direct send would be dropped by ask()'s in-flight guard.
- Reconcile the 202 ACK against on_steer_applied events that beat it over
  the SSE, so a chip can't be re-minted after its removal event passed.
- Allow the per-send Steer override when the default action is queue.
- Apply the configured message rate limiters and the PII filter to
  POST /chat/steer — a steer is model-bound user text.

* ✅ ci: Assert Steering Capability Probe Against the Installed SDK

CI installs the published @librechat/agents pin (pre-injectedMessages),
where isSteeringSupported() is legitimately false — the probe test now
asserts it mirrors the installed SDK's capability flag instead of
hardcoding the capability-bearing build's value. Verified against both
the published 3.2.61 dist and the agents#299 build.

* 🛟 fix: Preserve Steer Text Across Run-End, Error, and Abort Races

Codex round 2 (4 P2s):
- Applied-steer-id set survives run end (capped at 100) and converted
  ids join it, so a 202 ACK that lands after final/abort drops its chip
  instead of re-minting a stranded pending one.
- Failed runs no longer strand acknowledged chips: both error paths
  convert local pending chips to queued follow-ups (chip text is
  client-local), and the server closes the steer queue before emitting
  the error so a racing steer POST gets 404 fallback instead of a 202
  whose payload dies with the job.
- sendQueuedNow keys on steer availability, not the default action —
  send-now on a queued chip is an explicit override for queue-preferring
  users.
- Stop path consumes pendingSteers from the abort HTTP response as a
  fallback for the SSE final event it may close before processing;
  conversion is deduped so double delivery is a no-op (shared
  useSteerConvert hook).

* 📎 feat: Carry Attachments Through During-Run Queued Messages

Steering stays text-only (SDK injection, inline STEER part, and replay
are all text), so a during-run submit with media now queues the whole
message as one unit instead of silently stranding the files:

- QueuedMessage gains `files`; composer attachments are consumed into
  the queued item at queue time (steerFromComposer / queueFromComposer /
  interruptAndSend), fixing the latent hazard where lingering composer
  files glued onto whatever `ask` vacuumed up next.
- Enter-steer with attachments degrades to queue with an explanatory
  toast; the per-send menu routes through the same composer-aware
  wrappers.
- The drain and sendQueuedNow pass the item's files as `overrideFiles`;
  media items never steer (send as a normal turn when idle, re-front
  otherwise). ask() no longer clears composer state for caller-supplied
  overrideFiles — only regenerate keeps that behavior.
- During-run submits hold while uploads are in flight, mirroring the
  send button's filesLoading gate; queued chips show a paperclip count.

* 🎛️ feat: Rework During-Run Chips into Action Rows

Full-width rows above the composer (reference-UI parity): each queued
message shows a primary Steer/Send-now action, delete, and a "…" menu
with Edit message (restores text + attachments into the composer) and a
Turn on queueing/steering toggle that flips the Enter default. Steer
rows share the layout with status text; failed steers keep retry /
edit / queue-convert. The per-send menu gains the same default toggle.
Queued file refs now retain filename + bytes so edit-restore rebuilds
real composer entries (draft-recovery shape).

* 🖇️ feat: Steer With Attachments (Multimodal Mid-Run Injection)

Steering now carries media end-to-end instead of degrading to queue:

- The steer POST accepts sanitized attachment refs (cap 10; only
  file_id is trusted — the drain re-fetches owner-scoped and re-derives
  everything else). SteerQueueItem/TPendingSteer/SteerContentPart carry
  `files` refs; encoded data is never persisted or queued.
- New api/server/services/Files/steering.js decouples attachment
  building from the request path: encodeSteerContent reuses the exact
  per-turn pipeline (addFileContextToMessage + processAttachments'
  single-pass categorize/encode, SDK formatMessage assembly,
  prependFileContext for extracted text) with zero new encoding code.
  buildSteerMedia feeds the drain hook's new buildMedia seam (any
  failure degrades that steer to text-only — words always land);
  stampSteerPartMedia re-encodes past steer parts per turn with ONE
  batched owner-scoped fetch and stamps a transient `media` array,
  replaced immutably so it can never leak into a save. Replay honors
  resendFiles like regular message media.
- The SDK's formatAgentMessages (the formatter agents actually use)
  gained the steer replay branch on the PR branch; the local
  formatMessages.js branch now mirrors the media preference.
- Client: steerFromComposer consumes composer files into the POST,
  chips/seeding/conversions carry files everywhere (retry, queue
  convert, abort/error recovery), queued media items steer for real,
  and SteerBubble renders the steered attachments inline.

* 🧵 fix: Harden Steer Recovery Races and Drain Isolation

Codex round 3 (7 fixes):
- A 202 ACK landing after the run ended converts straight to a queued
  follow-up (server queue is gone; no event will ever resolve a pending
  chip for a finished run). Covers stream errors with in-flight POSTs.
- A Stop that lands pre-completion can arrive as a final with
  unfinished:true and no aborted flag — runEnd now treats it as aborted
  so queued messages are not auto-sent against the user's Stop.
- Leftover-steer conversion merges chronologically by createdAt instead
  of appending, preserving the order the user composed.
- Auto-drained queued messages pass explicit (possibly empty)
  overrideFiles/overrideQuotes/overrideManualSkills: a drain can no
  longer vacuum up files, quotes, or skill picks staged in the composer
  for the user's NEXT message (ask() treats overrideFiles != null as
  authoritative).
- Failed-steer Retry and resume-on-load chip restoration keep the
  steer's attachments.
- The job-replacement guard moved INSIDE the store's atomic
  drain/close-and-drain (Lua createdAt compare; in-memory equivalent):
  a stale run's hook or finalization can neither consume, close, nor
  steal a replacement job's steer queue, and the drain hook drops its
  separate check-then-drain round trip.

* 🧰 refactor: Typed Steer Controller, Single-Query Media Pass, Round-4 Fixes

Codex round 4 + efficiency tightening in one pass:

- Moved the steer guard ladder (validation, file sanitization via a
  shared toSteerFileRef picker, ownership/tenant checks, status-guarded
  enqueue) into packages/api as handleSteerRequest; api/steer.js is now
  a thin wrapper. Ladder covered against the REAL in-memory job manager
  in request.spec.ts; the api spec pins only the wrapper contract.
- Folded the steer replay stamp into the turn's ONE historical-files
  query: collectHistoricalFileRefs also gathers steer-part refs, the
  owner-scoped doc map rides client state, and stampSteerPartMedia
  consumes it (no second round trip) while encoding parts in parallel.
- Stamped steer media now counts against the run budget (existing
  multimodal counter over the non-text parts, folded into
  indexTokenCountMap/promptTokens after the stamp).
- Steer route runs the PII filter BEFORE moderateText, matching chat.js
  so blocked sensitive text never reaches the external moderation API.
- Interrupt & send survives the abort-response-beats-SSE-final race:
  stopGenerating writes the run-end signal itself when the one-shot
  interrupt flag is armed and no signal landed (double-fire safe).
- Resume reconciles chips against the server's still-queued list even
  when EMPTY, clearing chips for steers applied while disconnected.
- The local formatter's steer flush preserves non-text assistant parts
  (array-content AIMessage) instead of folding to text.

* 🔒 fix: Replay-Aware Capability Gate and Round-5 Race Closures

- isSteeringSupported now requires BOTH halves of the SDK contract:
  injection (HOOK_INJECTED_MESSAGES_CAPABLE) AND replay
  (ContentTypes.STEER, shipped in the same SDK commit as the
  formatAgentMessages steer branch). An SDK that can inject but not
  replay 501s the steer route — no release window can create steer
  parts that would leak into provider-facing assistant content.
- The local formatter mirrors the SDK's anchor reset: a post-steer
  tool_call mints a fresh AIMessage instead of attaching to the
  pre-steer anchor (invalid provider ordering).
- Queued-chip send-now and the NO_ACTIVE_RUN fallback pass explicit
  (possibly empty) overrideFiles so an idle send can't vacuum composer
  files staged for a different draft.
- Redis createJob deletes the stale steer list BEFORE the replacement
  hash is written as running — a steer 202-accepted against the new job
  can never be wiped by the reset.
- Resumed-turn finalization mirrors the normal path's terminal drain:
  createdAt-guarded close-and-drain, leftovers ride the resumed final
  event as pendingSteers instead of being cleared by completeJob.
- buildSteerMedia restores composer order over the $in result so
  multi-attachment steers reach the model in the order the user saw.

* ⚛️ fix: Atomic Job Replacement and Boundary-Clean Steering Module

Codex round 6 (5 fixed, 1 standing deferral):
- createJob resets the steer queue and writes the job hash in ONE
  same-slot Lua script (JOB_CREATE_LUA): a steer POST can no longer
  interleave between them on cluster, so a steer accepted against one
  run can never be drained into another. Redis-validated.
- The steering media pipeline moved to packages/api
  (agents/steering/media.ts) with injected getFiles and a structural
  client interface — /api keeps zero steering logic; specs ported to
  the DI seam.
- handleSteerRequest checks the job BEFORE the capability gate: a steer
  racing completion on an unsupported SDK gets 404 (send-now) instead
  of a 501 queue with no run-end signal left to drain it.
- useQueueDrain binds to the active conversation: navigating away
  between the final SSE and the drain effect leaves the signal
  unconsumed instead of submitting A's follow-up into B; the drain
  fires on return.
- abortJob closes and drains the steer queue BEFORE the content
  snapshot, so a drain-hook apply that lands pre-drain is captured
  inline rather than lost between the snapshot and the terminal drain.

* 🚦 fix: Parked Run-End Signals, Interrupt Priority, Settled-Run Fallbacks

Codex round 7 (5 fixes):
- Run-end signals for a non-active conversation are PARKED per
  conversation instead of squatting the shared index slot: a later run
  finishing on the same pane can no longer overwrite them, and the
  parked drain fires when the user returns.
- "Interrupt & send" front-inserts carry a priority flag that outranks
  createdAt when abort leftovers merge back chronologically — the
  urgent redirect drains first, not the oldest steer.
- STEER_UNSUPPORTED/RUN_PAUSED/QUEUE_FULL rejections landing after the
  run settled mirror the NO_ACTIVE_RUN fallback and send immediately
  (queueing would strand the text with no run-end signal left); on the
  pinned SDK this is the common Enter-near-run-end path.
- A failed abort (e.g. 404 when the run completed first) still signals
  the interrupt drain, so the queued interrupt message can't strand and
  the armed flag can't leak onto a later run.
- Steered-image fallback alt text is localized (com_ui_attached_image).

* 📌 chore: Adopt Published @librechat/agents Types Post-Bump

dev's pin bump to ^3.2.62 (the release carrying injection + steer
replay) landed via merge; the steering runtime now uses the SDK's real
InjectedMessage/hook-output types instead of the local structural
mirrors that bridged the pre-publish window. The two-half capability
probe stays as the defensive gate for mismatched deployments — and the
capability spec now exercises its TRUE path against the published
package in CI.

* 🛅 feat: Park-and-Claim Steer Recovery + Host-View Content Reads

Codex round 8 (6 fixed incl. both P1s, 1 push-back):
- The long-deferred no-subscriber gap is closed: every terminal drain
  (final, aborted-final, error, abortJob, resumed finalize) PARKS
  acknowledged leftovers on the job hash (unrecoveredSteers), and the
  status route claims them exactly once for inactive jobs — a client
  that closed/reloaded past the transient final event restores its
  steers as queued chips within the post-terminal TTL. A replacement
  run clears the parked copy (a live client started it).
- Same-instance content reads are steer-complete: RedisJobStore now
  caches the HOST content array (WeakRef) via setContentParts and
  prefers it over the SDK graph cache, whose view never contains
  host-authored steer parts; the graph fallback splice-INSERTS steer
  chunks at their recorded host-view indices (the graph array is
  unshifted, so assignment would overwrite SDK parts).
- Replay token accounting now counts prepended file-context text: full
  stamped content minus the steer body (already counted), so large
  steered documents hit the budget instead of bypassing pruning.
- The queue drain restores an item when ask() refuses without sending
  (history not yet in cache after navigating back) — text is never
  silently dropped.
- The armed interrupt flag travels WITH a parked run-end signal, so
  another run on the same pane can neither consume nor clear it.
- parseTextParts extracts steer text (search indexing / audio).

* 🎛️ refactor: Single Send Slot + In-Thread Steer Messages

- Merge the during-run send affordance into the send/stop button slot:
  with composer text the send button replaces Stop (Enter = default
  action), hover reveals Steer/Queue/Interrupt rows with shortcuts;
  drop the separate DuringRunActionsMenu chevron
- Add during-run keyboard chords: Cmd/Ctrl+Enter = non-default action,
  Alt+Enter = interrupt & send (plain-Enter submitters only)
- Render steers as standard user messages in the thread: SteerPart
  (icon + author header + user text presentation) replaces the
  SteerBubble, and submitted steers appear immediately at the projected
  injection point via the PendingSteers slot on the streaming message
- Keep composer rows only for recoverable states: failed steers
  (retry/edit/queue) and queued follow-ups

* 🩹 fix: Keep the Replacement Submission Alive Across Abort Settlement

The aborted run's final SSE event fires before the abort HTTP response
resolves, so an armed interrupt & send drains and starts the NEXT
submission while the abort POST is still in flight. The response
handler's unconditional clearAllSubmissions() then reset the new
submission, aborting its stream attach before the subscribe — the
follow-up ran and persisted server-side but the live placeholder
finalized empty (content appeared only after reload).

useAbortCleanup captures the submission before the abort round-trip
and both settlement paths (success and 404-catch) clear only when the
captured submission is still current; a replacement stays untouched.
Plain Stop behavior is unchanged.

* 🧭 test: Playwright E2E for Mid-Run Steering and Queuing

- Add e2e/specs/mock/steering.spec.ts: steer mid-run (202 + immediate
  in-thread pending part + real MCP tool boundary + words survive run
  end), Cmd/Ctrl+Enter queue with auto-send after clean completion,
  and Alt+Enter interrupt & send with the follow-up streaming into the
  live view
- Add the E2E_STEER_TOOL_REPLY fake-model marker: slow preamble, a
  real remember_fact MCP tool call (PostToolBatch boundary), then a
  final turn
- Test 1 pins the run-end degradation contract while the SDK's
  top-level agentId stamping bug blocks live injection; its header
  documents the assertions to flip once the fixed SDK is pinned

* 🧷 fix: Job-Independent Steer Recovery + Expiry and Resume-Gap Parking

Codex round 10: the park-and-claim recovery had lifecycle holes.

- Move parked steers off the job hash onto their own bounded-TTL store
  key (JOB_CREATE_LUA resets it; deleteJob leaves it alone): the default
  completeJob path deletes the job record immediately, and the Redis
  read path never deserialized the old hash field — recovery previously
  worked only with STREAM_KEEP_COMPLETED_JOBS on the in-memory store
- Carry the owner identity inside the parked payload and authorize the
  claim against it, so the status route recovers steers on its jobless
  branch too (the common reload-after-terminal case); a non-owner claim
  returns nothing and re-parks the payload
- Park queued steers on approval expiry: snapshot the frozen queue
  before the requires_action→aborted CAS (whose terminal cleanup drops
  the steers key) and park only when the CAS wins
- Mirror the terminal drain/park block in resume.js's failure path,
  which previously let completeJob's backstop clear 202-accepted steers
- Close the Redis snapshot→subscribe resume gap: re-peek the queue
  after attaching and re-surface missed on_steer_applied events from
  the durable content view (synthesizeAppliedSteerEvents), updating
  resumeState.pendingSteers to the live queue

* 📌 chore: Require @librechat/agents 3.2.63 + Applied-Steer E2E Contract

- Bump the @librechat/agents pin to ^3.2.63 in api/ and packages/api/:
  it scopes the hook agentId marker to subagent child graphs, so the
  steering drain hook fires at top-level tool-batch boundaries and
  mid-run injection is active (danny-avila/agents PR 307)
- Flip e2e steering test 1 from the documented degradation contract to
  the applied-steer contract: the optimistic in-thread part transitions
  to the persisted part at the tool boundary and survives inside the
  response after run end, with no queued follow-up turn

* 🎗️ feat: Steered Messages Join the Message-Nav Ribs

Steers are user messages, so they get their own clickable rib on the
navigation rail, interleaved at their in-thread position inside the
response that absorbed them (one DOM query in document order). SteerPart
anchors itself as #steer-<id> with a steer-render marker — both the
optimistic pending entry and the persisted part — and the rib carries
the user role label with a preview drawn from the steer's text body,
skipping the author header.

* ❎ feat: Cancel a Queued Steer Before Injection + True User-Message Alignment

- Add POST /chat/steer/cancel: removes ONE still-queued steer by id via
  an atomic list rebuild (Redis Lua preserves order and TTL), authorized
  against the job owner; removed:false is advisory — the cancel lost its
  race to the drain or the run end, never an error
- Surface an × on the in-thread pending steer (server-acknowledged
  entries only): optimistic removal, restored if the POST fails since
  the server would still inject the words
- Outdent SteerPart past the response's icon column so steers sit flush
  with top-level message rows, reading as regular user messages

* 🧯 fix: Round-11 Recovery Hardening + Provider-Free Pending Slot

- Reconcile the resume steer gap by steerId SETS, not queue length — a
  steer added in the gap (or an equal-length drain+enqueue swap) now
  refreshes resumeState.pendingSteers and still synthesizes the missed
  on_steer_applied events
- Make completeJob's terminal backstop park: direct error-path callers
  without the controllers' close-and-park no longer silently clear
  202-accepted steers (createdAt-guarded closeAndDrain + owner park
  before the terminal write)
- Persist the steer part BEFORE media encoding in the drain hook: an
  abort inside the encode window can no longer lose a file-steer (the
  part refs come from the enqueue-sanitized item; replay re-encodes
  per turn unchanged)
- Move the parked-claim owner check INSIDE the atomic store claim
  (substring gate in the Lua / in-memory equivalent): a non-owner probe
  can no longer transiently delete the recovery payload; the app-side
  parse stays authoritative
- Park queued steers in BOTH stores' own requires_action expiry
  cleanup, which bypassed the manager-level sweep
- Sweep expired parked steers from the in-memory store's periodic
  cleanup; restore a queued chip when send-now's submit is refused;
  upsert steer ACKs so an SSE reconnect reseed cannot duplicate chips
- Mount the cancel mutation per steer item so the pending slot needs no
  QueryClient on ordinary streaming renders (fixes the CI failure in
  ContentParts.integration.test)
- Skipped delivery-gated parking (finding 8): transport receiver counts
  cannot prove browser delivery, and gating the only durable copy on
  them trades cosmetic chip resurrection for real text loss; the window
  is already bounded by claim-on-read, createJob reset, and the TTL

* 🩺 fix: Annotate PARKED_STEERS_TTL_MS for isolatedDeclarations

tsdown's d.ts generation requires explicit types on exported consts
with computed initializers; tsc --noEmit does not run that check, so
the round-11 export slipped past local verification and broke Build
packages (and every downstream CI job that consumes the built dist).

* 🛟 fix: Round-12 Terminal-Path Recovery + Durable Steer Events

- Park queued steers before the stale-running reap deletes a crashed or
  hung job in BOTH stores — the one terminal path with no controller
  finalization; requires_action expiry parking refactored onto the same
  snapshot/park helpers
- Enqueue instead of dropping when a steer fallback send is refused:
  both the NO_ACTIVE_RUN branch and the settled-run rejection branch
  now observe sendNow's false return
- Recover on the SSE reconnect-404 terminal path: convert local pending
  steers to queued, claim parked steers via /chat/status, and write a
  non-completed run-end signal so interrupt flags release without
  auto-sending an unknown outcome
- Fall back to a positive parked-recovery TTL when completedTtl is 0
  (SET EX 0 is invalid and silently killed recovery)
- Make on_steer_applied durable before publish: emitChunk gains a
  durable option that awaits the chunk-log append (best-effort) ahead
  of the transport publish; the default delta path stays fire-and-forget

* 🔐 fix: Round-13 Steer Authorization + Trusted File Refs

- Resolve client-supplied steer file refs against the DB owner-scoped
  at enqueue and queue only DB-derived shapes (same filter as the
  injection fetch, shared via refs.ts); any unresolved id fails loud
  with 400 — spoofed type/filepath metadata can no longer be persisted
  into assistant content or rendered in chat/share views
- Enforce agent authorization on /chat/steer against the ORIGINATING
  run's job identity: the chat path's role gate (AGENTS:USE, with the
  same non-agents-endpoint skip) plus the per-agent ACL check with the
  capability bypass — revoked access mid-run can no longer inject;
  cancel stays ownership-only (nothing model-bound)
- Mark steered uploads used after a successful enqueue (owner-scoped,
  best-effort) so the upload-window TTL cannot reap a file the
  persisted steer part references
- Consume the parked recovery copy after live delivery: converting
  final/abort/error pendingSteers fires one owner-gated claim-on-read,
  so dismissed chips can no longer resurrect on a later reload

* 🎙️ fix: Round-14 Composer-Context Fidelity + TTS and Queue-State Gaps

- Keep steer text out of generic assistant text extraction:
  parseTextParts excludes STEER parts by default with an includeSteer
  opt-in for the full-record surfaces (Meili indexing, aborted-response
  persistence) — TTS callers no longer speak the user's own mid-run
  words
- Mark queued uploads used at enqueue time via a minimal owner-scoped
  POST /files/usage (fail-closed without a user; upload limiters do not
  apply to a metadata touch), fired once wherever composer files enter
  the queued state — the upload-window TTL can no longer reap a file
  waiting out a long run or approval pause
- Carry quote chips and manual skill picks on queued items: captured
  and consumed from the composer at queue/interrupt time exactly like
  files, threaded through the drain and send-now overrides, and
  restored by the queued row's Edit message
- Key an early-aborted FIRST turn's run-end signal to NEW_CONVO
  (resolveRunEndTarget) so queued follow-ups stay visible on the
  restored new-chat composer instead of parking under an optimistic
  stream id the user never sees again

* 🧿 fix: Round-15 Gap Coverage + Consolidated Sweep (Share Leak, Abort Ids, Chip Hygiene)

- Run the resume steer-gap check for every still-active job: an empty
  snapshot no longer skips the re-peek, and synthesis now keys on the
  FRESH content view so an applied-in-gap steer that was never
  snapshotted still re-surfaces (over-emission is benign — applied-id
  dedupe, index-stable parts)
- Thread queued context through steer degradation: sendQueuedNow passes
  the item's quotes/skills into submitSteer, and every fallback
  (requeue or settled send) restores them instead of dropping to
  text+files
- Stop shared links from leaking steer attachment refs: the share
  snapshot now walks content — files-excluded shares strip steer-part
  files entirely; files-included shares sanitize and share-route them
  like top-level files (copy-on-write, non-steer content by reference)
- Seed pending-steer chips unconditionally on load/return so a steer
  applied while away cannot linger as a stale chip beside its part
- Use the abort response's resolved job id: chips/drain-signal land
  where the user actually is (NEW_CONVO for a new-held first turn,
  consistent with resolveRunEndTarget) while the parked-copy claim hits
  the resolved id instead of a no-op /chat/status/new
- Open steered documents like normal message files (FilePreviewDialog)
- Cap the applied-steer id set on the live path via a shared helper;
  kept surviving run end deliberately (late-ACK race depends on it) and
  fixed the atom comment that claimed otherwise

* 💡 fix: Un-light Steer Ribs When Their Node Is Replaced

Two stacked gaps kept a steer rib lit after scrolling away: the
pending→applied swap replaces the DOM node under the same id, which
produces no IntersectionObserver exit and — because the entry list
dedupes on (id, preview) — no entries change either, so the observer
kept watching a detached node; and the rail's mutation filter only
reacted to .message-render nodes, so steer-node swaps and removals
never triggered a refresh at all.

- reconcileObservedElements re-points the observer at replaced nodes
  from the mutation-driven refresh regardless of entries identity,
  dropping stale visibility until the fresh node reports (the observer
  fires its initial intersection immediately, so a truly visible part
  re-lights within a frame)
- The mutation filter now recognizes steer-render nodes alongside
  message rows

* 🪪 fix: Round-16 Recovery Owner Fields + Context Stickiness + Share Labels

- Park resumed-run leftovers with the manager facade's metadata owner
  fields: a bare job.userId is undefined on that shape, which made
  every parked payload from a resumed HITL run unclaimable
- Keep a queued item's quotes/skills sticky through a successful steer
  ACK: the pending chip carries them (client-only), reseeds preserve
  them across reconnects, and every terminal conversion — local or
  server-list, merged by steerId — restores them onto the queued item
- Convert resumeState.pendingSteers on the inactive status branch
  (deduped against unrecoveredSteers) so steers observed in the
  expired-pause-before-sweeper window convert instead of vanishing
  until a later reload
- Label shared steer parts share-safely via the existing ShareContext:
  a viewer's own name no longer appears on the sharer's steered
  messages

* ✂️ fix: Carry Steer Context Through the Failed-Chip Edit Action

Retry and convert-to-queue already preserve a failed steer's carried
quotes/skills; Edit message dropped them on the way back to the
composer. It now restores them through the same context path.
2026-07-14 10:11:10 -04:00
Danny Avila
cb1d536874
📻 fix: Replay MCP OAuth Prompts for Coalesced Connections (#13565)
* fix: Replay MCP OAuth URL for Joined Connections

* chore: Sort MCP OAuth Imports

* test: Restore MCP OAuth Registry Spies

* fix: Replay pending MCP OAuth prompts

* fix: Replay MCP OAuth on Stream Resume

* fix: Preserve MCP OAuth Replay Context

* chore: Format MCP OAuth Replay Context

* test: Expect MCP OAuth Replay Expiry

* fix: Render pending MCP OAuth prompts

* chore: Clean MCP OAuth Replay Type Narrowing

* fix: Stabilize new MCP OAuth chats

* fix: Re-emit cached MCP OAuth prompts

* fix: Replay pending OAuth for selected MCP tools

* fix: Avoid stalling pending MCP OAuth replay

* test: Clean MCP OAuth review findings

* test: Restore MCP OAuth registry spy

* fix: Resolve OAuth Typecheck Regressions

* fix: Harden MCP OAuth replay edge cases

* test: Cover MCP OAuth joined prompt expiry

* test: Mark joined OAuth replay fixture

* test: Use OAuth fixture for joined replay expiry

* fix: Anchor resumed MCP OAuth prompts

* fix: Seed resumable turn metadata before MCP init

* test: Format resume metadata regression

* fix: Prioritize resumable stream routes

* fix: Preserve MCP OAuth resume message tree

* test: Fix MCP OAuth Resume Test Types

* fix: Replay MCP OAuth Regenerate Prompts

* fix: Skip OAuth-only Abort Persistence

* fix: Stabilize OAuth Resume Replay

* fix: Target Non-Tail Regenerate Responses

* fix: Scope Regenerate Step Updates

* fix: Clean Up OAuth Abort State

* fix: Preserve Regenerate Branch Siblings

* fix: Preserve OAuth Resume Branch State

* fix: Preserve OAuth Branch Resume State

* chore: Sort OAuth Resume Imports

* fix: Address OAuth Resume Review Findings

* test: Fix Abort Fixture Typing
2026-06-07 10:45:54 -04:00
Danny Avila
877c2efc85
🏗️ feat: bulkWrite isolation, pre-auth context, strict-mode fixes (#12445)
* fix: wrap seedDatabase() in runAsSystem() for strict tenant mode

seedDatabase() was called without tenant context at startup, causing
every Mongoose operation inside it to throw when
TENANT_ISOLATION_STRICT=true. Wrapping in runAsSystem() gives it the
SYSTEM_TENANT_ID sentinel so the isolation plugin skips filtering,
matching the pattern already used for performStartupChecks and
updateInterfacePermissions.

* fix: chain tenantContextMiddleware in optionalJwtAuth

optionalJwtAuth populated req.user but never established ALS tenant
context, unlike requireJwtAuth which chains tenantContextMiddleware
after successful auth. Authenticated users hitting routes with
optionalJwtAuth (e.g. /api/banner) had no tenant isolation.

* feat: tenant-safe bulkWrite wrapper and call-site migration

Mongoose's bulkWrite() does not trigger schema-level middleware hooks,
so the applyTenantIsolation plugin cannot intercept it. This adds a
tenantSafeBulkWrite() utility that injects the current ALS tenant
context into every operation's filter/document before delegating to
native bulkWrite.

Migrates all 8 runtime bulkWrite call sites:
- agentCategory (seedCategories, ensureDefaultCategories)
- conversation (bulkSaveConvos)
- message (bulkSaveMessages)
- file (batchUpdateFiles)
- conversationTag (updateTagsForConversation, bulkIncrementTagCounts)
- aclEntry (bulkWriteAclEntries)

systemGrant.seedSystemGrants is intentionally not migrated — it uses
explicit tenantId: { $exists: false } filters and is exempt from the
isolation plugin.

* feat: pre-auth tenant middleware and tenant-scoped config cache

Adds preAuthTenantMiddleware that reads X-Tenant-Id from the request
header and wraps downstream in tenantStorage ALS context. Wired onto
/oauth, /api/auth, /api/config, and /api/share — unauthenticated
routes that need tenant scoping before JWT auth runs.

The /api/config cache key is now tenant-scoped
(STARTUP_CONFIG:${tenantId}) so multi-tenant deployments serve the
correct login page config per tenant.

The middleware is intentionally minimal — no subdomain parsing, no
OIDC claim extraction. The private fork's reverse proxy or auth
gateway sets the header.

* feat: accept optional tenantId in updateInterfacePermissions

When tenantId is provided, the function re-enters inside
tenantStorage.run({ tenantId }) so all downstream Mongoose queries
target that tenant's roles instead of the system context. This lets
the private fork's tenant provisioning flow call
updateInterfacePermissions per-tenant after creating tenant-scoped
ADMIN/USER roles.

* fix: tenant-filter $lookup in getPromptGroup aggregation

The $lookup stage in getPromptGroup() queried the prompts collection
without tenant filtering. While the outer PromptGroup aggregate is
protected by the tenantIsolation plugin's pre('aggregate') hook,
$lookup runs as an internal MongoDB operation that bypasses Mongoose
hooks entirely.

Converts from simple field-based $lookup to pipeline-based $lookup
with an explicit tenantId match when tenant context is active.

* fix: replace field-level unique indexes with tenant-scoped compounds

Field-level unique:true creates a globally-unique single-field index in
MongoDB, which would cause insert failures across tenants sharing the
same ID values.

- agent.id: removed field-level unique, added { id, tenantId } compound
- convo.conversationId: removed field-level unique (compound at line 50
  already exists: { conversationId, user, tenantId })
- message.messageId: removed field-level unique (compound at line 165
  already exists: { messageId, user, tenantId })
- preset.presetId: removed field-level unique, added { presetId, tenantId }
  compound

* fix: scope MODELS_CONFIG, ENDPOINT_CONFIG, PLUGINS, TOOLS caches by tenant

These caches store per-tenant configuration (available models, endpoint
settings, plugin availability, tool definitions) but were using global
cache keys. In multi-tenant mode, one tenant's cached config would be
served to all tenants.

Appends :${tenantId} to cache keys when tenant context is active.
Falls back to the unscoped key when no tenant context exists (backward
compatible for single-tenant OSS deployments).

Covers all read, write, and delete sites:
- ModelController.js: get/set MODELS_CONFIG
- PluginController.js: get/set PLUGINS, get/set TOOLS
- getEndpointsConfig.js: get/set/delete ENDPOINT_CONFIG
- app.js: delete ENDPOINT_CONFIG (clearEndpointConfigCache)
- mcp.js: delete TOOLS (updateMCPTools, mergeAppTools)
- importers.js: get ENDPOINT_CONFIG

* fix: add getTenantId to PluginController spec mock

The data-schemas mock was missing getTenantId, causing all
PluginController tests to throw when the controller calls
getTenantId() for tenant-scoped cache keys.

* fix: address review findings — migration, strict-mode, DRY, types

Addresses all CRITICAL, MAJOR, and MINOR review findings:

F1 (CRITICAL): Add agents, conversations, messages, presets to
SUPERSEDED_INDEXES in tenantIndexes.ts so dropSupersededTenantIndexes()
drops the old single-field unique indexes that block multi-tenant inserts.

F2 (CRITICAL): Unknown bulkWrite op types now throw in strict mode
instead of silently passing through without tenant injection.

F3 (MAJOR): Replace wildcard export with named export for
tenantSafeBulkWrite, hiding _resetBulkWriteStrictCache from the
public package API.

F5 (MAJOR): Restore AnyBulkWriteOperation<IAclEntry>[] typing on
bulkWriteAclEntries — the unparameterized wrapper accepts parameterized
ops as a subtype.

F7 (MAJOR): Fix config.js tenant precedence — JWT-derived
req.user.tenantId now takes priority over the X-Tenant-Id header for
authenticated requests.

F8 (MINOR): Extract scopedCacheKey() helper into tenantContext.ts and
replace all 11 inline occurrences across 7 files.

F9 (MINOR): Use simple localField/foreignField $lookup for the
non-tenant getPromptGroup path (more efficient index seeks).

F12 (NIT): Remove redundant BulkOp type alias.
F13 (NIT): Remove debug log that leaked raw tenantId.

* fix: add new superseded indexes to tenantIndexes test fixture

The test creates old indexes to verify the migration drops them.
Missing fixture entries for agents.id_1, conversations.conversationId_1,
messages.messageId_1, and presets.presetId_1 caused the count assertion
to fail (expected 22, got 18).

* fix: restore logger.warn for unknown bulk op types in non-strict mode

* fix: block SYSTEM_TENANT_ID sentinel from external header input

CRITICAL: preAuthTenantMiddleware accepted any string as X-Tenant-Id,
including '__SYSTEM__'. The tenantIsolation plugin treats SYSTEM_TENANT_ID
as an explicit bypass — skipping ALL query filters. A client sending
X-Tenant-Id: __SYSTEM__ to pre-auth routes (/api/share, /api/config,
/api/auth, /oauth) would execute Mongoose operations without tenant
isolation.

Fixes:
- preAuthTenantMiddleware rejects SYSTEM_TENANT_ID in header
- scopedCacheKey returns the base key (not key:__SYSTEM__) in system
  context, preventing stale cache entries during runAsSystem()
- updateInterfacePermissions guards tenantId against SYSTEM_TENANT_ID
- $lookup pipeline separates $expr join from constant tenantId match
  for better index utilization
- Regression test for sentinel rejection in preAuthTenant.spec.ts
- Remove redundant getTenantId() call in config.js

* test: add missing deleteMany/replaceOne coverage, fix vacuous ALS assertions

bulkWrite spec:
- deleteMany: verifies tenant-scoped deletion leaves other tenants untouched
- replaceOne: verifies tenantId injected into both filter and replacement
- replaceOne overwrite: verifies a conflicting tenantId in the replacement
  document is overwritten by the ALS tenant (defense-in-depth)
- empty ops array: verifies graceful handling

preAuthTenant spec:
- All negative-case tests now use the capturedNext pattern to verify
  getTenantId() inside the middleware's execution context, not the
  test runner's outer frame (which was always undefined regardless)

* feat: tenant-isolate MESSAGES cache, FLOWS cache, and GenerationJobManager

MESSAGES cache (streamAudio.js):
- Cache key now uses scopedCacheKey(messageId) to prefix with tenantId,
  preventing cross-tenant message content reads during TTS streaming.

FLOWS cache (FlowStateManager):
- getFlowKey() now generates ${type}:${tenantId}:${flowId} when tenant
  context is active, isolating OAuth flow state per tenant.

GenerationJobManager:
- tenantId added to SerializableJobData and GenerationJobMetadata
- createJob() captures the current ALS tenant context (excluding
  SYSTEM_TENANT_ID) and stores it in job metadata
- SSE subscription endpoint validates job.metadata.tenantId matches
  req.user.tenantId, blocking cross-tenant stream access
- Both InMemoryJobStore and RedisJobStore updated to accept tenantId

* fix: add getTenantId and SYSTEM_TENANT_ID to MCP OAuth test mocks

FlowStateManager.getFlowKey() now calls getTenantId() for tenant-scoped
flow keys. The 4 MCP OAuth test files mock @librechat/data-schemas
without these exports, causing TypeError at runtime.

* fix: correct import ordering per AGENTS.md conventions

Package imports sorted shortest to longest line length, local imports
sorted longest to shortest — fixes ordering violations introduced by
our new imports across 8 files.

* fix: deserialize tenantId in RedisJobStore — cross-tenant SSE guard was no-op in Redis mode

serializeJob() writes tenantId to the Redis hash via Object.entries,
but deserializeJob() manually enumerates fields and omitted tenantId.
Every getJob() from Redis returned tenantId: undefined, causing the
SSE route's cross-tenant guard to short-circuit (undefined && ... → false).

* test: SSE tenant guard, FlowStateManager key consistency, ALS scope docs

SSE stream tenant tests (streamTenant.spec.js):
- Cross-tenant user accessing another tenant's stream → 403
- Same-tenant user accessing own stream → allowed
- OSS mode (no tenantId on job) → tenant check skipped

FlowStateManager tenant tests (manager.tenant.spec.ts):
- completeFlow finds flow created under same tenant context
- completeFlow does NOT find flow under different tenant context
- Unscoped flows are separate from tenant-scoped flows

Documentation:
- JSDoc on getFlowKey documenting ALS context consistency requirement
- Comment on streamAudio.js scopedCacheKey capture site

* fix: SSE stream tests hang on success path, remove internal fork references

The success-path tests entered the SSE streaming code which never
closes, causing timeout. Mock subscribe() to end the response
immediately. Restructured assertions to verify non-403/non-404.

Removed "private fork" and "OSS" references from code and test
descriptions — replaced with "deployment layer", "multi-tenant
deployments", and "single-tenant mode".

* fix: address review findings — test rigor, tenant ID validation, docs

F1: SSE stream tests now mock subscribe() with correct signature
(streamId, writeEvent, onDone, onError) and assert 200 status,
verifying the tenant guard actually allows through same-tenant users.

F2: completeFlow logs the attempted key and ALS tenantId when flow
is not found, so reverse proxy misconfiguration (missing X-Tenant-Id
on OAuth callback) produces an actionable warning.

F3/F10: preAuthTenantMiddleware validates tenant ID format — rejects
colons, special characters, and values exceeding 128 chars. Trims
whitespace. Prevents cache key collisions via crafted headers.

F4: Documented cache invalidation scope limitation in
clearEndpointConfigCache — only the calling tenant's key is cleared;
other tenants expire via TTL.

F7: getFlowKey JSDoc now lists all 8 methods requiring consistent
ALS context.

F8: Added dedicated scopedCacheKey unit tests — base key without
context, base key in system context, scoped key with tenant, no
ALS leakage across scope boundaries.

* fix: revert flow key tenant scoping, fix SSE test timing

FlowStateManager: Reverts tenant-scoped flow keys. OAuth callbacks
arrive without tenant ALS context (provider redirects don't carry
X-Tenant-Id), so completeFlow/failFlow would never find flows
created under tenant context. Flow IDs are random UUIDs with no
collision risk, and flow data is ephemeral (TTL-bounded).

SSE tests: Use process.nextTick for onDone callback so Express
response headers are flushed before res.write/res.end are called.

* fix: restore getTenantId import for completeFlow diagnostic log

* fix: correct completeFlow warning message, add missing flow test

The warning referenced X-Tenant-Id header consistency which was only
relevant when flow keys were tenant-scoped (since reverted). Updated
to list actual causes: TTL expiry, missing flow, or routing to a
different instance without shared Keyv storage.

Removed the getTenantId() call and import — no longer needed since
flow keys are unscoped.

Added test for the !flowState branch in completeFlow — verifies
return false and logger.warn on nonexistent flow ID.

* fix: add explicit return type to recursive updateInterfacePermissions

The recursive call (tenantId branch calls itself without tenantId)
causes TypeScript to infer circular return type 'any'. Adding
explicit Promise<void> satisfies the rollup typescript plugin.

* fix: update MCPOAuthRaceCondition test to match new completeFlow warning

* fix: clearEndpointConfigCache deletes both scoped and unscoped keys

Unauthenticated /api/endpoints requests populate the unscoped
ENDPOINT_CONFIG key. Admin config mutations clear only the
tenant-scoped key, leaving the unscoped entry stale indefinitely.
Now deletes both when in tenant context.

* fix: tenant guard on abort/status endpoints, warn logs, test coverage

F1: Add tenant guard to /chat/status/:conversationId and /chat/abort
matching the existing guard on /chat/stream/:streamId. The status
endpoint exposes aggregatedContent (AI response text) which requires
tenant-level access control.

F2: preAuthTenantMiddleware now logs warn for rejected __SYSTEM__
sentinel and malformed tenant IDs, providing observability for
bypass probing attempts.

F3: Abort fallback path (getActiveJobIdsForUser) now has tenant
check after resolving the job.

F4: Test for strict mode + SYSTEM_TENANT_ID — verifies runAsSystem
bypasses tenantSafeBulkWrite without throwing in strict mode.

F5: Test for job with tenantId + user without tenantId → 403.

F10: Regex uses idiomatic hyphen-at-start form.

F11: Test descriptions changed from "rejects" to "ignores" since
middleware calls next() (not 4xx).

Also fixes MCPOAuthRaceCondition test assertion to match updated
completeFlow warning message.

* fix: test coverage for logger.warn, status/abort guards, consistency

A: preAuthTenant spec now mocks logger and asserts warn calls for
__SYSTEM__ sentinel, malformed characters, and oversized headers.

B: streamTenant spec expanded with status and abort endpoint tests —
cross-tenant status returns 403, same-tenant returns 200 with body,
cross-tenant abort returns 403.

C: Abort endpoint uses req.user.tenantId (not req.user?.tenantId)
matching stream/status pattern — requireJwtAuth guarantees req.user.

D: Malformed header warning now includes ip in log metadata,
matching the sentinel warning for consistent SOC correlation.

* fix: assert ip field in malformed header warn tests

* fix: parallelize cache deletes, document tenant guard, fix import order

- clearEndpointConfigCache uses Promise.all for independent cache
  deletes instead of sequential awaits
- SSE stream tenant guard has inline comment explaining backward-compat
  behavior for untenanted legacy jobs
- conversation.ts local imports reordered longest-to-shortest per
  AGENTS.md

* fix: tenant-qualify userJobs keys, document tenant guard backward-compat

Job store userJobs keys now include tenantId when available:
- Redis: stream:user:{tenantId:userId}:jobs (falls back to
  stream:user:{userId}:jobs when no tenant)
- InMemory: composite key tenantId:userId in userJobMap

getActiveJobIdsByUser/getActiveJobIdsForUser accept optional tenantId
parameter, threaded through from req.user.tenantId at all call sites
(/chat/active and /chat/abort fallback).

Added inline comments on all three SSE tenant guards explaining the
backward-compat design: untenanted legacy jobs remain accessible
when the userId check passes.

* fix: parallelize cache deletes, document tenant guard, fix import order

Fix InMemoryJobStore.getActiveJobIdsByUser empty-set cleanup to use
the tenant-qualified userKey instead of bare userId — prevents
orphaned empty Sets accumulating in userJobMap for multi-tenant users.

Document cross-tenant staleness in clearEndpointConfigCache JSDoc —
other tenants' scoped keys expire via TTL, not active invalidation.

* fix: cleanup userJobMap leak, startup warning, DRY tenant guard, docs

F1: InMemoryJobStore.cleanup() now removes entries from userJobMap
before calling deleteJob, preventing orphaned empty Sets from
accumulating with tenant-qualified composite keys.

F2: Startup warning when TENANT_ISOLATION_STRICT is active — reminds
operators to configure reverse proxy to control X-Tenant-Id header.

F3: mergeAppTools JSDoc documents that tenant-scoped TOOLS keys are
not actively invalidated (matching clearEndpointConfigCache pattern).

F5: Abort handler getActiveJobIdsForUser call uses req.user.tenantId
(not req.user?.tenantId) — consistent with stream/status handlers.

F6: updateInterfacePermissions JSDoc clarifies SYSTEM_TENANT_ID
behavior — falls through to caller's ALS context.

F7: Extracted hasTenantMismatch() helper, replacing three identical
inline tenant guard blocks across stream/status/abort endpoints.

F9: scopedCacheKey JSDoc documents both passthrough cases (no context
and SYSTEM_TENANT_ID context).

* fix: clean userJobMap in evictOldest — same leak as cleanup()
2026-03-28 16:43:50 -04:00