Commit graph

892 commits

Author SHA1 Message Date
Danny Avila
3fa33b740b
🛫 refactor: Promote Generation Protocol V2 Automatically (#15324)
* refactor: promote generation protocol v2 automatically

* fix: remove unused protocol import
2026-08-28 17:17:09 -04:00
Danny Avila
04a7577821
🧵 fix: Retry Shared Links After Message Persistence Gaps (#15306)
Some checks failed
Backend Unit Tests / Build packages (push) Waiting to run
Backend Unit Tests / Codegraph select (push) Waiting to run
Backend Unit Tests / TypeScript type checks (push) Blocked by required conditions
Backend Unit Tests / Circular dependency checks (push) Waiting to run
Backend Unit Tests / Tests: api (shard 1/3) (push) Blocked by required conditions
Backend Unit Tests / Tests: api (shard 2/3) (push) Blocked by required conditions
Backend Unit Tests / Tests: api (shard 3/3) (push) Blocked by required conditions
Backend Unit Tests / Tests: data-provider (push) Blocked by required conditions
Backend Unit Tests / Tests: data-schemas (push) Blocked by required conditions
Backend Unit Tests / Tests: @librechat/api (shard 1/4) (push) Blocked by required conditions
Backend Unit Tests / Tests: @librechat/api (shard 2/4) (push) Blocked by required conditions
Backend Unit Tests / Tests: @librechat/api (shard 3/4) (push) Blocked by required conditions
Backend Unit Tests / Tests: @librechat/api (shard 4/4) (push) Blocked by required conditions
Codegraph E2E Votes / vote (full suite) (push) Waiting to run
Docker Dev Branch Images Build / build (Dockerfile, lc-dev, node) (push) Waiting to run
Docker Dev Branch Images Build / build (Dockerfile.multi, lc-dev-api, api-build) (push) Waiting to run
Frontend Unit Tests / Codegraph select (push) Waiting to run
Frontend Unit Tests / Build packages (push) Waiting to run
Frontend Unit Tests / TypeScript type checks (client) (push) Blocked by required conditions
Frontend Unit Tests / Tests: @librechat/client (push) Blocked by required conditions
Frontend Unit Tests / Tests: Ubuntu (shard 1/2) (push) Blocked by required conditions
Frontend Unit Tests / Tests: Ubuntu (shard 2/2) (push) Blocked by required conditions
Frontend Unit Tests / Vite build verification (push) Blocked by required conditions
Sync Locize Translations & Create Translation PR / Sync Translation Keys with Locize (push) Has been cancelled
Sync Locize Translations & Create Translation PR / Create Translation PR on Version Published (push) Has been cancelled
* fix: Retry shared links after message persistence gaps

* chore: Sort share test imports

* fix: Address share review feedback

* fix: Normalize hydrated share target
2026-08-28 08:05:52 -04:00
Danny Avila
a5fd00cd7a
🎼 feat: Unify Agent Turn Execution Behind One Immutable Plan (#15296)
* feat: unify agent turn execution planning

* fix: preserve automatic subagent activity handles

* fix: gate subagent wakeups for ephemeral parents

* fix: scope subagent wakeup guidance by agent
2026-08-27 22:26:46 -04:00
Ravi Kumar L
a70bcf66a4
🪢 feat: show Langfuse session link in shared chats (#15273)
Some checks are pending
Backend Unit Tests / Build packages (push) Waiting to run
Backend Unit Tests / Codegraph select (push) Waiting to run
Backend Unit Tests / TypeScript type checks (push) Blocked by required conditions
Backend Unit Tests / Circular dependency checks (push) Waiting to run
Backend Unit Tests / Tests: api (shard 1/3) (push) Blocked by required conditions
Backend Unit Tests / Tests: api (shard 2/3) (push) Blocked by required conditions
Backend Unit Tests / Tests: api (shard 3/3) (push) Blocked by required conditions
Backend Unit Tests / Tests: data-provider (push) Blocked by required conditions
Backend Unit Tests / Tests: data-schemas (push) Blocked by required conditions
Backend Unit Tests / Tests: @librechat/api (shard 1/4) (push) Blocked by required conditions
Backend Unit Tests / Tests: @librechat/api (shard 2/4) (push) Blocked by required conditions
Backend Unit Tests / Tests: @librechat/api (shard 3/4) (push) Blocked by required conditions
Backend Unit Tests / Tests: @librechat/api (shard 4/4) (push) Blocked by required conditions
Codegraph E2E Votes / vote (full suite) (push) Waiting to run
Docker Dev Branch Images Build / build (Dockerfile, lc-dev, node) (push) Waiting to run
Docker Dev Branch Images Build / build (Dockerfile.multi, lc-dev-api, api-build) (push) Waiting to run
Frontend Unit Tests / Codegraph select (push) Waiting to run
Frontend Unit Tests / Build packages (push) Waiting to run
Frontend Unit Tests / TypeScript type checks (client) (push) Blocked by required conditions
Frontend Unit Tests / Tests: @librechat/client (push) Blocked by required conditions
Frontend Unit Tests / Tests: Ubuntu (shard 1/2) (push) Blocked by required conditions
Frontend Unit Tests / Tests: Ubuntu (shard 2/2) (push) Blocked by required conditions
Frontend Unit Tests / Vite build verification (push) Blocked by required conditions
Sync Locize Translations & Create Translation PR / Sync Translation Keys with Locize (push) Waiting to run
Sync Locize Translations & Create Translation PR / Create Translation PR on Version Published (push) Blocked by required conditions
* feat: show Langfuse sessions in shared chats

* fix: wait for optional auth before loading shares

* fix: require Langfuse access for shared session links

* refactor: move shared Langfuse policy into API package

* fix: wrap shared chat actions on mobile

* fix: coordinate shared Langfuse link loading

* perf: batch shared Langfuse capability checks
2026-08-27 14:29:57 -04:00
Marco Beretta
0383030817
⚡ perf: Avoid Parallel Full and Paged Prompt Loading on Startup (#15031)
Co-authored-by: Danny Avila <danny@librechat.ai>
2026-08-25 21:21:17 -04:00
Marco Beretta
124e357cbf
✏️ feat: Edit Pasted Text and Clear It on New Chat (#15017)
* fix: stop an unsent paste from following every new chat

An explicit new chat now drops the unsaved-chat draft key before the
composer resets. `newConversation` empties the composer, but the key
outlives it and `useAutoSave` restores from that key on the way in, so a
long paste that was never sent came back as an attachment on every later
new chat. Per-conversation drafts are untouched.

Clicking a pasted-text chip opens the text in an editor so it can be
corrected before sending, and the chip's subtitle offers returning the
paste to the composer. The text comes from the in-memory blob, falling
back to the `text` field the file record already carries, so neither
needs a new endpoint.

`FileContainer` grows a `subtitleAction` prop for the second control.
Supplying it swaps the chip's own button wrapper for a full-bleed one
behind the content, since a button inside a button is invalid markup and
browsers drop the inner one's events.

* chore: sort imports to fix static checks

* fix: address paste edit review findings

- Keep the original paste attached until the replacement upload succeeds,
  so a rejected or failed save cannot destroy the only copy
- Guard edits and queued replacements against conversation switches and
  new-chat resets, mirroring the long-paste lifecycle guards
- Recover text for restored pastes by downloading the stored bytes, which
  Assistants and agent uploads persist without a text field
- Delete uploaded attachments when an explicit new chat discards the
  draft that referenced them, instead of orphaning the records
- Mark paste provenance explicitly (session registry plus files draft)
  instead of inferring it from a filename a deliberate upload can share
- Keep the subtitle action revealed on devices without hover

* fix: scope draft cleanup to its tab and delete restored pastes

- Stamp unsaved-chat files drafts with the writing tab's session id and
  skip deletion when another tab owns the record, so a new chat in one
  tab cannot discard the uploads attached in another
- Delete a restored paste's upload when an edit replaces it or returns
  it to the composer, which the attached flag otherwise preserved

* fix: harden paste edit lifecycle guards

- Re-check the originating composer and the file map before detaching an
  edited original on upload success, so navigation or a send during the
  request cannot remove or delete a file the old draft or sent message
  still references
- Record the replacement upload's paste provenance in the session
  registry and the files draft, keeping Edit and Move back on the new chip
- Bind move-inline to the unsaved-chat token as well, and abort the move
  when the chip is no longer attached to an unsent composer
- Restrict new-chat draft cleanup to ids the composer still owns: library
  re-attaches and ids with unknowable ownership are spared, and uploads
  still in flight are deleted once their records reach the files cache,
  unless the file came back attached in the meantime

* fix: match restored paste identities and recheck before opening the editor

- Treat a chip as attached when any of its ids (map key, file id, temp
  id) matches the composer map, since draft restoration keys entries by
  their temporary upload id while the value carries the server-assigned
  one; the previous key-only check made Move back silently do nothing
  and left both chips attached after an edit
- Recheck the originating composer and the attachment map after the
  text resolve before opening the editor, so a send during the download
  cannot stage a replacement upload into the emptied composer
- Extend the provenance predicate to temp ids for the same restored shape

* fix: spare re-attached sent pastes and discard stale editor resolves

- Force-delete a restored paste only when the composer's own draft
  claims its id, so a paste that was already sent and re-attached from
  the library keeps its shared record through Edit and Move back
- Sequence editor-open requests so a slow text resolve cannot overwrite
  the chip a later click selected

* fix: carry deferred discards across resets and clear the pending draft

- Merge newly deferred upload ids with the pending set instead of
  replacing it, so a second reset cannot orphan an earlier in-flight
  upload's eventual record
- Match deferred ids against temp_file_id as well, since the files
  cache keys records by the server id while the discard tracked the
  request uuid
- Clear the pane's pending draft key on an explicit new chat, or a
  running response's queued text and attachments come back with the
  next run

* fix: mint a fresh tab id when sessionStorage was inherited

Duplicated and opener-created tabs start with a copy of the original's
sessionStorage, so a stored tab id only proves continuity when the
document is a reload of the same tab. Every other entry into a document
now mints a fresh id, keeping an inherited one from attributing another
tab's live drafts to this composer.

* fix: gate restored-paste deletion on draft tab ownership and unclip the chip focus ring

- Stamp every files draft with the writing tab, not just unsaved-chat
  ones, and require the stamp to match before a restored paste's record
  is deleted, so another tab restoring the same draft is not destroyed
- Draw the full-chip Edit control's focus indicator as an inset ring
  with the surface's radius, since the offset ring was clipped away by
  the surface's overflow-hidden

* fix: resolve paste ownership before restoration and across draft migration

- Treat a draft's own pastedTextIds as composer-owned at discard time,
  so a reload-then-new-chat click deletes or defers those uploads
  before the composer map has been rebuilt, instead of skipping them
- Read both the pending and idle draft keys when claiming a restored
  paste for deletion, since a response finishing mid-edit migrates the
  record between them

* fix: keep failed edits recoverable and retry failed draft deletions

- Reopen the paste editor with the user's corrections when the
  replacement upload is rejected or fails, instead of leaving only the
  original's text to reopen
- Retain deferred and immediate discard ids when the delete request
  fails, and retry them on the next files-cache update, so an offline
  or transient failure cannot orphan the uploads

* fix: retry restored-paste deletions that fail

A failed delete of a detached restored paste retains its payload in a
session store, and the discard retry effect drains retained payloads
alongside its own batch on every files-cache update, so an offline or
transient failure cannot orphan the upload once its chip is gone

* fix: queue failed edits and lock chips with actions in flight

- Queue a failed edit behind whatever dialog is open instead of dropping
  its corrections, and reopen it when that dialog closes
- Track an in-flight action per source paste and hide its Edit and Move
  back affordances until the replacement uploads or the move settles,
  so the same original cannot be acted on twice

* fix: address PR review bot findings

Codex:
- Keep the tab id on back-forward restoration
- Preserve the original tab owner when rewriting drafts
- Skip clearing idle and pending drafts another tab still owns
- Delete pending-draft uploads before clearing them
- Return failedFileIds from DELETE /files and retry those records
- Spare reattached files from retained deletion retries
- Trigger retained-deletion retries when a delete is retained
- Persist deferred discards across reload
- Delete embedded owned uploads with a discarded draft
- Ignore stale paste-editor failures before toasting
- Abort a queued edit after the original is sent
- Serialize Move back with a synchronous in-flight lock
- Prune paste provenance ids that left the draft

* fix: retain paste deletions the server reports as failed

The delete route answers 200 with `failedFileIds` when a record's storage
delete fails, so the detach path's `.catch()` never fired and the orphaned
upload lost its only cleanup reference once the draft provenance was pruned.
Inspect the resolved response and retain the deletion when it names the file.

Extract the `failedFileIds` reader `useNewChat` already had into the file
utils so both deletion paths read the response the same way, and give the
paste editor coverage for the failed and accepted responses.

Also add the missing `size` on a composer file literal that was failing the
client type-check.

* fix: release draft claims when their tab is gone and keep cleanup durable

A tab stamped its id on a files draft and nothing ever took it off, so a draft
saved in a tab the user then closed became unreachable for good: no other tab
would restore it, write to it, or clean it up, and the closed tab's id can never
be presented again. Tabs now report themselves in a small liveness registry and
release the claim on pagehide, and a claim whose tab is no longer around is
treated as free. Writers restamp a dead claim rather than preserving it.

Ownership also only existed once something was attached, so a typed-but-unattached
draft on a shared composer key read as nobody's and another tab's New Chat cleared
it. Saving text to one of those keys now claims it the same way.

Two more from the same review:

- The delete route answers a partial failure as 200, so treating an id as still
  present unless the response reports it deleted kept a ghost row for a file
  another tab had already removed. Read it the other way around: only a reported
  failure keeps a record cached.
- A retained deletion whose retry failed again moved no effect dependency, so it
  was never attempted a second time, and the payload only lived in memory. It is
  now persisted for the session and asks for a backed-off retry, plus one on
  regaining connectivity.

* fix: keep bfcache claims, retain failed deletes, and move the delete contract to TS

Four findings from the latest review round:

- pagehide fires with persisted: true when a document enters the back-forward
  cache rather than closing. Releasing the tab's claim there let another tab
  take the draft and delete files the restorable document still had attached,
  so the claim is now only handed back on a real unload; a bfcached tab that is
  never restored still ages out through the liveness window.
- useFileDeletion issued its batch and never looked at the outcome, so a fresh
  paste whose delete failed was orphaned with no retry. It now retains whatever
  the server did not delete, reading failedFileIds as well as the rejection.
- A partial failure answers 200, so the unconditional success toast told the
  user a file was deleted while it was still on disk and back in their list.
- The delete response contract lived in the legacy JS route. It moves to
  packages/api as buildDeleteFilesResponse, leaving the route a thin caller.

The useFileDeletion spec's mutateAsync mock returned undefined; react-query
always hands back a promise, so it now resolves like the real one.

* fix: park bfcached tab claims and drop ownership left by an emptied draft

A document in the back-forward cache has a frozen heartbeat, so the ordinary
liveness window expired its claim after 150s even though it could still be
restored with those attachments on screen, letting another tab take the draft
and delete the files underneath it. Entering the cache now parks the tab as
suspended, which holds the claim for 30 minutes: comfortably past the point a
browser keeps a bfcache entry, and still bounded, since a claim that never
expires is what stranded drafts under owners that no longer existed. Restoring
the document beats normally again and clears the flag. The registry entry grew
a shape for this and still reads records written as a bare timestamp.

Clearing the text of a shared composer key also left the ownership-only record
behind, locking the key to a tab with nothing in it: the next tab to type there
could neither restore its own draft nor take the key back. That claim is now
released when the text goes and nothing is attached.

* fix: keep unlinks out of the delete retry and claim shared text before writing

Four findings from the latest round:

- Retaining a failed agent or assistant unlink sent it through the generic
  retry, which replays files alone. That drops the tool_resource context, so
  the route would take its ordinary delete branch and destroy a record the
  agent and other references still point at. A failed unlink orphans nothing,
  so those deletions are simply not queued.
- A retry that resolved naming files in failedFileIds left both stores
  untouched, so nothing moved the effect that would try again. It now asks for
  another attempt on a reported failure, the same as on a rejection.
- The reattachment guard read only the idle new-chat key. After a reload the
  composer map is empty until the autosave restore renders, so a file the user
  had reattached to the conversation they were viewing, or to the pending key,
  could be deleted underneath them. All three keys are checked now, including
  their paste provenance.
- Text was written to a shared composer key before ownership was resolved, so
  a tab could overwrite another's saved text and still be refused the claim,
  leaving it unable to restore what it had just typed. The claim is taken
  first, and a claim with no attachment behind it follows whoever's text is
  actually stored; one backed by an attachment stays with its open owner.

* fix: merge shared discard state and keep restored file-search pastes retrievable

Four findings from the latest round:

- Every mount of useNewChat (header, sidebar, mobile bar, shortcuts) kept its
  own snapshot of the pending-discard list and wrote it back over one shared
  session store, so an id recorded by one instance was dropped by the next
  write from another, orphaning the upload it pointed at. An update now only
  resolves the ids that instance knows about and carries the rest through.
- Refusing an attachment-backed claim still let the text write land, destroying
  the owning tab's text for a tab that could not have restored it anyway. The
  claim now reports whether it succeeded and the write is dropped with it.
- A restored paste has no tool_resource on its record, so an edit to one that
  had been uploaded for file search was re-uploaded as a plain context file and
  the vector-backed original detached, dropping it out of retrieval. embedded
  does survive on the record and is only set for a vectorized file, so it is
  what the destination falls back to.
- The reattachment guard collected map keys and server ids but not
  temp_file_id, while the retry lookup resolves that alias: reattaching a file
  whose discard was pending under its temporary id would not have protected it.

* fix: stop the draft owner refusing its own writes and guard shared pending keys

Three findings, the first a regression from the previous commit:

- The attachment-backed refusal was evaluated before the owner check, so the
  tab that owned the draft was refused its own key: once anything was attached,
  nothing typed after it was saved. Ownership is settled first now, and the
  refusal applies only to another live tab.
- A long paste wrote its provenance and pending-paste record into the shared
  composer key without checking who owned it, and setFilesDraft preserves the
  existing owner rather than rejecting the write, so the paste was recorded
  into another tab's draft, which could then restore and delete the upload
  while this tab still showed the chip. Both write sites now check first.
- Two concurrent runs share the default pending key, and the migration to the
  new conversation ran before the ownership check: the finishing run moved the
  other tab's text and attachments under its own conversation and left that tab
  nothing to carry over. Ownership of the source is verified before migrating,
  and this composer's own text is still saved either way.

* fix: protect cross-tab reattachments and orphaned pastes on every discard path

Five findings from the latest round:

- The retry guard only read this pane's own draft keys, so a file a second tab
  had reattached to a conversation this pane never opened was deleted anyway.
  Drafts live in localStorage and are readable from every tab, so the guard now
  sweeps every persisted files draft rather than three known keys.
- Clearing the composer removed the shared text record without checking who
  owned it, so an empty composer in one tab erased text another tab was still
  holding behind its attachments. The clear path takes the same guard as the
  write path, and both now share one ownership predicate.
- When another tab owned the pending key, this tab's own queued attachments
  were cleared from the map and never written anywhere, because the autosave
  that would have persisted them had been refused that key for the whole run.
  They are now written under the conversation the run just became.
- A draft write that storage refuses (private mode, quota) left a generated
  paste with no record to discard it by. New Chat now also collects the live
  marked pastes the composer is still showing, skipping re-attached ones.
- With draft saving off, the reset path deleted files without awaiting or
  reading the response, so a failure orphaned the upload. It retains what the
  server did not delete, like every other deletion path.

* fix: keep reloading tabs live and spare pastes an active run is using

Three findings:

- pagehide cannot tell a reload from a close, and the tab id survives a reload
  on purpose, so releasing the claim there handed this tab's own draft to
  another one while the document was still bootstrapping. A closing tab is left
  to the ordinary liveness window instead, which is what the window is for.
  Entering the back-forward cache is still marked, since that heartbeat freezes.
- The text-ownership guard only covered the shared composer keys, but a
  conversation key is reachable from every tab viewing that chat and is stamped
  the same way, so one tab could overwrite text another was holding behind its
  attachments. The guard now applies to any key; the ownership stub is still
  only created for the shared keys, which tabs otherwise share freely.
- Submitting empties the file map but leaves the draft's paste provenance until
  the final SSE event, so New Chat during a streaming response treated the empty
  composer as still owning what the message had just sent and deleted files the
  message, and the run reading them, still referenced. The provenance promotion
  is skipped while a run is in flight.

* fix: give each tab its own presence record and publish live attachments

Four findings:

- Tab presence lived in one shared localStorage map, so two tabs beating at the
  same time read the same snapshot and wrote back rival copies; the loser
  disappeared until its next beat, long enough for another tab to treat its live
  draft as abandoned. Each tab now writes only its own key, and expired records
  are swept while reading.
- With draft saving off nothing is written to a draft at all, so a file
  reattached in another tab was invisible to a retry running here and could be
  deleted underneath it. A tab now publishes what its composers are holding into
  its own presence record, and cleanup unions that with the drafted ids.
- The record written when another tab owns the pending key kept only attachment
  ids, so a restored chip stopped being recognised as a paste and lost editing
  and cleanup. Provenance is rebuilt from the session registry. The unsent paste
  text cannot come along: this tab was refused that key all run, so it was never
  stored anywhere to carry.
- New Chat with draft saving off skipped every embedded record, leaving an
  unsent file-search paste with its metadata, storage and vectors intact. A
  paste this composer owns is now included with its real embedded value, while
  other embedded files are still left alone.

* fix: elect one cleanup worker, scope the queue to its account, guard edit writes

Three findings:

- Every mounted useNewChat (header, sidebar, mobile bar, shortcuts) entered the
  cleanup effect against one shared store, so a single retry issued the same
  DELETE several times and toasted about each. A pass is now claimed before it
  runs; an instance that is turned away asks for a later one rather than
  dropping the work.
- The retained queue outlived a sign-out, so the next account retried the first
  one's payloads, was refused by the ownership check, and rescheduled forever.
  Logging out clears the queue and cancels the pending retry.
- The paste path checks draft ownership before recording provenance, but the
  edit path did not, so a replacement could be written into a record another
  open tab owns, which that tab could then delete while this one still showed
  the chip. It takes the same check.

* fix: match paste identities everywhere and stop migrations clobbering a foreign draft

Five findings:

- The presence record published only composer map keys, but a restored upload
  is keyed by its temporary id while the value carries the server one, and a
  retained deletion in another tab names whichever it recorded. All three
  identities are published now, matching the local guard.
- Migrating a finished run checked that the pending record was ours but not the
  destination, so a conversation draft another tab owned with attachments on
  screen was overwritten and restamped. Both ends are checked, and the
  non-owner fallback no longer writes over a foreign destination either.
- The live-paste fallback matched the registry against file_id alone, so a
  completed paste, marked under its client upload id, read as somebody else's
  file and its upload survived New Chat. It matches every identity now.
- An upload still in flight has no filepath or source, so no discard path can
  build a payload and the reset drops the chip anyway. Its id is deferred so the
  record is deleted when it arrives, with draft saving on or off.
- An edited paste that had been staged into the code sandbox was re-uploaded as
  a plain context file, since only the file-search case was reconstructed.
  metadata.codeEnvRef is durable and now routes it back to execute_code.

* fix: silence background cleanup and keep cross-tab protection past a send

Four findings:

- The reset path matched the paste registry on file_id alone, the same alias
  gap already fixed in New Chat, so a completed embedded paste read as somebody
  else's file and survived with its vectors. It matches every identity now.
- The background cleanup pass used the ordinary delete mutation, so a storage
  failure that kept failing announced itself on every retry, and success
  arrived minutes after the action behind it. The mutation takes a silent
  option and the retry pass uses it; direct user actions still report.
- Each hook instance loaded the pending-discard list once and was never told
  when another instance wrote it, so work deferred by an instance that then
  unmounted stalled. Writes now notify every mounted instance, which re-read
  and apply only a real change.
- Cross-tab protection sampled only what a composer was holding right then, and
  sending clears both the map and the draft, so a file reattached in another
  tab and then sent could be deleted between retries. A tab now remembers what
  it recently held for ten minutes, which is long enough for the other tab's
  next pass to see it and cancel that deletion for good.

* fix: honour draft ownership in every clear and track what a message consumed

Five findings:

- The ownership contract was only applied at the new call sites; the SSE final
  event, the steering handoff and the debounced text clear still erased records
  through clearAllDrafts and clearDraft. The check moved inside those helpers,
  so every path that clears a draft respects it.
- Only the explicit logout cleared the retained queue, leaving a silent refresh
  that returns nothing and a failed user query to carry it into the next
  account. It clears wherever the session is lost instead, in the one place all
  three paths pass through.
- Using isSubmitting to decide whether a paste was consumed was wrong for a
  stopped or errored turn: those clear the flag without clearing the draft, so
  New Chat afterwards deleted files the turn already referenced. Submission now
  records the ids it took, and those are excluded by name.
- The presence sweep only ran from deletion cleanup, so a profile that never had
  a failed delete accumulated a record per tab until the origin quota ran out
  and draft writes began failing silently. The heartbeat sweeps.
- When a run finished into a conversation another tab owned, this tab's own
  queued text and attachments were dropped for want of a writable destination.
  They stay on the key it does own and are restored from there.

* fix: mint a tab id when the browser has no randomUUID

crypto.randomUUID is absent on insecure origins and in older webviews, and the
throw left the tab with an empty identity: every draft was then written without
an owner and every ownership guard read another tab's record as its own,
reinstating exactly the loss this layer exists to prevent. Falls back to
getRandomValues, then to a local mint. The id only has to tell tabs apart.

* fix: address PR review bot findings

Clear the retained deletion queue on every direct authentication exit, not
just the debounced context update: an empty or rejected silent refresh, a
failed user query, and the external-IdP logout all leave the page without
passing through setUserContext, so the queue survived into whoever signed in
next and retried under credentials the ownership check rejects forever.

Settle the edit lock when a replacement upload is aborted. Removing the
replacement chip mid-upload consumes the lifecycle through onAbort, which the
paste editor never handled, so the source paste kept its Edit and Move-back
actions hidden for the rest of the session and the typed correction was lost.

Keep the temporary-file cleanup payload for whatever the server reports as
failed. The delete route answers a partial storage failure with a 200 carrying
failedFileIds, and the cleanup mutation cleared FILES_TO_DELETE wholesale on
any success, dropping the only automatic retry those orphans had.

Persist both paste registries per tab. They lived in module-level sets, so a
reload kept the files draft but forgot the paste had been consumed, and New
Chat then classified an already-sent paste as unsent and deleted a file the
persisted message still references.

Withdraw discarded ids from tab presence. A removed, moved, or discarded chip
kept its recent entry for the whole window, and the retry sweep read that as
evidence the file had been reattached: it cancelled its own cleanup and left
the failed upload orphaned on the server. Presence records whose heartbeat
cannot be read are skipped rather than rewritten, since giving one a fresh
seenAt would revive a dead tab's claims over every id it still held.

* fix: address second round of PR review bot findings

Stop a settled deletion from undoing the logout clear. A DELETE that was
already in flight when the session ended settles afterwards, and its handler
is the last reference to that payload, so it wrote the departing account's
records straight back into session storage. Clearing now latches retention
shut and only a newly established session reopens it, which also covers the
paste editor's own retention and the discard paths, not just this one writer.

Reinsert a failed paste when nothing durable holds it. A composer the user
has typed into is deliberately left alone while a recovery record exists,
because that record restores at an anchored offset later. When the shared
draft key belongs to another live tab the guard skips the record entirely, so
the upload callback held the only copy and refusing dropped the text outright.
It now goes back in at the offset its anchors resolve to, which is where a
restore from a record would have put it.

* fix: address third round of PR review bot findings

Withdraw attachment presence from this tab only. The sweep cleared the
withdrawn ids out of every tab's recent map, which is the one record a second
tab has left once it has reattached a file and sent it: its composer and its
draft are both empty by then, so erasing that entry handed the next retry a
file it read as abandoned and let it delete the upload out of the message now
referencing it. The withdrawing tab always published what it withdraws, so its
own record is all it needs to touch.

Guard the direct New Chat deletion against other tabs. The retry effect
consults every other tab's drafts and published presence before deleting, but
the discard that runs on New Chat went straight to the request, so it raced
past that guard and could delete a file another tab still had attached or had
already sent. It now consults the same two sources, excluding its own draft
keys and its own presence record, which hold exactly what the discard is
throwing away.

Fix the import order in Presentation.tsx, which CI static checks flagged.

* fix: address fourth round of PR review bot findings

Read this tab's presence before sweeping stale keys. Timers pause while the
machine sleeps, so a live tab can beat again with its own record already past
the liveness window; the sweep reaped it and the write that followed published
an empty presence, and nothing republished it because the file map had not
changed. Another tab's retry then saw no claim on chips this one still had on
screen.

Keep submitted-use evidence when a later chip is withdrawn. The same file can
be sent on one message and reattached afterwards, and once the composer and
draft have cleared, its recent entry is the only cross-tab record that a
message still references it. Withdrawing a chip no longer erases an entry for
an id a submission already consumed; it ages out on the ordinary window.

Clear composer drafts when the account changes. A files draft carries the whole
text of a paste held as a file, and the browser tab keeps its identity across
an in-app account switch, so the ordinary draft restore could hand the next
account the previous one's writing. Both draft families are now dropped on the
sign-in and sign-out paths, ahead of the skipFirst exception.

Spare a submitted paste from the edit path's explicit deletion. Editing or
moving a reattached library file that an earlier message sent deleted the
server record underneath that message, because the draft-ownership check
succeeded and nothing consulted the submitted marker.

Validate an edited paste as a replacement rather than an extra file. The
original is deliberately still attached while the replacement uploads, so the
shared validation counted both and rejected the edit at the file-count or
total-size limit; with a limit of one, a lone paste could never be edited.

Preserve failed rows after a table deletion. The table's own cache update
removed every requested file without consulting failedFileIds, undoing the
partial-aware update and hiding a file whose storage delete had failed.

Document the two deliberate dependency omissions in AuthContext, which CI now
lints at zero warnings because the file is part of this change.

* fix: address fifth round of PR review bot findings

Clear composer drafts on the way out of a session, not only on the way in.
Clearing them from the login mutation missed social sign-in entirely: OAuth,
OpenID and SAML leave through direct links and come back through the silent
refresh, so a different account could arrive in the same tab with the previous
account's drafts and tab identity intact and have its paste text restored. The
draft clearing is now paired with the retained-deletion clearing in one helper
used by every authentication exit, so neither can be wired into a path the
other was missed from.

Rebuild paste provenance when restoring a queued upload. A paste queued during
a run has its pending draft taken by takeComposerDraft, so choosing Edit
message restored the upload into an empty composer with nothing recording that
it was a generated paste. Filtering existing provenance could not recover that,
and an unmarked restored chip is treated as a shared attachment: removing it
would not delete it and New Chat skipped it, orphaning the unsent upload. The
session registry still knows, so it is consulted.

Drop paste provenance when a rejected upload is removed. Validation can reject
a paste before it reaches composer state, and the failure path removes it with
removeFile, but the id stayed in pastedTextIds. That left a record
hasDraftAttachments reads as a real attachment claim with no chip behind it,
and with the file map unchanged nothing pruned it, so it locked every other tab
out of the shared composer key.

* fix: address sixth round of PR review bot findings

Centralise the foreign-claim guard. Every path that deletes an upload has to
ask whether another tab or pane still claims the file, and the guard was being
assembled by hand at each site, which is exactly why it was missing from three
of them. collectForeignAttachmentClaims now builds that set once, and the New
Chat discard, the no-draft reset fallback in useNewConvo, and the paste
editor's explicit deletion all consult it. A record another tab claims is
skipped rather than retained, since it was never this pane's to delete.

Scope presence withdrawal to the pane that owns it. One tab holds several
composers and the presence record is flat, so the hook that won the global
deletion pass swept every pane's entry while knowing only its own file map,
erasing the evidence of a chip a sibling pane still had on screen. Withdrawal
now takes the pane index, and an id another pane still lists keeps its recent
entry too.

Guard destructive draft clearing against text-only claims. claimComposerDraftTab
stamps a key that holds nothing but text, and the write guard ignores a claim
with no attachment behind it, so a tab finishing a run that began as an unsaved
chat cleared the shared new-chat key and took another tab's half-written
message with it. Clearing now honours any live foreign stamp, while text writes
keep their deliberate last-writer-wins behaviour.

Mark queued override files as submitted. A during-run queued message drains
through overrideFiles into the reuseFiles branch and skipped the marker loop
entirely, so reattaching that paste later left isPasteSubmitted false and New
Chat or an edit could delete a file the queued message still referenced.

* fix: address seventh round of PR review bot findings

Treat publishing an attachment as proof of liveness. The publisher carried the
old seenAt over, so a tab whose timers had been paused past the liveness window
published a chip and stayed expired until its next interval tick, long enough
for another tab's cleanup to sweep the record and delete the file under the
chip that had just appeared.

Count a sibling pane as a claim. The foreign-claim helper excluded this tab
entirely, so the pane doing the discarding could not see the other composer in
the same tab and deleted a file it still had on screen. Live claims are now
gathered per pane: only the discarding pane's own entry is left out, along with
this tab's recent map, which is flat and cannot say which pane an id came from.

Keep a tab identity when session storage is unusable. It can be blocked or full
while localStorage still works, and returning an empty id left the document
unattributed, which every ownership and liveness guard reads as no owner, so
tabs could destructively clear each other's attachment-backed drafts. An id
that lives only for this document still tells the open tabs apart.

Scope attachment withdrawal in the deletion hook to the originating pane, and
thread the composer index through ChatForm, FileFormChat and FileRow to supply
it. Removing a file from one side-by-side composer withdrew the id for every
pane, and with drafts off the sibling never republished its claim.

Check foreign claims before deleting a live edit source. The guard only covered
the restored path, because detach returns early for an in-memory upload before
reaching it, so editing a live paste another tab had reattached from the library
deleted the file underneath that tab's chip.

Keep autosaving to the pending key while the destination is not writable. The
preserved queued work was written under the pending key but the destination was
still recorded as the active conversation, so later edits autosaved against a
foreign key and a reload mounted straight onto the destination, losing the work
that had just been preserved.

* fix: address eighth round of PR review bot findings

Make submitted-use evidence durable and readable across tabs. The tab that
retries a retained deletion is rarely the tab that sent the message, and this
evidence lived in the sending tab's session storage, which left published tab
presence as the only cross-tab record. That ages out on a fixed ten-minute
window, so a retry resuming after a longer freeze classified a sent file as
abandoned and deleted it out of its message. It is timestamped in localStorage
now, with a horizon wide enough to outlast any plausible freeze and a hard cap
so a long-lived profile cannot grow it without bound. Paste provenance stays in
session storage, since which chips offer the paste affordances really is
per-tab.

Tie a blocked pending draft to its intended destination. Keeping the pending
key active while a live tab owned the destination left the pending state with
no memory of where it was heading, so any later navigation looked like the
awaited transition and carried the queued text and attachments into an
unrelated conversation.

Defer an in-flight paste on direct conversation resets. Callers that reach
newConversation without going through New Chat left an upload with no filepath
yet unrecorded, so once the request landed nothing remained to delete the
server file.

* fix: restore the composer clear after send

CI e2e caught this: after sending a message with an attachment the chip stayed
in the composer, so the sent message and the composer both showed it.

Two causes, both from keying composer storage off state that lags a render.
`currentConversationId ?? conversationId` is the previous conversation during
every transition, so the file-cache restore ran against the outgoing key and
put the just-sent attachment straight back into the map that the submit had
cleared. The active key is now the conversation unless the switch effect has
deliberately parked storage on the pending key.

Separately, treating any first mount as the awaited pending transition ran the
pending migration on every direct load of a conversation. That is narrowed to a
pending record this tab owns which actually holds something, which is what a
reload with real queued work looks like.

Verified against the two failing specs locally, then the whole mock chat spec:
8 passed.

* fix: address ninth round of PR review bot findings

Stop expiring submitted-use evidence on a timer. The work it has to outlast is
a retained deletion, and those carry no expiry of their own, so any interval
chosen could be outlived by a suspended tab still holding cleanup work, which
is the same bug with a longer fuse. The ledger is bounded by count instead,
evicting oldest first only when it would otherwise grow without limit.

Consult that ledger before retrying a deletion. The retry pass built its
protection set from drafts and published presence only, both per-tab and
time-bounded, so a file sent from a tab that has since been suspended had
nothing left to speak for it. The record is resolved before judging, because
the discard is often keyed by the temporary upload id while the pane that sent
it marked only the server id.

Refresh liveness when withdrawing presence, matching the publication side. A
retained-deletion pass resuming after paused timers withdrew its own entry and
then swept the record as stale, taking sibling panes' claims with it.

Keep queued attachments when neither draft key is writable. With another tab
owning the pending key and a second owning the destination, the effect cleared
the live map and could persist it nowhere, so unsent attachments vanished the
moment the run got its conversation id.

Remove the replacement provenance when an edited paste is not accepted. The
edit path records the replacement id before routing the upload, and a rejected
upload left a provenance-only draft that reads as a live attachment claim with
no chip behind it.

Verified with the mock chat e2e spec after rebuilding the frontend: 8 passed.
2026-08-25 20:06:59 -04:00
Danny Avila
6d499ba3ce
⏳ fix: Anchor Resumed Elapsed Time at the Generation's Real Start (#15204)
Some checks failed
Publish `librechat-data-provider` to NPM / pack (push) Waiting to run
Publish `librechat-data-provider` to NPM / publish-npm (push) Blocked by required conditions
Docker Dev Images Build / build (Dockerfile, librechat-dev, node) (push) Waiting to run
Docker Dev Images Build / build (Dockerfile.multi, librechat-dev-api, api-build) (push) Waiting to run
Sync Helm Chart Tags / Ignore non-main push (push) Waiting to run
Sync Helm Chart Tags / Sync chart tags (push) Waiting to run
Publish `@librechat/client` to NPM / pack (push) Has been cancelled
Publish `@librechat/data-schemas` to NPM / pack (push) Has been cancelled
Publish `@librechat/client` to NPM / publish-npm (push) Has been cancelled
Publish `@librechat/data-schemas` to NPM / publish-npm (push) Has been cancelled
* ⏳ fix: Anchor Resumed Elapsed Time at the Generation's Real Start

A reload emptied the Recoil anchor, so the indicator fell back to its
mount time and visibly reset to 0s over a run that had been generating
for much longer. The stream status the resume path already reads carries
the server-recorded generation start; the fill now prefers it, so a
reattached run reports real elapsed time. The fill remains fill-only:
a same-session reattach keeps its original ask baseline, and the
indicator's existing clamp absorbs any client/server clock skew.

* 🕰️ fix: Rebuild the Resumed Baseline From the Server-Computed Age

Codex round 1: anchoring at the server's raw createdAt compares two
clocks — a client behind the server froze the resumed reading at 0s for
the skew, one ahead inflated it. The status route now also reports the
generation's age computed on its own clock, and the client rebuilds a
clock-local anchor as Date.now() minus that age, so each machine only
ever compares to itself. Raw createdAt stays as the fallback for an
older server mid-rolling-deploy.

* 📥 fix: Compute Elapsed Age in TypeScript, Anchor It at Status Receipt

Codex round 2: the elapsed computation moves into packages/api as
getGenerationElapsedMs — the route now just delegates, keeping the
response contract type-checked and the /api surface a thin wrapper —
and the client subtracts the age from the moment the status response
arrived (dataUpdatedAt) rather than from apply time, so a slow history
fetch between receipt and apply can no longer shrink the reading.
Declined with rationale: a shared clock source across replicas — the
residual is inter-replica NTP drift, milliseconds against the
minutes-scale client skew this PR eliminates, and the helper gives any
future shared-clock upgrade a single home.
2026-08-25 09:35:10 -04:00
Danny Avila
69e7c73614
🎛️ feat: Expose Authoritative Subagent Controls (#15169)
* feat: expose authoritative subagent controls

* fix: reconcile subagent control races

* fix: reconcile durable control conflicts

* fix: preserve authoritative subagent control outcomes

* fix: fence subagent controls to child thread

* fix: validate subagent control targets before routing

* fix: close subagent control boundary gaps

* fix: keep control reservations private

* fix: close subagent control admission gaps

* fix: preserve authoritative control history

* style: sort subagent control imports

* fix: preserve authoritative subagent control retries

* style: sort control state imports
2026-08-24 20:37:49 -04:00
Danny Avila
d85bea41cc
🧪 ci: Mock the Parent Subagent Index Handler in Convos Route Tests (#15146) 2026-08-23 22:54:19 -04:00
Danny Avila
fc2b8584c4
📇 feat: Surface Event Child Activity Through a Bounded Parent Index (#15142)
* feat: surface event-driven child activity

* fix: keep child task aggregation documentdb-compatible

* fix: address event activity review findings

* test: provide markdown message context defaults

* fix: report bounded child history truncation

* fix: preserve current child activity state

* fix: preserve durable event child activity

* fix: handle missing task timestamps

* fix: keep active event snapshots live

* fix: preserve event activity across valid anchors

* fix: close event child activity gaps

* fix: preserve event activity across resume
2026-08-23 18:50:00 -04:00
Danny Avila
d864597731
⚡ perf: Append Saved Message Ids Instead of Rebuilding the Conversation Array (#15141)
* ⚡ perf: Append Saved Message Ids Instead of Rebuilding the Conversation Array

Every saveConvo read every message id in the conversation (sorted) and
wrote the array back onto the document — twice per chat turn, O(n) in
conversation length, from a write path. The turn's savers know exactly
which message they just wrote, so they now pass it as
metadata.appendMessageIds and saveConvo $addToSet-s it, skipping the
read and the full-array rewrite. Every save without the option — titles,
archive, fork, import, threads — still rebuilds from the database, which
remains the heal point for the drift that message deletion has always
left behind (deletes never ran saveConvo).

The array's consumers read presence or length, or use it as an
optimistic cache placeholder, so incremental maintenance is
behaviorally identical; on traced turns the array stays exactly equal
to the messages collection.

Per-turn queries: 15 -> 13 (two Message.find gone), and the growing
array payload no longer crosses the wire twice per turn.

* 🎯 fix: Brand the Lineage-Only Resolved Conversation Instead of Guessing by Shape

The resolved-conversation files fast path treated an absent files
property as unresolved so the lineage-only partial from a bound
agent-event continuation could not silently hide a conversation's
uploads. But MongoDB never stores an empty files array, so nearly every
real conversation also lacks the property and the fast path never fired
— a follow-up turn on an upload-free conversation still paid the
getConvoFiles round trip.

The synthesized partial is the one object that cannot speak for the
database, so it now carries an explicit symbol brand
(PARTIAL_RESOLVED_CONVERSATION, non-serializing and invisible to key
iteration), and a stored document without files means what it means:
no files. Traced follow-up turns drop from 14 queries to 13.

* 🧪 test: Expect the Appended Message Id in the Route's saveConvo Metadata

messages-get.spec.js pins the exact metadata POST /api/messages passes to
saveConvo; the route now forwards the saved message's _id as
appendMessageIds, which is the behavior the append path depends on.
2026-08-23 16:52:44 -04:00
Danny Avila
f3c6e24f84
⚡ perf: Cut Serial Round Trips and a 100ms Admission Stall from Chat Turns (#15138)
* ⚡ perf: Stop Awaiting the Conversation Access Marker Write

Without Redis the CONVO_ACCESS violations namespace is backed by keyv-file,
whose debounced write resolves after ~100ms. validateConvoAccess awaited
that write before calling next(), so the first message to any existing
conversation waited ~100ms before the request was even admitted — once
per conversation per ten-minute window, on every default deployment.

The marker only short-circuits the next check, so the write no longer
gates the request. The same read now stashes the full document on
req.resolvedConversation (null when absent) for downstream consumers.

First-turn ack on an existing conversation: 109ms -> 5ms.

* ⚡ perf: Read the Conversation Once per Chat Turn

A chat turn read the same conversation document four times: the access
check (two fields), the subagent thread guard (full document), agent
initialization (the files field), and the first save. The access check
now reads the full document and leaves it on req.resolvedConversation,
the guard accepts that pre-resolved document instead of re-reading, and
initializeAgent takes the conversation's file refs from it rather than
issuing a separate findOne.

Two serial round trips removed from every turn; the same document still
serves the first save as before.

* ⚡ perf: Remove Duplicate JWT Authentication on Agents Routes

routes/agents/index.js applies requireJwtAuth and then mounts the v1
router at '/', which applied requireJwtAuth again. Every request through
the agents router — chat turns included — ran the passport strategy
twice: two signature checks and two user document reads. The v1 router
is mounted nowhere else; its separately exported avatar router carries
its own auth in files/index.js.

* ⚡ perf: Skip the History Read for Root-Parent Turns and Walk the Tree in O(n)

loadHistory fetched every message in the conversation and then walked the
parent chain from the request's head. For a new conversation — or a new
branch from the root of an existing one — the head is the root sentinel,
which no message carries as its id, so the walk was empty by construction
and the fetch was wasted. It now returns early.

getMessagesForConversation found each ancestor with Array.find inside the
walk, O(n^2) on a linear conversation (~5ms at 1000 messages). A Map by
messageId makes it O(n); first-match semantics are preserved.
2026-08-23 15:17:35 -04:00
Danny Avila
1de88e7e91
📨 feat: Continue Bound Child Agents from Events (#15112)
* feat: add authenticated agent event ingress

* style: sort agent ingress imports

* fix: harden agent event ingress

* fix: bind event provenance to API keys

* fix: inspect event input with legacy PII filters

* fix: scope event status reads to source keys

* fix: bind event status reads to remote sources

* feat: add bound event-driven child turns

* fix: harden event-bound child continuations

* fix: satisfy event binding type contracts

* fix: close event actor lifecycle races

* fix: harden event actor dispatch continuity

* fix: fence event actor resume lifecycle

* fix: bind event actor state to lifecycle

* fix: preserve cascade write outcomes

* test: type cascade failure injection

* style: sort cascade test imports

* fix: harden event child lifecycle boundaries

* fix: make event cleanup retryable

* fix: annotate event retention clock

* fix: reconcile partial cascade metadata

* fix: recheck event binding expiry on resume

* fix: fence event actors by retention deadline

* fix: close event actor lifecycle races

* fix: harden event child lease acquisition

* fix: lazy-load event child lease adapter
2026-08-23 01:15:57 -04:00
Danny Avila
d3e70159ca
📡 feat: Stream Detached Subagent Activity (#15111)
* feat: stream detached subagent activity

* fix: annotate activity stream limits

* fix: isolate subagent activity imports

* fix: harden detached subagent activity lifecycle

* test: cover synchronous activity transport failure

* test: include required subagent activity identity

* fix: identify and reconnect subagent activity events

* fix: bound subagent activity lifecycles

* fix: close subagent activity handoff races

* fix: bind and synchronize activity subscriptions

* fix: detect fresh activity attachment

* fix: complete activity synchronization handoff

* fix: bind activity sync and failure circuits

* fix: expose subscription-bound synchronization

* fix: fence activity reconnect publications

* test: make detached timeout settlement deterministic

* fix: fence Redis activity attachments

* fix: close failed activity streams

* perf: reuse fenced activity frontier

* style: sort subagent thread imports

* fix: preserve queued subagent activity

* test: type activity publication counter

* fix: disconnect subagent activity subscriber

* fix: close background activity lifecycle gaps

* fix: preserve streamed activity spacing

* fix: preserve bounded live subagent activity

* fix: merge durable subagent activity safely

* fix: model detached activity coverage

* fix: type detached activity inputs

* fix: order overlapping subagent activity

* chore: sort activity test imports

* fix: buffer subagent activity handoff gaps

* fix: flush activity after parent close

* fix: advance closed activity suffixes

* fix: preserve detached activity ordering

* fix: close detached activity delivery races

* fix: bound shared Redis subscriber readiness

* fix: expire shared Redis subscription readiness

* fix: clean up late Redis subscriptions

* fix: preserve late Redis subscription fallback
2026-08-22 09:45:27 -04:00
Danny Avila
3ebef4c84e
📨 feat: Add Authenticated Agent Event Ingress (#15110)
* feat: add authenticated agent event ingress

* style: sort agent ingress imports

* fix: harden agent event ingress

* fix: bind event provenance to API keys

* fix: inspect event input with legacy PII filters

* fix: scope event status reads to source keys

* fix: bind event status reads to remote sources
2026-08-22 01:09:04 -04:00
Danny Avila
67b7b441b2
🛂 feat: Filter Model-Bound Content by Source (#14425)
* feat: introduce optional content protection seam

* feat: enforce source-aware content filters

* feat: complete source-aware content enforcement

* test: activate skill file-text fail-close fixtures

* fix: harden source-aware content filters

* fix: harden model-bound content filtering

* fix: preserve legacy filters and generated files

* fix: inspect shared scalar metadata

* test: align mocks with current dev dependencies

* feat: add persisted content filter safeguards

* feat: complete source-aware content filter enforcement

* fix: move resume content preflight into TypeScript

* fix: close content inspection edge cases

* fix: harden content protection boundaries

* fix: complete content protection safeguards

* test: align persisted memory filter coverage

* fix: reconcile content protection with current dev

* fix: reconcile content protection with latest dev

* fix: close content protection review gaps

* fix: enforce source-aware provider boundaries

* fix: preserve legacy PII preflight semantics

* test: stabilize stored branch preflight fixture

* fix: defer agent writes until protected model admission

* perf: harden source-aware model-bound filtering

* fix: canonicalize provider lineage before validation

* fix: satisfy model-bound callback type checks

* perf: Bound content protection filtering work

* fix: Bound submission array traversal

* fix: Stabilize bounded content snapshots

* fix: Scope model-bound traversal overflows

* fix: Preserve scoped content inspection

* fix: Accumulate aggregate traversal scopes

* fix: centralize content policy boundaries

* test: align deferred tool policy context

* test: align controller policy mocks

* style: normalize content protection imports

* fix: close content policy review gaps

* fix: narrow active skill policy config

* fix: address content protection review boundaries

* fix: retain exact provenance overflow sentinel

* fix: preserve literal and scoped provenance updates

* fix: narrow persisted edit provenance

* fix: isolate exact overflow attribution

* fix: centralize stored prompt protection

* fix: fail closed on incomplete transcript evidence

* fix: align canonical transcript routing

* refactor: centralize content policy preflights

* fix: isolate upload policy error typing

* style: sort policy preflight imports

* refactor: centralize content policy boundaries
2026-08-21 22:43:32 -04:00
Marco Beretta
199de92c51
⚡ perf: Warm Feature Catalogs in the Background After First Paint (#15047)
* perf: warm feature catalogs in the background after first paint

Prompt groups and MCP server/tool queries no longer fire on the app
startup path. A catalog warmup store releases them after first paint
on browser idle, staggered with jitter so a fleet of clients does not
burst the API all at once. Panels opened before warmup activates
their catalog immediately and fall back to their existing loading
states. The prompts list endpoints now also run their independent
access lookups in parallel instead of in three serial rounds.

* perf: gate MCP icon observers and re-arm warmup across sessions

MCP icon/name observers mounted from rendered messages now wait for the
warmup gate like every other server-catalog consumer, so conversations
with MCP tool calls no longer pull the server list onto the first-render
path. The warmup schedule resets on logout so a second login in the same
tab warms on its own stagger instead of releasing every catalog at once.
Panel mount activations now require a visible sidebar, since a persisted
active panel stays mounted while hidden.

* perf: void stale warmup callbacks and gate the agent panel tools query

Reset now bumps a generation captured by every idle callback and its
stagger timer, so callbacks pending across a logout can no longer release
catalogs into the next session. The agent form's MCP tools query keeps
its own readiness gate so a hidden persisted panel cannot pull the tools
request ahead of its stagger once the server list resolves.

* perf: reset warmup on Root unmount and honor the insights route collapse

Root can unmount in the same render that flips authentication on logout,
so the warmup effect now resets from its cleanup as well as the
unauthenticated branch. Panel activations mirror UnifiedSidebar's
panelExpanded condition, treating the insights route as collapsed instead
of reading the raw sidebar atom.

* test: re-expand the approval tool card the saved message remounts

The helper opened the card once and then waited on its body. Saving the
response swaps the placeholder message id for the persisted one, which
rekeys every part in the turn: the card remounts collapsed, its body
unmounts, and the output assertion waits out its timeout against a
disclosure nothing is going to reopen. The redis transport lane pays a
round trip per stream event, so its finalization lands late enough to
catch the helper mid-assertion.

Wait for the closing model turn before expanding anything, then re-open
the group and the card on each attempt until the scoped output is on
screen.

* Update AgentPanelContext.tsx import order

---------

Co-authored-by: Danny Avila <danny@librechat.ai>
2026-08-21 20:33:29 -04:00
Danny Avila
8ae94afa91
🪡 fix: Thread Parent Message ID Through MCP Request-Scoped Bodies (#15095)
* fix: Unify MCP request-scoped headers

* fix: address request-scoped MCP review findings

* test: preserve request scope on status errors

* fix: treat authorized on-demand MCP servers as ready

* refactor: separate MCP readiness from connection state

* fix: preserve on-demand MCP readiness labels

* test: satisfy OpenAI conversation ownership guard

* fix: keep MCP action predicates boolean

* fix: close deferred MCP request context gaps

* fix: preserve on-demand MCP configuration actions

* fix: fail closed on unavailable MCP parent context

* test: complete MCP connecting-state mocks

* fix: preserve missing MCP parent on continuations

* fix: align native MCP request identities

* fix: preserve edited MCP parent identity

* test: use scoped Agent initializer fixture

* test: expose MCP request body helper

* fix: preserve MCP turn identity across resume

* style: sort stream metadata imports

* fix: carry normalized MCP identity to execution
2026-08-21 16:33:01 -04:00
Oliver Faust
b399ad8370
📄 fix: Serve Stored Text for "Upload as Text" File Downloads (#14723)
* 📄 fix: Serve Stored Text for Text-Source File Downloads

"Upload as Text" attachments store extracted content in the DB with
source 'text'; OCR uploads persist the OCR strategy name (e.g.
'mistral_ocr') as a filepath placeholder since no backing file exists.
The download route resolved these records to the local strategy and
passed the placeholder to fs.createReadStream, which failed with
ENOENT — and the response was never ended after the stream error, so
the request hung until the client timed out.

Serve the stored text directly as a .txt download for text-source
files (re-fetched by _id, as getFiles excludes 'text' by default),
and end the response on stream errors: 500 without the download
headers before headers are sent, otherwise abort the truncated
response so clients detect the failure.

* fix: Preserve text-source preview semantics

* fix: Complete text-source download coverage

* fix: Tie text downloads to blob lifecycle

* fix: Isolate preview downloads and share text snapshots

* fix: Keep shared previews in share scope

* style: Sort text download imports

---------

Co-authored-by: Danny Avila <danny@librechat.ai>
2026-08-21 14:24:43 -04:00
Marco Beretta
29f6ec6eae
🙈 feat: Config Option to Hide Response Feedback Buttons (#15085)
* feat: add interface option to hide response feedback buttons

Adds `interface.feedback` to librechat.yaml. When set to false, the
thumbs up/thumbs down buttons are removed from the message action row
and the feedback endpoint rejects writes with 403, so deployments that
do not consume the data can stop collecting it. Defaults to true.

* refactor: hoist the feedback gate out of the message row and into typed middleware

Reading startup config inside HoverButtons put a query observer and two Recoil
subscriptions on every message row, and rows never unmount, so the cost grew
with the conversation. Resolve the flag once per chat in useChatHelpers and
carry it on TMessageChatContext; useMessageActions withholds handleFeedback
when it is off, which the action row already treats as "no feedback controls".
The flag now stays false until the config resolves, so a disabled deployment
never flashes controls whose writes are rejected.

Move the server-side policy into requireFeedbackEnabled under packages/api so
the route keeps no policy of its own.

* test: stub the feedback gate in specs that replace the api package

The messages router now imports requireFeedbackEnabled, and express rejects an
undefined handler at require time, so every spec that mocks @librechat/api
wholesale has to carry the export.
2026-08-21 11:16:44 -04:00
Danny Avila
634432b2ae
🪟 feat: Read Child Threads Through Their Parent (#15073)
* feat: add parent-scoped subagent thread reads

* fix: tighten child thread read bounds

* perf: project child activity messages

* test: update child activity route fixtures

* test: satisfy response mock types

* fix: bound child activity reads at storage

* style: sort child activity imports

* fix: bound child activity storage reads
2026-08-21 11:10:26 -04:00
Danny Avila
8c14f03432
🗂️ feat: Scope Scheduled Chats to Chat Projects (#15056)
* 🗂️ feat: Scope Scheduled Chats to Chat Projects

Adds an optional chat-project destination to a schedule, plus the operator
config to require one — or to pin every scheduled run to a specific project.

Feature:
- `chatProjectId` on the schedule row, accepted on create/update, projected on
  the wire, and carried into the run's conversation through the durable trigger
  envelope's `run` context.
- `interface.schedules.requireProject` refuses schedules that are not filed
  under a project; `interface.schedules.projectId` pins every run to one
  project and implies the requirement.
- Dialog gains a project picker (required when configured, a read-only row when
  pinned); the card shows the destination and the new disabled reasons.

Invariants:
- ONE resolver (`resolveScheduleProjectId`) decides the destination for the
  write handler, the fire path, and the wire projection alike, and an operator
  pin OUTRANKS the stored id in all three. Tightening the config therefore
  redirects — or stops — existing schedules instead of grandfathering where
  their runs land. A pin implies `requireProject` for the same reason: without
  it, a row created before the pin would keep firing with no project at all.
- Create/fire precheck symmetry, mirroring `resolveAgentFireAccess`: a write
  this handler accepts is one the next fire also accepts. Any edit leaving a
  schedule ENABLED re-validates its EFFECTIVE (possibly stored) project, like
  the existing stored-agent and cadence-floor rechecks. A DISABLING edit skips
  the requirement, or a schedule auto-disabled for `project_required` could
  never be turned off.
- Fire-time enforcement auto-disables rather than filing runs loose, matching
  agent_deleted: new `project_required` (requirement raised after creation) and
  `project_deleted` (gone, or pinned to a project this owner does not have)
  reasons, both refused BEFORE a billed generation is dispatched, and both
  advancing so a schedule can never wedge on the occurrence.
- `computeCreateDigest` appends the field only when present, so a payload
  without a project digests byte-identically to one from before this change —
  a create retried across the upgrade still matches its own row instead of
  reading as key reuse.
- Project reads are scoped to the owner, so ownership and existence are the
  same lookup; a read ERROR propagates instead of failing closed, so a Mongo
  blip retries the fire rather than auto-disabling the schedule.

The trigger idempotency key hashes principal/event/target and never
`envelope.run`, so the added run field cannot destabilize delivery identity.

* 🗜️ fix: Keep the Schedule Dialog Inside Its Height Budget

The project picker landed as a new ROW in the schedule dialog, which broke the
e2e edit spec: `md:overflow-visible` turns off the template's scrolling from
`md` up, so the dialog's content must fit the viewport. The extra row pushed the
footer's Save button below a 720x1280 window, where Playwright reported a
visible, enabled button it could never click — 226 scroll-into-view retries and
a 2-minute timeout, on all three attempts.

Measured against `dev` at 1280x720 (Save button's bottom edge, viewport 720):
  dev              688   (32px slack)
  project row      ~790  (off-screen, CI failure)
  3-column row     704   (16px slack — half the budget spent)
  this commit      686   (34px slack, 2px better than dev)

The identity row is now three columns — name, agent, project — and its caption
moved out of the agent cell to sit full width beneath the row: at a third of the
dialog that sentence wraps an extra line, and the row is the tallest thing
competing for the budget. The caption is grouped with the row rather than left
to the form's own 4-unit rhythm, which spent more height on the gap than the
caption occupies.

The e2e spec now asserts the button is in the viewport before clicking it, so
the next field that overflows this dialog says so in one line instead of a
two-minute timeout on a visible element. FOLLOWUPS.md records what the planned
dialog controls (multi-day weekly, timezone, attachments) need first: give
`ControlCombobox` the `portalElement` prop `Dropdown` already has, portal the
popovers into the dialog content, and let the form scroll again.

* 🧹 fix: Address Codex Review on Scheduled Chat Project Scope

Four P2 findings, all real.

Unreachable clearing path (ScheduleDialog). The picker only held live projects, so
`com_ui_schedule_project_none` was a PLACEHOLDER — nothing selectable. Once a
schedule had a project the owner could never take it away, leaving the server's
`chatProjectId: null` path reachable only by API. The picker now carries a real
"No project" option whenever a project is optional, and omits it when one is
required, where there is nothing valid to select.

Placeholder shown for a real project (ScheduleDialog). A stored or pinned project
outside the first loaded page had no name in the paged map, and the combobox
renders its placeholder for an empty display value — telling the owner a scoped
schedule had no project. That one project is now read by id, with the raw id as a
last resort: a poor label, but an honest one.

Project policy skipped at the resume boundary (service.ts). `claimScheduleResume`
re-applied the schedules gate, the revision fence, the kill switch and
SCHEDULES:USE, but not the project policy this PR added. Approving a paused run
whose project was deleted — or whose owner now sits under a requirement or a pin
it no longer satisfies — billed a continuation the very next scheduled fire would
refuse and auto-disable the schedule for. The effective-project resolution now
runs there too, refused before the lease and the capacity slot so a policy refusal
costs nothing and leaves no state to unwind. NOTE: agent access and balance are
still not rechecked on resume; that gap predates this PR and is left alone.

Per-card project derivation (ScheduleCard). Every card ran the projects hook and
rebuilt the full option array, name map, and one icon element per project, to use
a single name — O(schedules x projects) per render and per project-list refresh.
The hook is split: `useChatProjectNames` (map only, for the panel, which resolves
every card's name once and passes it down) and `useChatProjectPicker` (options and
pagination, for the dialog's one combobox). The panel skips the query entirely
until some schedule actually has a scope.

Tests: four at the resume boundary (verified failing without the gate) and four in
the dialog spec. The picker selection in one existing test now goes through the
search field — the popover's VIRTUALIZED renderer sizes its window from a scroll
height jsdom always reports as 0, so with three options it materialized only two.
Full schedules e2e re-run green against the rebuilt client.

* 🎯 fix: Settle Project-Policy Refusals and Keep Create Retries Idempotent

Second Codex round, four P2s. Three were consequences of the resume gate added in
the previous commit, which was half-built: it admitted where it should not and
stranded the run where it refused.

Project policy moves from `claimScheduleResume` into `isScheduleLive`'s `policy`
branch. Both entry points consult that branch FIRST, and both already route its
refusal through abort-and-settle — so a policy stop now settles the occurrence
instead of answering a bare 409 while the job stays `requires_action`, the card
keeps reading "Needs approval", and every retry repeats the same 409 until expiry.
No change to resume.js: the existing branch does the work.

The rule is deliberately NARROW. It refuses only where no valid destination is
left — the requirement is on with nothing satisfying it, or the schedule's own
project is gone (which also unset it on the conversation). It does NOT refuse
because an operator's pin moved: the paused conversation cannot be rebound
(`chatProjectId` is excluded from the resume context and the continuation reuses
the same conversationId), so refusing would strand a pending approval over a
destination it can never reach, for a pin that governs only where the NEXT run
lands — which the fire path already redirects.

Create retries are idempotent again. Project policy had been applied BEFORE the
`clientRequestId` replay lookup, so a raised requirement, a deleted project, or a
moved pin could answer 400 for a create that already committed — pushing the
client to rotate its key and create a DUPLICATE schedule, the exact failure the
key exists to prevent. Policy now applies only to a genuinely new insert, and the
digest is computed from the CLIENT's payload rather than the resolved destination,
so today's policy can no longer re-digest a genuine retry into a mismatch.

An explicit `chatProjectId: null` under a pin is refused rather than silently
resolved to the pin. The payload contract defines `null` as clearing the scope;
answering 201 while filing under the pin reported success for the opposite of what
was asked. Only an OMITTED field takes the pin silently.

Tests: five on the policy branch (both refusals verified failing without it, plus
guards that a moved pin and a live project still admit) and two on the handlers
(the pinned explicit clear, and a committed create recovered by retry after the
policy tightened — verified answering 400 without the reordering).

* 🧭 fix: Converge the Stored Project on the Destination a Fire Resolved

Third Codex round, four P2s.

The root confusion behind the resume findings: an operator pin outranks the stored
id at fire time, `fireSchedule` sends the pin in the trigger envelope, and the row
keeps its old value. The row therefore LIED about where that occurrence's
conversation went, and every later re-validation — the resume boundary above all —
checked a project the conversation was never filed under. A schedule storing A,
pinned to B, with B later deleted and A still live, was admitted for resume into a
conversation that had just been unscoped.

Fixed at the source rather than at each reader: a fire that resolves a destination
different from the stored one writes it back, claim-token fenced like every other
worker-side write and deliberately WITHOUT a configRevision bump — this is the
server reconciling itself to policy, not an owner edit, and a bump would fence an
in-flight occurrence off its own run. Written only AFTER the destination validates,
so an unusable pin never lands in the row, and best-effort: the envelope already
carries the right destination, so a failed write costs accuracy on a later recheck,
never the run itself. The wire projection already reported the pin, so this also
stops the row and the UI disagreeing.

An explicit `chatProjectId: null` under a pin is now refused on the DISABLING edit
path too. The pin check and the requirement are independent rules, and folding them
together let `{enabled: false, chatProjectId: null}` skip the pin check entirely,
unset the row, and answer with a wire projection still naming the pin. Only the
requirement is waived for a disabling edit.

The dialog no longer requires a project for an edit that leaves a schedule DISABLED.
The server waives the requirement there precisely so a row auto-disabled for
`project_required` can still be renamed or tidied up; requiring it in the form made
that unreachable, and an owner with no projects could not edit the stopped schedule
at all.

FOLLOWUPS.md records the two residual gaps with their exact triggers: the sub-second
deletion race inside the resume claim window (which needs the effective project
persisted per OCCURRENCE plus a distinct policy conflict routed through
abort-and-settle), and the fact that convergence happens only when a schedule fires.

Tests: four on convergence (pin written, no write when unchanged, no write for a
destination that failed validation, fire survives a failed write), two on the
disabling-edit pin rules, two in the dialog. Schedules e2e re-run green.

* 🔑 fix: Keep an Explicit Project Clear Out of an Omitted Field's Digest

`computeCreateDigest` appended `chatProjectId` on `!= null`, so an OMITTED field and
an explicit `null` produced the same digest. Because the replay lookup and
`matchesCreateIntent` deliberately run before project policy, a request could reuse a
pinned create's `clientRequestId` while explicitly sending `chatProjectId: null` and
receive 201 describing the pinned row — success reported for the opposite of what it
asked, and the pinned-clear refusal the normal create path applies never reached.

Now `!== undefined`: an omitted field still digests byte-identically to a payload from
before project scope existed, so a create in flight across the upgrade still matches
its own row, while an explicit clear is a distinct intent and digests differently. A
pre-scope client never sent the field at all, so nothing legacy can carry an explicit
null.

* 📍 fix: Validate a Paused Run Against the Project Its Own Occurrence Used

The schedule-wide convergence from the previous commit was not enough, and the
reason is the single-active run index: it covers `status: 'started'` only, so a
PAUSED run does not block the next occurrence. While run 1 sat paused in project A,
a pin move plus one later fire rewrote the schedule row to B — and the resume
policy then validated B while run 1's conversation was still filed under A. Delete
A and the continuation was admitted into a conversation that had just been unscoped.
That window lasts as long as the pause, not the sub-second race the previous commit
documented.

The reservation now records the destination THIS occurrence used, and
`isScheduleLive` validates that record when given the occurrence's `scheduledFor` —
which `resume.js` already reads two lines above the call. No new conflict type and
no settlement-path surgery: the refusal rides the abort-and-settle branch that check
already has.

An ABSENT record falls back to the schedule-level resolution rather than reading as
unscoped. A pre-scope occurrence, or one whose row is gone, must never be treated as
evidence to stop a run — the fallback keeps legacy paused runs behaving exactly as
they do today.

Schedule-wide convergence stays: it keeps the row honest for the UI and for every
check that has no occurrence in hand.

FOLLOWUPS.md now describes the one remaining gap accurately — a deletion inside the
claim window, which needs a distinct policy conflict routed through abort-and-settle
rather than the bare 409 an `inactive` conflict produces.

Tests: three on occurrence-vs-row precedence (the decisive one verified failing
without the lookup), two on what the reservation records, and the resume controller
spec now pins `scheduledFor` in the policy call. Schedules e2e re-run green.

* 🏷️ fix: Tell a Deliberately Unscoped Occurrence From an Unrecorded One

The occurrence fallback added in the previous commit conflated two different
absences. A post-upgrade run that deliberately went unscoped omitted the field
exactly like a row written before the field existed, so both took the fallback —
and a paused unscoped run was then validated against the schedule's CURRENT
project. Under a requirement or a pin added while it sat paused, that admitted a
billed continuation into a conversation satisfying no present policy.

The reservation now ALWAYS records its decision, `null` for unscoped, and the read
reports `recorded` from key PRESENCE rather than truthiness. Only an unknown record
— a pre-scope row, or no row at all — falls back to the schedule-level resolution;
a recorded null is the genuinely unscoped occurrence it says it is, and is refused
once a project becomes required.

The distinction rests entirely on a stored `null` surviving as a present key while a
never-written field stays absent, so that is asserted against real Mongo rather than
assumed: if it ever stopped holding, unscoped runs would silently start being
validated against the schedule's current project again.

Tests: two against mongodb-memory-server (recorded null vs never-written vs missing
row), plus refusal of a recorded-unscoped occurrence under a new requirement and the
preserved fallback for a pre-scope one. Schedules e2e green.

* 🔁 fix: Validate an Initial Scheduled Start Against Its Own Occurrence

The initial-start policy check in `request.js` called `isScheduleLive(..., { policy:
true })` without `scheduledFor`, so it fell back to the schedule-level resolution
even though the run row already carries the occurrence's recorded scope. An
occurrence reserved unscoped, with a pin introduced while its loopback request sat
queued, was therefore admitted against the new pin — producing a billed unscoped
conversation under a requirement it does not satisfy, from an envelope already built
without a project.

`scheduledFor` was already in scope there. Passing it makes the initial start and the
resume validate the same way: against the destination the occurrence itself recorded.
2026-08-21 11:09:04 -04:00
Danny Avila
d0f9d5625e
🧵 fix: Close Child-Thread Read and Search-Cleanup Gaps (#15055)
* fix: close child thread read and cleanup gaps

* fix: preserve child search cleanup invariants

* test: complete mocked update result

* perf: parallelize scoped message reads

* fix: close child thread compatibility gaps

* test: expect preserved cleanup failure

* fix: reconcile legacy Meili cleanup markers
2026-08-21 00:44:37 -04:00
Danny Avila
c5276fc63d
⏱️ feat: Run Scheduled Chats Through Durable Agent Triggers (#14939)
* feat: Scheduled Chats — agent-centric scheduled runs creating real conversations

Squash of the full review-hardened branch (PR 14540, supersedes 14373) onto
latest dev, preserving the exact verified tree. History prior to this commit
lived on the pre-squash branch; every invariant below survived 25 Codex review
rounds plus two external audit rounds (R26) with regression tests that fail
without their fixes.

Feature:
- Schedules CRUD + side-panel UI (cadence dialog, run cards, Run Now), roles/
  permissions (SCHEDULES:USE), interface.schedules availability, per-user limits
  and capacity slots, timezone-aware cadence with DST-conservative floors and
  misfire grace.
- Engine: single-process claim/fire loop with leases, loopback POST dispatch
  (signed schedule-fire JWT claims, per-occurrence idempotency key inside the
  route's clientRequestId charset), overlap/balance/capacity/duplicate skip
  policies with auto-disable streaks (too_many_failures, insufficient_balance),
  reconciliation from retained terminal-job evidence, erasure sweep.
- Scheduled runs create real conversations through the resumable agents chat
  path: HITL pauses surface on the card (requires_action), resumes re-apply the
  fire boundary's admission policy (revision fence, enabled, global kill switch,
  SCHEDULES:USE, availability) before continuing a billed generation.

Correctness invariants (the audit surface):
- Settlement discipline: every persistence-producing write happens-before a
  run's terminal outcome write; Stop/complete/pause race through single-winner
  terminal CAS claims (dev's TerminalJobClaim substrate) with retained,
  completedAt-less evidence for scheduled fires plus an owner-intended outcome
  stamp (scheduleOutcome) the reconciler prefers over re-derived success —
  round-tripped through the Redis hash mapper.
- Swallowed generation failures (client error content parts) classify to
  error/skipped_balance instead of success on both initial and resumed paths;
  stale stamps are refreshed evidence-first when persistence plus the Mongo
  outcome write both fail.
- Abort honesty: delivery judged by generation ownership and the CAS's actual
  from-status; republication escalates to the transport's acknowledged variant;
  the Stop route settles only genuinely paused runs.
- Account deletion: one-way barrier (deletionRequestedAt) with auth-cache
  tombstone-before-stamp, boundary rechecks across all auth strategies, quiesce
  of scheduled + interactive work with durable per-stream abort fences
  (positive-evidence acknowledgement only), owner-side finalization markers for
  post-terminal billed writes, deferred-deletion sweep (explicitly ensured
  partial index) that completes cascades autonomously. Remote OpenAI-compatible/
  Responses requests are documented as outside the quiesce and tracked in issue
  14594.
- Store compatibility: finalization markers optional on the legacy IJobStore
  contract with coherent degradation (registration fails -> synchronous title
  fallback; count reads 0).

* fix: fence terminal response persistence from deletion; retain stale-pause evidence

Two blockers from the third external review round.

Terminal persistence visible to account deletion:

- The finalization marker was registered only when post-terminal TITLE work was
  possible, but every persistence-owning terminal CAS opens the same window: the
  claim drops the job out of the active set BEFORE the response save (and the
  background user-message/convo saves), so a deletion quiesce landing there saw
  neither an active job nor a marker and could cascade while the admitted request
  could still recreate messages. Both controllers now register the marker before
  every persistence-owning claim — the fresh-turn path and the HITL resume — and
  release it only after their pending saves (and any post-terminal title) have
  landed, on success and failure paths alike. The TTL bounds a crash.
- settleAbortFence no longer clears a complete/error fence while
  `terminalPersistencePending` is set: terminal at the CAS is not settled while
  the owner is still persisting. The job facade now surfaces the flag.
- The marker trio is REQUIRED by the runtime store contract (assertJobStoreV2
  refuses a store without it at configure time, keeping the failure loud and
  deterministic) while remaining optional on the legacy public IJobStore type for
  source compatibility. The silent degrade path from the previous round is gone —
  it was not deletion-safe.

Stale-pause recovery retains scheduled evidence:

- Three crash/timeout recovery paths — ApprovalLifecycle.failStalePausePersistence
  and the InMemory/Redis stale-pause cleanups — unconditionally stamped
  `completedAt`, putting a scheduled fire's failed-pause error terminal on the
  short completed TTL. A Mongo outage longer than that TTL erased the evidence and
  the reconciler recovered the run as `interrupted` instead of `error`. All three
  now follow the controller-observed path from the previous round: scheduled jobs
  omit `completedAt` (retained-evidence TTL) and stamp the error outcome.

The PR description now explicitly narrows the deletion guarantee for the remote
OpenAI-compatible/Responses paths (tracked in issue 14594).

* fix: deletion-fence marker protocol — generation-scoped, atomic, fail-closed, all terminal paths

One consolidated pass over the finalization-marker mechanism, per the fourth
external review round. The invariant it establishes: NO persistence-owning
terminal CAS runs without a durable, generation-qualified marker covering the
window it opens, and every consumer treats a pending terminal as unsettled.

- Generation-scoped markers. Entries were keyed (userId, streamId), so a
  Stop-superseded generation finishing late could clear the marker its
  replacement registered on the same conversation. Marker fields are now
  qualified by the generation's createdAt; clears must present the same
  identity, and an unqualified legacy clear cannot drop a qualified entry.
- Atomic Redis registration. HSET-then-EXPIRE loses the fresh marker when the
  user's existing hash expires between the two commands (or the process dies
  there); registration is now a single Lua script carrying both.
- Fail closed everywhere. Registration failure (after one retry) now REFUSES
  the terminal CAS instead of proceeding uncovered: the completion claim throws
  into the error path, the error path skips completeJob and leaves the job
  ACTIVE — deletion-visible by itself, recovered by the stale-running reaper —
  and abortJob returns a new retryable `fence_unavailable` failure the Stop
  route answers with 503 and the deletion quiesce treats as an unacknowledged
  stop (fence kept). The previous round's log-and-proceed is gone.
- Every terminal path enrolled. abortJob now owns its window (register before
  the abort CAS, clear in its finally — the Stop route's checkpoint prune and
  partial save run inside beforePublish, between CAS and publication); the
  interactive and background generation-error paths register before their
  completeJob; the resume controller's error finalization registers before its
  completeJob. A lost or thrown claim releases the marker after pending saves
  flush instead of holding the user's deletion behind the TTL.
- Every pending terminal unsettled. settleAbortFence defers on
  terminalPersistencePending for ALL statuses — including `aborted`, whose
  route-side persistence the previous guard missed.

Also: a direct Redis regression for scheduled stale-pause retention (the P2
test gap), and the PR description no longer claims to carry every commit.

* fix: lease-token lifecycle fences — same-generation isolation, admission fence, undelivered-Stop retention, legacy abort enrollment

Fifth external review round; four P1s, handled as the requested consolidated
lifecycle-fence pass.

- Lease tokens. Marker fields were (streamId, createdAt), shared by every
  contender on the same generation — completion and Stop, or two racing Stops —
  so a losing contender's cleanup erased the winner's still-live marker. Every
  registrant now carries a unique lease token in the field and may only ever
  clear its own lease; unqualified legacy clears cannot touch qualified entries.

- Admission fence. Authentication can pass before the deletion barrier goes up,
  and the durable createJob is several async steps later — a deletion quiesce in
  that window saw neither an active job nor a marker and could cascade before
  the admitted request created its job. The controller now registers an
  admission lease and THEN rereads the deletion barrier: the ordering guarantees
  either this request observes the barrier (403, lease released, slot/claim
  cleanup) or the quiesce observes the lease and defers. Held until createJob is
  durable; released on every refusal and initialization-error path. Fail closed
  when the lease itself cannot be registered (503 retryable).

- Undelivered-Stop retention. abortJob released its lease in a finally even when
  delivery AND publication had provably failed — the job reads terminal
  (invisible to active-set scans), a user Stop writes no durable abort fence,
  and the remote owner keeps generating and will persist its abort-catch writes
  whenever the signal finally lands. The lease is now retained in exactly that
  case, and each resignal attempt heartbeats a fresh lease so the fence outlives
  the TTL for as long as delivery is still being driven. The abort-winning
  turn's own loser-side pending saves are additionally fenced in the controller
  catch (best-effort — those writes are already in flight).

- Legacy abort enrollment. abortMiddleware (assistants abort route fallback for
  non-assistants endpoints) awaited abortJob and then spent usage and saved the
  stopped response AFTER the abort's lease was released. Both writes now run
  inside `beforePublish`, between the abort CAS and publication, covered by the
  same lease as every other abort.

Barrier tests, each verified to fail without its fix: same-generation lease
isolation (store), racing two-Stop loser cleanup (manager, stale-read forced
CAS race), undelivered-Stop lease retention, resignal heartbeat, admission
refusal with lease-before-reread ordering plus release-on-durable-create, and
legacy-abort persistence inside beforePublish.

* fix: heartbeat-backed owner-lifecycle leases close the settlement handoff races

Sixth external review round: the remaining P1 interleavings were one structural
problem — lease handoffs that were not atomic — resolved as the requested
consolidated lifecycle-lease pass.

- Quiesce reads leases BEFORE the active-job scan. The admission-lease -> durable
  -job handoff is only atomic against a reader in the OPPOSITE order of the
  writer: writers hold the lease strictly until the job is active-set visible,
  so leases-first shows every interleaving either the lease or the job.
  Jobs-first allowed a request to create its job after the scan and release its
  lease before the count — hiding both, cascading, and letting the new
  generation persist into a deleted account.

- The abort acknowledgement is fenced by the owner-lifecycle lease. Redis ACKed
  the moment the owner's AbortController tripped; the stopping side released its
  lease on that ACK while the owner's asynchronous abort-catch persistence was
  still ahead. The transport now awaits a manager-installed pre-ACK hook that
  registers a DETERMINISTIC owner lease (exactly one owner exists per
  generation, and determinism is what lets the signal-time registrant and the
  owner's catch-side release agree across processes) before the acknowledgement
  is persisted or published; an owned same-replica abort bridges to the same
  lease before tripping its local controller. Both generation-owner catches
  (fresh turn, resume) release it once their writes land.

- A failed replacement handoff no longer orphans the predecessor. The atomic
  replacement removes it from active storage, and an unconfirmed handoff
  terminalizes the replacement too — leaving nothing a quiesce could discover
  while the predecessor's provider may still be generating. Its owner lease is
  now retained at the point the receipt fails delivery; the owner replica renews
  it through the pre-ACK fence when the signal finally lands.

- Leases HEARTBEAT while held. The five-minute store TTL only bounds a crashed
  holder; live persistence — a stalled save, a long deferred title — must never
  outlive its own fence. holdUserFinalization registers and renews every minute
  until released; the controllers' completion/error/admission leases all hold.
  The undelivered-Stop retention moved to a deterministic `stop` lease that
  every resignal attempt renews and the first successful one clears (no more
  opaque leases accumulating to TTL), and a THROWN abort transition releases the
  contender lease instead of leaking it.

- The user-document abort-fence mutations now invalidate the auth user-doc
  cache, matching every other user-doc write.

Barrier tests, each verified to fail without its fix: quiesce lease-scan
ordering (plus the observed-lease defer), pre-ACK fence ordering at the
transport, owned-abort owner-lease bridging, replacement-handoff predecessor
retention, held-lease heartbeat past the TTL, and failed-then-successful
resignal reaping the retained stop lease.

* fix: one manager-owned owner-lease span across every abort delivery path

Seventh external review round; four lifecycle-fence gaps, closed by making the
owner-lifecycle lease a single manager-owned, heartbeat-held span.

- Fail-closed acknowledgements. The pre-ACK hook registered a one-shot lease and
  the transport ACKed even when it failed; a same-replica owned abort likewise
  swallowed registration failure. The hook now acquires a HELD owner lease
  (heartbeat until the owner's catch releases it via releaseOwnerLease) and a
  rejection SUPPRESSES the acknowledgement — the stopping side stays retryable
  behind its retention lease, and every resignal re-drives the handler. The hook
  also stopped gating on `job.createdAt === generationId`: during a replacement
  handoff the store holds the replacement while the abort targets the
  predecessor, and that gate silently skipped exactly the generation being
  acknowledged (the store job is owner identity, never a generation gate). A
  local owned abort acquires the same held lease before tripping its provider;
  post-CAS the trip cannot be withheld, so acquisition failure downgrades
  delivery and the retention handoff keeps the user fenced.

- Committed-but-lost-reply disambiguation. A thrown abort transition released
  the contender lease as if nothing had happened, but a Lua CAS can commit and
  lose its reply — an aborted job invisible to active-set scans whose provider
  was never signalled, with no fence left. The throw path now re-reads the exact
  generation: only a job still live under the caller's identity proves no
  commit; committed or ambiguous outcomes hand the fence to the deterministic
  stop lease (kept on the contender lease if even that fails) before rethrowing.

- Replacement handoff covered end to end. A LOCAL replacement abort acquires the
  predecessor's held owner lease before the trip (failure reports the receipt
  undelivered, engaging retention). Failed-handoff retention is no longer a
  swallowed one-shot: it heartbeats with the durable acknowledgement proof as
  its renewal predicate — acquisition failures keep retrying for as long as the
  fence is needed, and the retainer stands down (without clearing the shared
  field) once the remote owner ACKs and thereby holds its own lease.

- A LOCAL resignal delivery hands off to the owner lease BEFORE clearing the
  retained stop lease, and keeps the stop lease when that handoff fails.

Barrier tests: hook rejection suppressing the ACK, commit-then-lost-reply
retention with its provably-uncommitted counterpart, local-resignal owner
handoff ordering, pre-ACK owner lease held past the store TTL until release
(and provably stopped after), and failed-handoff retention retrying on its
heartbeat — verified fail-before/pass-after by stashing the fixes.

* fix: finish scheduled chat lifecycle hardening

* fix: close scheduled chat review follow-ups

* fix: generation-fence abort recovery evidence

* test: wait for settled approval tool output

* test: preserve scheduled init reconciliation option

* refactor: rebuild scheduled chats on durable agent triggers

* test: reset MCP cache mock between cases

* test: isolate scheduler startup in server specs

* fix: harden scheduled run lifecycle

* fix: normalize schedule capacity conflicts

* test: type schedule collision fixture

* fix: re-fence scheduled resume and expiry

* fix: fence scheduled resume handoffs

* fix: release superseded manual schedule leases

* fix: release failed run-now claims

* fix: release superseded engine claims

* fix: repair schedule dialog interaction and rework its form

The agent picker was unusable: ControlCombobox portals its popover to the
body by default, which lands it outside the dialog's Radix focus trap. Clicks
passed through it, it could not be tabbed into, and the trap fighting Ariakit
for focus locked the page up on selection. The prop is documented for exactly
this case — pass `portal={false}` and give the dialog `overflow-visible`, as
ProjectButton already does. The time and day dropdowns defaulted the same way.

Alongside that:

- Extract the agent builder's instructions editor (special-variable menu plus
  expand-to-fullscreen) into a controlled `VariableEditor` and use it for the
  schedule prompt. Insertions now route through `onChange`, so react-hook-form
  sees them — the schedule PATCH is built from `dirtyFields`, and a `setValue`
  that skipped dirty tracking would have dropped an inserted variable silently.
- Replace the hand-rolled frequency buttons with the shared `Radio`. They marked
  the selection with `bg-surface-hover` on an outline button whose hover is the
  same token, so the selected option was indistinguishable from a hovered one;
  `Radio` is also a real radiogroup rather than four `aria-pressed` toggles.
- Wrap the fields in a real `<form>` and associate the footer button by id, so
  Enter submits. Group the frequency, day and time controls in fieldsets.
- Add placeholders for name and prompt, match the textarea fill to the other
  fields, and label the hourly case as minutes past the hour.
- Widen the dialog to `md:max-w-3xl` and pair name with agent so the form fits
  without scrolling on desktop.
- Move scheduled chats below skills and above prompts in the side nav.

The new dialog spec fails when the portal fix is reverted.

* test: cover scheduled and subagent deletion drains

* refactor: own the form-control appearance in the client primitives

Addresses the codex finding on ScheduleDialog: a feature-local `FIELD_CLASS`
restated the `Input` primitive's border, radius, height and background so it
could be pasted onto the schedule dropdowns, leaving those controls with no
connection to the primitive they were imitating.

Move that appearance into `packages/client/src/components/Field.ts` as the
single source `Input` and `Textarea` now compose, and give `Dropdown` and
`ControlCombobox` a `variant="field"` that applies it. The schedule dialog
passes the variant and carries no class strings of its own.

This also repairs a break the dev merge would otherwise have introduced: the
newer `Dropdown` splits `className` (wrapper) from `triggerClassName`, so the
old pasted classes would have landed on the wrapper and left the triggers
unstyled.

The semantic-token guard now watches the shared module and asserts each
primitive still composes it, which covers more than the two files it read
before.

* fix: keep an explicit schedules disable from becoming an opt-in

`use` is two things at once for a dual-purpose runtime interface field: a
permission bit, which DB overrides strip, and the runtime disable signal that
`getLimits` reads. Stripping it from `{ use: false, maxPerUser: 2 }` leaves an
object, and `getLimits` treats any object without `use: false` as enabled — so
an admin override written to stop scheduled billing for a role or user started
it instead.

Collapse an explicit disable to the boolean form before the strip, on both
paths that accept it: the `interface.schedules` field patch, which admitted the
object wholesale because bare runtime paths deliberately bypass the permission
gate, and the overrides merge, which reached the composite-field branch and
kept `maxPerUser`. Objects that only narrow limits are untouched, so a
principal can still be given a smaller cap.

Both regressions fail without the normalizer.

* fix(schedules): Wave A — null-balance CAS, atomic paused-card clear, clustered erasure sweep

Slice 1 (thread r3804518381): route the existing-null balance initialization
through a { user, tokenCredits: null } compare-and-set instead of a blind $set,
so a concurrent initializer/charge landing between the preflight read and the
write is never handed back its spent starting balance. On a CAS miss the
preflight re-reads the winner. The absent-record $setOnInsert path and the
credited-record refill-config sync are unchanged. Adds the initializeNullBalance
adapter (no upsert) and regression coverage for winner/miss/sync cases.

Slice 2 (thread r3804518388): updateScheduleById now drops a `requires_action`
lastRun projection atomically with the configRevision bump. Any pause present at
edit time was projected under the pre-edit revision and can never be replaced by
its own revision-fenced terminal outcome — a disabling edit would strand the
card on "Needs approval" forever. Implemented as classic-operator CAS branches
(DocumentDB rules out a conditional pipeline $unset), fenced on the card STILL
being the pause so a terminal outcome or newer occurrence that races in is
preserved. Terminal history survives untouched.

Slice 4 (thread r3803826204): expose initializeScheduleErasureSweep from the
schedule runtime facade and start it in every clustered (experimental) worker
after Mongo is up. It re-drives eraseScheduleIfDrained for soft-deleted rows so
a hidden prompt cannot outlive its drain when the delete/erase-on-settle
attempts miss. It arms nothing else and never infers owner death from a
process-local missing job (isTopologySafeToArm gates that).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq

* fix(schedules): Wave B.1 — reversible account-deletion schedule suspension

Thread r3804518383. Account-deletion quiesce marked every schedule `deleting`,
disabled it, and cleared nextRunAt destructively. When a later cascade step (or
the drain itself) failed, the controller cancelled the user-deletion fence —
restoring the user — but the schedules stayed `deleting` and were erased by the
sweep, silently losing all of a live user's scheduled prompts.

Replace the destructive marking with a REVERSIBLE, token-fenced suspension:

- suspendUserSchedulesForDeletion(userId, token) snapshots each schedule's prior
  enabled/nextRunAt under a per-attempt token, then fences firing (disable, clear
  nextRunAt, rotate claimToken). It never sets `deleting`, so a suspended row is
  not erasure-eligible. Snapshotting reads then bulkWrites (a classic update
  cannot copy field values under DocumentDB), fenced per row so an already-
  suspended/soft-deleted/edited row is left alone; idempotent per token.
- restoreUserSchedulesFromDeletion(userId, token) reverses it, re-enabling and
  re-arming only rows still carrying the exact attempt token and not independently
  deleted — so an owner-deleted or newer-attempt-suspended schedule is never
  resurrected.
- deleteUserController generates the attempt token, passes it to quiesce, and on
  any failure that cancels the user-deletion fence restores the suspended rows. A
  successful deletion hard-deletes them (and their snapshots) via the existing
  cascade and never restores.

Adds a `deletionSuspension` embedded field (excluded from the wire schedule),
data-method regression tests (suspend/restore/fence/idempotency/no-resurrect),
and controller tests (restore on drain-false and post-quiesce cascade failure,
no restore on success).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq

* fix(schedules): Wave B.2 — wire the interactive Stop persistence protocol

Thread r3804255932. The durable Stop primitives (requestRunAbort 'stop',
getScheduleRunAbortState, markRunAbortPersisted) already existed but production
only used requestRunAbort(..., 'deletion'). An interactive Stop flipped the job
to `aborted` and then persisted its partial message + checkpoint inside
`beforePublish`, without ever stamping the schedule Stop or acknowledging it —
so reconciliation, the generation owner, or a concurrent schedule/account
deletion could terminalize the run and release its capacity (and erase data)
mid-write.

Wire the request -> persist -> acknowledge -> settle barrier through the
schedule runtime (the route never touches raw Mongo):

- Expose beginScheduledStop / acknowledgeScheduledStopPersistence on the service.
- The abort route stamps the Stop BEFORE signalling abortJob (a serialized
  'in_progress' loser returns 409 STOP_IN_PROGRESS without a second abort),
  acknowledges only after beforePublish persistence succeeds, and on a
  persistence failure leaves the barrier unresolved so the run stays preserved
  (client retries; stale-owner timeout is the bounded recovery). A failed abort
  releases the stamp it placed so a replacement/retry is never blocked through
  its predecessor.
- recordScheduleOutcome (the owner settlement path) now waits, bounded, for the
  Stop acknowledgement before terminalizing; a resolved/non-stop/stale marker
  proceeds immediately. The paused Stop settles only after its own ack.

Adds service-layer barrier tests and route-level ordering/persistence-failure/
in-progress tests; the data-layer serialization is already covered.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq

* fix(schedules): Wave C.1 — reconcile durable trigger delivery with the reservation

Threads r3803826192 (manual limiter) and r3804255924 (PII/moderation), plus the
independently-found long-Retry-After race. fireSchedule reserves a `started` run
and a global capacity slot BEFORE the durable trigger delivery reaches the chat
route, where an interactive limiter (manual Run Now), PII, or moderation can
reject it before any generation job exists — dead-lettering the delivery while
the run sat `started` until the 30-minute orphan sweep mislabeled it interrupted.
And a valid delivery deferred by Retry-After (up to 24h) could be orphan-settled
and have its capacity released, then fire anyway.

Translate durable delivery state into the schedule outcome:

- Store the deterministic trigger deliveryKey on the ScheduleRun reservation
  (computed from the envelope BEFORE enqueue, so an ambiguous commit still has it).
- Add a getTriggerDelivery engine dep (wired to the merged trigger service's
  getDelivery) that reads the durable delivery by key.
- Schedule reconciliation, for a jobless `started` run: staging/pending/leased →
  admission is live, never orphan; dead → record `error` from the durable
  lastError and release capacity promptly (no 30-minute wait), through the
  ordinary outcome/auto-disable path; succeeded or no record → the existing
  legacy orphan policy (interrupted only past the cutoff); a delivery lookup
  failure defers rather than orphaning a possibly-live delivery. Limiter/PII/
  moderation middleware writes no schedule state.

Adds reconcile state-mapping tests (dead/pending/leased/staging/succeeded/none/
lookup-failure) and a fire test that the reservation's deliveryKey equals the
enqueued delivery's idempotency key.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq

* fix(stream): Wave C.2 — durable retry/ack for terminal host lifecycle actions

Thread r3804518375. Approval expiry won the `requires_action → aborted` CAS and
then invoked the host hook best-effort: `runApprovalExpiredHandler` swallowed a
failure, and because later sweeps enumerate only `requires_action` jobs, the now-
aborted job was never offered again. In the clustered entrypoint (no schedule
reconciler) the ScheduleRun stayed `requires_action` and its retained job
persisted indefinitely.

Make the host lifecycle work durable rather than schedule-specific:

- Add a generic `terminalHostActionPending` marker, set ATOMICALLY in the same
  terminal transition (ApprovalLifecycle.expireWithIdentity), only when a host
  adapter is installed.
- Retain and index such jobs: both stores keep them out of terminal reaping and
  expose getTerminalHostActionJobs(); Redis adds a set + extended (24h-bounded)
  TTL, in-memory a bounded 24h retention so a permanently-failing hook cannot leak.
- The manager clears the marker only after the adapter acknowledges success,
  fenced by generation identity (clearTerminalHostAction), so a replacement
  generation can neither clear nor execute its predecessor's action.
- cleanup()/expireStaleApprovals() enumerates unacknowledged terminal host actions
  across restarts and replicas and retries the idempotent hook; the relay only
  re-invokes while the marker is unacknowledged, so a successful ack prevents
  duplicate work. Store-won expiry marks it too, so a loser-replica relay still
  crosses the hook.
- Terminal SSE notification continues regardless of host-hook outcome.

Covers in-memory behavior (retry after failure, restart/other-replica retry, ack
prevents duplicates, identity fence, terminal notification on failure, no marker
accumulation for non-scheduled jobs) and updates the Redis cluster-membership
contract test for the new index.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq

* style(schedules): satisfy import sorting in fire.ts and fire.spec.ts

CI "Static checks" failed on IMPORT_SORT for the two files Wave C.1 added imports
to (the AgentTriggerEnvelope type import and getAgentTriggerIdempotencyKey).
Applied scripts/sort-imports.mts to exactly those files — imports-only reordering,
no behavior change. Other files reported by a repo-wide check are pre-existing on
dev and deliberately left untouched so this PR is not widened.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq

* fix(schedules): repair the deletion CLI and order restore before the fence release

Addresses three findings from the fresh Codex review.

P1 — config/delete-user.js called methods.disableUserSchedulesForDeletion, which
Wave B removed in favor of suspendUserSchedulesForDeletion. The file is
@ts-nocheck and its spec mocked the removed name, so neither typecheck nor tests
caught it; the real CLI would throw a TypeError before deleting anything and then
only unwind the fence. The CLI now uses the tokenized protocol: it mints a
suspension token, suspends with it, and restores that exact attempt's rows in its
finally block when the deletion does not commit. Its spec mocks the real methods,
so the breakage can no longer hide.

P2 — both the HTTP controller and the CLI released the user-deletion fence BEFORE
restoring schedules. That fence is what refuses new schedule writes/claims, so the
gap let an owner PATCH edit a still-suspended row and have its enabled/next-run
state overwritten by the older snapshot, and let a second deletion attempt
re-suspend under a new token — making the first restore a no-op and stranding the
disabled snapshot permanently. Restore now runs first, while writes are still
fenced.

Hardening for the same defect class across a crash: suspendUserSchedulesForDeletion
now ADOPTS an existing suspension's snapshot when re-suspending a row abandoned by
an earlier attempt, instead of re-capturing the row's current (already-suspended)
state. Without this, an attempt that died before restoring would have its
successor snapshot "disabled, no next run" and permanently strand the schedule.

Tests: CLI restore-before-fence ordering and no-restore-on-success; the same
ordering assertion on both controller post-quiesce failure paths; a data-method
regression that a second attempt adopts the abandoned snapshot and restores the
original enabled/next-run state.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq

* fix(schedules): converge dead deliveries in topology-safe maintenance

Codex finding: the `dead` delivery mapping added in Wave C.1 lives only inside
startScheduleEngine's reconciler, but the clustered entrypoint arms no engine — it
runs erasure-only maintenance. A delivery queued before a restart into clustered
mode and then rejected before generation creation (interactive limiter, PII,
moderation) dead-letters while its ScheduleRun stays `started`, holding a global
capacity slot indefinitely for an ordinary non-deleting schedule.

Add a dead-delivery convergence pass to the erasure sweep, so every topology that
runs schedule maintenance settles it. The pass is POSITIVE-EVIDENCE-ONLY and is
therefore safe where absence-based reconciliation is not: a `dead` delivery is
durable shared state proving no generation owns the reservation. It settles only
when the job is confirmed absent or identity-mismatched (an identity-matched job
still owns the run), defers on an unknown job lookup, on an in-flight abort, and
on an in-flight resume hand-off, ignores legacy reservations with no deliveryKey,
and applies a short grace so an accepted delivery still creating its generation is
never settled mid-handoff. Auto-disable policy is deliberately left to the armed
engine; this path records the failure and frees the slot.

Deliberately does NOT touch api/server/experimental.js — the clustered entrypoint
already starts this sweep, so the convergence arrives through the existing
initializer and the shared-file footprint stays as-is.

Tests: settles a dead delivery as error under an explicitly UNSAFE topology,
leaves live deliveries alone, never settles under an identity-matched running
generation (delivery is not even consulted), defers an in-flight abort, and
ignores a reservation with no deliveryKey.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq

* fix(stream): refresh host-action retention on each retry attempt

Codex finding: unacknowledged terminal host-action evidence was capped at the
24h pause TTL, so a host dependency (Mongo) unreachable for longer than that let
the Redis key — and its pending marker — expire with no generation-fenced
acknowledgement, stranding the ScheduleRun where no reconciler is armed.

Measure retention from the LAST retry rather than from the terminal transition:
enumerating a pending host action IS the retry attempt, so both stores refresh
its retention as they hand it to the hook (Redis re-EXPIREs the job key; the
in-memory store stamps terminalHostActionRefreshedAt and bounds from it). Evidence
therefore survives as long as some replica is still actively retrying, while a
deployment that stops sweeping entirely still lets it age out — so this does not
reintroduce the unbounded leak the cap existed to prevent.

Test: after a failed hook, a later cleanup pass keeps the marker pending and moves
its retention basis forward.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq

* fix(schedules): defer settlement when the Stop barrier times out

Codex finding: waitForStopPersistence returned after its 5s poll budget even when
the Stop was still fresh and unacknowledged, and recordScheduleOutcome then went
straight on to terminalize the run — releasing its capacity, deletion, and erasure
barriers while beforePublish may still have been writing. Slow checkpoint cleanup
is indistinguishable from a dead route on that signal, so the timeout was being
treated as if the barrier had been satisfied.

The poll budget now means "undecided", not "clear". On timeout with a fresh,
unacknowledged Stop the barrier DEFERS: recordScheduleOutcome returns false
without recording, leaving the run active/preserved. Settlement then happens
either when the route acknowledges, or once the existing stale-owner cutoff
(ABORT_OWNER_PRESUMED_ALIVE_MS) authorizes a later attempt — which the loop
already treats as clear-to-settle. Callers with durable retry (the approval-expiry
host action, reconciliation) re-drive it, so a deferral converges rather than
stranding the run.

Test: a fresh Stop that never acknowledges within the budget reports not-settled
and records no outcome.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq

* fix(schedules): require definite delivery failure, extract its message, converge deferred Stops

Third Codex round. Three findings in code from this closeout, plus one pre-existing
P1 that is a one-line operator-safety fix.

P1 — dead-delivery settlement demanded too little evidence. `dead` does not prove a
request was rejected: the trigger host marks response timeouts and invalid success
responses `certainty: 'ambiguous'`, and the engine dead-letters those once retries
are exhausted. The erasure sweep treated every dead letter as positive evidence, so
an ambiguous one sitting over a generation a peer had accepted could terminalize the
run and release its capacity mid-flight. It now settles only on a DEFINITE rejection,
unless job absence is deployment-authoritative (safe topology), where the
confirmed-absent job is itself the evidence.

P1 — `lastError` is an `AgentTriggerDeliveryFailure` object, not a string. A
duplicated local interface declared it `string` (against CLAUDE.md's no-duplicate-
types rule), so both the sweep and the engine reconciler passed the object into the
String-typed run/schedule `error` fields; Mongoose would reject the cast, the per-row
catch would swallow it, and the run would keep its global capacity slot. The dep type
now reuses the canonical `AgentTriggerDeliveryFailure` and both call sites pass
`.message`. Re-typing immediately surfaced a stale test that had asserted a string.

P1 (pre-existing) — the base-config global stop is honored in `getLimits` via
`isRuntimeDisabled` rather than a literal `=== false`. The stop has two shapes, and
deepMerge turns base `{ use: false }` plus a principal override of `true` into
`{ use: true }`, so the literal check reported the feature enabled and Run Now
dispatched straight through fireSchedule, bypassing the operator's emergency stop.
Now the same predicate the engine gate already uses.

P2 — a Stop whose settlement DEFERRED past the poll budget had no convergence path
where no reconciler is armed. `acknowledgeScheduledStopPersistence` now optionally
re-drives the terminal outcome once the barrier clears; `recordRunOutcome` is
match-guarded and idempotent, so an owner that already settled makes it a no-op. The
abort route passes it for a running generation; a paused job still settles explicitly.

Tests: ambiguous dead letters refused under unsafe topology but settled when absence
is authoritative, definite rejections settled either way, the failure message carried
through, and the abort route's re-drive present for running / absent for paused.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq

* fix(schedules): converge terminal runs in clustered workers, retry suspension restore

Fourth Codex round on 74f7af8. Both findings are in code from this closeout.

P2 - a clustered worker never settled a run whose generation finished but whose
outcome write failed. `recordScheduleOutcome` retries three times and then returns
false; the owner honors that by PRESERVING the terminal job as the only surviving
evidence, and the armed engine's reconciler replays exactly that. The clustered
entrypoint arms no engine, and its sweep only covered deleting schedules
(settleAbandonedRuns) and dead deliveries - an identity-matched job was skipped
outright. So an ordinary live schedule's run stayed `started`, held its GLOBAL
capacity slot, and kept a preserved job that carries no `completedAt` and is
therefore invisible to the store's finished-job sweep: both leaked until store
expiry.

The live-schedule pass now also converges from a retained terminal job, mirroring
the reconciler branch it stands in for: honor the owner's stamped outcome over the
generic status (so a balance refusal still walks its streak rather than resetting
it), clear the reserved conversationId when the generation never emitted its
created event, and delete the retained job only AFTER the outcome write is durable.

This stays positive-evidence-only and safe in every topology. Presence, not
absence, is the evidence: an identity-matched job is authoritative wherever it is
observed - a shared store shows the real generation, a process-local store can
only be showing this process's own - which is why it needs no
canInferOwnerDeathFromMissingJob fence, unlike the absence-based paths. The
in-flight abort/resume fences still defer, so an `aborted` job cannot settle a run
whose owner may still be persisting. Both cases now share one pass over one window
rather than two, and `retainedOutcome` moved to types.ts so the sweep reuses the
engine's mapping instead of duplicating it (and stays independent of the engine).

P2 - a failed restore stranded a live account's schedules. Cancelling an account
deletion restores the suspended rows while the deletion fence is still armed and
then releases that fence; nothing re-drives the restore afterwards, so one
transient write failure left the user with silently disabled, next-run-less
schedules. The restore is now retried at the single choke point both the HTTP
controller and the CLI share. Retrying is safe because each attempt re-reads only
the rows STILL carrying the token: a partially-applied unordered write converges
on exactly the stragglers, and a fully-applied one finds nothing.

The fence is still released when every attempt fails, deliberately: retaining it
would refuse the live account's schedule writes AND make beginAgentTriggerUserDeletion
report `in_progress` forever, blocking the retry that is the convergence path (a
later attempt adopts this snapshot, so its cancel restores these exact rows). Both
callers now log the user id and suspension token so the state stays recoverable by
hand if that never happens.

Tests: retained terminal job settled and its evidence released, stamped outcome
preferred over the generic status, stamped failure reason carried, reserved
conversationId cleared for a never-created conversation, identity-mismatched
terminal job ignored, abort-in-flight deferred; restore retried past a transient
write failure and converging on a partially applied restore.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq

* fix(schedules): converge unprojected pauses and release replayed bookkeeping jobs

Two of the four findings I flagged as pre-existing to my closeout commits. Both are
in code this PR introduces, so merging would have shipped them; both are the same
capacity/evidence-leak class the last four rounds have been closing. The remaining
two (slotless rows escaping the per-user cap, and the non-rotating `started`
reconciliation bucket) are genuinely latent and deliberately left alone.

An UNPROJECTED PAUSE held a global capacity slot forever. The pause projection is
what moves a run row off `started`; `recordScheduleOutcome` already retries it, but
the request controller discarded the result, so three failed attempts were dropped
silently. The armed engine's reconciler replays that state, but the clustered sweep
did not: a paused job is not terminal, so the retained-job path returned without
settling, and the dead-delivery path never inspects an identity-matched job at all.
The row stayed `started` with no cutoff that would ever clear it.

The call site now surfaces the failure, and the sweep converges it, mirroring the
reconciler's pause branch: project `requires_action` (which frees the slot) but do
NOT release the job's evidence — unlike a terminal job it is still live, awaiting an
approval. The resume hand-off fence still defers, so re-projecting cannot release a
slot a continuation just claimed.

The BOOKKEEPING REPLAY pass leaked its retained job. A run reaches that pass only
because its owner crashed before bookkeeping — which is also before it could release
the job it retained for exactly this recovery. The active-run pass clears its own;
this one never did, and a preserved job is deliberately kept WITHOUT `completedAt`
so the store's finished-job sweep cannot reap it early, so nothing else ever would.
It now clears after `finalizeBookkeeping` succeeds — identity-guarded, a no-op when
no job is retained, and skipped entirely when the replay itself failed, since the
retained job is the only surviving evidence in that case.

Tests: pause projected and the slot freed while the live job's evidence is kept,
paused job deferred during a resume hand-off, retained job released once replayed
bookkeeping is durable, and retained job kept when the replay fails.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq

* fix(schedules): fence the clustered pause replay against a concurrent resume claim

Codex round 5, on code I added in f735341. The finding is correct and the exposure
is one I introduced.

The pause replay lives in a sweep that runs in EVERY clustered replica, so several
sweepers can observe the same unprojected pause. Its guards were all derived from an
in-memory row SNAPSHOT: hasResumeHandoffInFlight reads the snapshot's
resumeClaimedAt, and recordRunOutcome's pause branch matched any row currently in
`started`/`requires_action` with no fence of its own. The race: sweeper A projects
the pause and frees the slot; the owner's approval then claims a fresh one
(markRunResumeClaimed takes the row to `started` WITH resumeClaimedAt in one write);
sweeper B, still holding the pre-projection snapshot, passes its hand-off check and
replays — `$unset: { capacitySlot, resumeClaimedAt }` under a continuation that is
already running. The run reverts to `requires_action` while its generation proceeds
outside global capacity.

The engine's reconciler makes the same call and has the same snapshot-derived guard,
but v1 arms exactly one engine, so it has no concurrent racer; the sweep is the first
thing to run this transition in parallel. Left the engine alone rather than widening
the change: its `requires_action` re-affirmation is deliberate and single-writer.

Fixed where the race is, in the write itself. `recordRunOutcome` takes an optional
`requireNoResumeClaim`, which adds `resumeClaimedAt: { $exists: false }` to the pause
filter, and the sweep sets it. Because the stamp is written in the SAME update that
moves the row to `started`, its absence is atomic proof no resume owns the row. The
flag is deliberately NOT set by the generation owner: its own re-pause legitimately
clears the stamp as the hand-off's completion signal.

The fence cannot block the recovery it exists to enable: markRunResumeClaimed only
matches `requires_action`, so a genuinely stuck `started` row can carry no resume
claim — it is in fact blocking its own approval until this replay frees it.

Tests: a replay from a stale snapshot leaves a resume-claimed row's status, slot, and
claim stamp intact, while a stuck row with no claim still recovers; and the sweep is
asserted to send the fence.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq

* fix(schedules): fence stale resume claims by age, not existence

Codex round 6, on the fence I added in ee00c60. Correct again, and the defect is
the mirror image of the one it fixed.

The fence was an EXISTENCE check (`resumeClaimedAt: { $exists: false }`) while its
caller's guard is a FRESHNESS check (hasResumeHandoffInFlight, bounded by
RESUME_HANDOFF_STALE_MS). They agree while a claim is fresh and disagree once it is
abandoned: a worker that dies after markRunResumeClaimed takes the row to `started`
and stamps resumeClaimedAt — but before the continuation resumes or
releaseRunResumeClaim rolls it back — leaves the stamp set forever. Past the bound
the sweep correctly stops deferring and tries to recover the row, but the write
rejected it purely because the field still existed. The row stayed `started` holding
its global capacity slot, and its approval was unresumable for good, since
markRunResumeClaimed only matches `requires_action`. That is exactly the stuck state
this replay exists to clear, so the fence had reintroduced it for the crashed-resume
case.

`requireNoResumeClaim: boolean` becomes `resumeClaimStaleBefore: Date`, and the
filter matches a row with no claim OR a claim older than that cutoff. The sweep
passes the SAME bound its in-flight check uses, so the two can no longer disagree.
A genuinely racing claim is by construction fresh — it is created after the sweeper's
snapshot — so the race from round 5 stays closed.

Tests: a row whose resume claim was abandoned by a dead worker now recovers (status,
slot and stamp all cleared), alongside the existing two — a fresh claim still repels
a stale-snapshot replay, and an unclaimed stuck row still recovers.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq

---------

Co-authored-by: Claude <noreply@anthropic.com>
2026-08-20 11:51:30 -04:00
Danny Avila
259f1e0c32
🛰️ feat: Route Live Subagent Controls Across Replicas (#14971)
* feat: route live subagent controls across replicas

* fix: initialize task routing in cluster workers

* fix: harden cross-replica task routing

* fix: expire routed task owners independently

* fix: close cross-replica routing edge cases

* fix: bound owner refresh and close routed cancellation gaps

Refresh owned task registrations in bounded parallel batches so a full
heartbeat pass stays well inside the 30-second directory lease instead of
serializing one Redis EVAL per registration.

Route conversation-deletion cancellation through a dedicated owner-side
scope operation. The owner applies the deletion predicate to its complete
local task set, so a scope holding more children than the model-facing
list cap no longer leaves live executors running after their parent is
removed.

Key a consumed claim's retained response by its operation rather than by
one caller's correlation id, so a later poll recovers a terminal result
whose responses were all lost. Live claim statuses stay uncached so a
poll always observes the task's current state.

Type the model-facing `maxLength` bounds with a narrow local string
schema; the SDK's JsonSchemaType does not declare the keyword, and the
runtime checks continue to enforce the same limits.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H1cCMDrTWaRNkmtKjpWELZ

* fix: retain claimed results apart from control replays

A consumed claim is the only routed response whose loss destroys data, so
it no longer shares one bounded cache with control replays that unrelated
command traffic can evict. Claims are retained under their own budget, and
the requester acknowledges a result it received so the owner releases the
copy immediately instead of holding it for the full replay window.

Resolve the post-delete cancellation pass from durable leases. The deleted
conversations cannot be read back, so re-reading each one only scaled the
cascade while probing the owner directory once per removed id; one lease
read now resolves every live child address instead.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H1cCMDrTWaRNkmtKjpWELZ

* fix: never consume a result the owner cannot replay

Retention for consumed claims is bounded, so a burst of undelivered results
could evict an earlier one and lose it for good. The owner now admits a claim
only while it can retain a worst-case result, and refuses the routed claim
otherwise instead of consuming it, leaving the result on the task for a later
poll. Retained claims are never displaced; control replays keep evicting.

Key a control replay by the command itself rather than by one caller's
correlation id. The transport's own retry reuses a single envelope, but a
caller that saw the owner as unavailable reissues the command under a new id,
which steered, queued, or interrupted the child a second time.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H1cCMDrTWaRNkmtKjpWELZ

* fix: own a claimed result until it is acknowledged

A consumed terminal result is task-owned state, not a cache entry. It now
carries no expiry at all: the owner holds it until a caller acknowledges
receipt, and only then is it released. Retention stays bounded by the
existing admission gate, which refuses a claim the owner could not keep
rather than consuming a result it might drop.

Identify a control by the caller's invocation instead of by its content.
The tool mints one id per invocation and routing carries it, so a routed
retransmission of that invocation replays the owner's result while two
deliberate identical commands arrive under distinct ids and both apply.
Content-derived identity could not tell those apart and would have
answered the second from a stale snapshot.

Wait for the dpkg frontend lock in the best-effort Playwright font step.
Its timeout kills npx while the apt-get it spawned keeps the lock, which
then failed the fatal Redis install and ended the MCP replica jobs before
any test ran (#14983).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H1cCMDrTWaRNkmtKjpWELZ

* fix: treat acknowledgement as part of delivering a result

Publishing an acknowledgement once and ignoring the outcome meant a result
could be reported as delivered while the owner never learned it could let
go, and since that retention neither expires nor evicts, enough lost
acknowledgements would fill it and refuse every later remote claim.

An acknowledgement is now confirmed: publishing to zero subscribers is not
success, it retries inside the ordinary request window, and a claim whose
acknowledgement cannot be confirmed reports the retryable unavailable path
instead of handing back a result the owner still holds. A later poll
recovers that result and acknowledges it, and releasing is idempotent.
Owner registration also outlives the task while a result is unacknowledged,
so the retained result cannot become unreachable.

Take the control invocation identity from the provider's tool-call id
rather than minting one per execution, so replaying the same tool call
stays idempotent while two distinct calls with identical payloads both
apply.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H1cCMDrTWaRNkmtKjpWELZ

* style: sort the widened node:crypto import

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H1cCMDrTWaRNkmtKjpWELZ

* fix: own control invocations and cancellation plans at the task seam

Applies one logical control exactly once for its owning task rather than
in the transport, so a local caller and a routed caller of the same
invocation agree, and reusing an invocation id for different content is
refused instead of silently applied. Invocation identity now comes from
the run, agent, and provider tool-call id hashed to a bounded 32
characters, so a repeated `call_0` never bleeds across tasks and no id can
overrun the routed bound.

Cancellation for conversation deletion is now resolved into a plan while
those rows are still readable, then replayed against the owner directory
after the cascade is deleted. Owner registration is awaited before any
provider work, so a child that cannot be addressed never starts.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H1cCMDrTWaRNkmtKjpWELZ

* style: separate the control invocation map from the next member

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H1cCMDrTWaRNkmtKjpWELZ

* fix: close subagent deletion, claim, and control invocation gaps

Bulk conversation deletion now runs behind a durable owner admission fence.
Draining alone could not close the race: a child admitted on another replica
after the drain read its leases would start provider work against a parent
about to disappear. The fence is written before any lease is read and each
child revalidates it after its own lease is written, so one of the two always
observes the other. It expires on its own, so a process lost mid-deletion
cannot leave an account unable to run subagents.

A terminal child result is no longer kept alive in the owning replica's
memory until someone acknowledges it. Collection is recorded durably on the
child's own message against the polling invocation, so the poll whose
response was lost recovers its own result while a different invocation is
told the result was already collected. Owner-side retention returns to an
ordinary bounded cache that expires, which is what abandoned polls needed:
they can no longer occupy claim capacity until the process restarts.

The deletion drain now cancels each task under one invocation held for the
whole drain, stops re-sending once the owner answers, and retries only
deliveries it could not confirm. A routed control replay also validates the
command fingerprint, so one invocation id carrying different content reaches
the owner to be refused instead of collecting the earlier command's success.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H1cCMDrTWaRNkmtKjpWELZ

* test: assert the drain's calls before restoring its spies

Restoring a spy also clears its recorded calls, so the drain assertions
ran against an emptied mock. Formats the durable claim method tests.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H1cCMDrTWaRNkmtKjpWELZ

* fix: close the follow-on gaps in the deletion fence and result claim

The admission fence now carries an ownership token, so an overlapping
deletion's fence is never lifted by the one that finishes first, and both
fence writes invalidate the cached auth user document. It also covers the
other bulk-delete path: `DELETE /` with no conversation filter removes every
conversation, so it runs behind the same fence rather than a bare drain.

The durable record now decides who holds a one-shot result. An owner replaying
a retained response could hand the same terminal claim to a second invocation;
that invocation is told the result was already collected, while the one that
consumed it still recovers its own. A task with no durable record to arbitrate
keeps whatever the owner answered.

Drain cancellation treats `not_found` as unconfirmed: a missing registration
while the durable lease is still live means the child may be running, so the
command is retried under its invocation once the owner republishes itself.
Control fingerprints are hashed, so retaining one per invocation costs a fixed
few bytes instead of a bounded message.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H1cCMDrTWaRNkmtKjpWELZ

* fix: hold every deletion fence and keep live idempotency records

An owner now holds one admission fence per concurrent bulk deletion instead
of one at a time, so admission reopens only when the last deletion finishes
regardless of completion order. Expired fences are pruned as new ones arrive
and the set is bounded, so an abandoned fence cannot accumulate or lock an
account out.

A failed durable claim write is no longer read as an absent record. Handing a
terminal result over without recording its claimant would let another
invocation collect the same one-shot output once the database recovered, so
the collection reports the retryable unavailable path and leaves the result
for a later poll.

Control invocation records now evict tasks the store no longer holds before
live ones, over a bounded scan. Dropping a live task's record would let a
caller retry apply its queue, steer, or interrupt a second time once the
transport replay had also expired.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H1cCMDrTWaRNkmtKjpWELZ

* fix: keep the deletion fence portable and never drop a live record

The admission fence is written with plain update operators again. DocumentDB
rejects pipeline-form updates, and this runs before any deletion, so the
pipeline form would have failed both bulk-delete endpoints outright on a
supported database target.

An excess deletion is now refused rather than silently displacing the oldest
active fence, which would have reopened admission for a deletion still
running. Expired fences are pruned before the cap is tested, so only genuinely
concurrent deletions count against it.

Control invocation records now sweep every settled task's entry when the
window fills, and a window of entirely live records refuses the new control
before touching the child instead of evicting one. Applying a command with no
room to record it would let the caller's own retry apply it twice.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H1cCMDrTWaRNkmtKjpWELZ

* fix: hold the fence, bound recovered results, and expire stale commands

The admission fence is renewed for as long as its deletion runs, so a very
large account or a stalled database cannot let it lapse while conversations
are still being removed. Only the deletion's own fence is renewed, and the
renewal stops with the operation.

Cancellation now covers every conversation the cascade removed, not only the
ones a plan named: a grandchild lives in its own parent's scope, which a plan
naming the deleted root never reaches.

A routed request carries the deadline its caller waits for, and an owner drops
one that arrives past it. A publisher disconnected mid-request queues the
envelope offline and delivers it after the caller was told the owner was
unavailable, which would otherwise steer a child the caller believes untouched.

A result recovered from its durable child message is bounded like a routed one.
The message keeps the child's untruncated output, so recovery could otherwise
return far more than the routed result limit allows.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H1cCMDrTWaRNkmtKjpWELZ

* test: size the fence window so a renewal can be observed

The renewal test set a 30ms drain timeout but the five-minute grace window
dominates it, so the interval was 100 seconds and no renewal could fire
inside the test's deletion. The grace window is an option now, matching the
store's other timings, and the test sizes the window to 90ms.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H1cCMDrTWaRNkmtKjpWELZ

* fix: wire the durable claim method and close the fence follow-ons

The production store never received `claimSubagentTaskResult`, so every
terminal result would have surfaced as unavailable once a task settled. The
host wires that object from JavaScript, where the factory's parameter type
checks nothing, so the factory now refuses a store missing any method it
calls rather than failing at the first claim.

The routing transport takes a dedicated publisher with the offline queue
disabled. The shared client held commands issued during a disconnect and
delivered them after the caller had given up, which the request deadline
narrowed but could not close inside the clock-skew allowance.

Fence renewal invalidates the cached auth document like the fence and release
paths, and a renewal reporting its entry gone re-takes the fence instead of
letting the deletion run on unfenced. The post-delete cancellation retries a
transiently unreachable owner: the conversations are already gone, so it is
the only pass that can still stop a late-admitted child.

A replaced replay entry no longer leaves its bytes counted, which would have
inflated the cache's total until unrelated responses were evicted.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H1cCMDrTWaRNkmtKjpWELZ

* test: wait on observed lease renewal instead of a fixed delay

The shared-lease renewal test held a 60ms lease and slept 100ms before
asserting an overlapping worker was refused, so a loaded runner that
starved the 10ms heartbeat past the TTL let the lease lapse and the
second worker run. Spy on acquisition and renewal, then wait until a
renewal succeeds past the acquired lease's own deadline — direct
evidence the heartbeat carried it past expiry, with no timing
assumption — and give the lease enough headroom that a stalled timer
no longer decides the outcome.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H1cCMDrTWaRNkmtKjpWELZ

* fix(agents): close the routing, fence, and cache gaps found in review

Five separate seams, each with its own failure:

`Cluster.duplicate` reads its first argument as a startup-node list and
its second as the overrides, unlike `Redis.duplicate`, so the publisher's
`enableOfflineQueue: false` was silently dropped under
`USE_REDIS_CLUSTER` and a command issued mid-disconnect could still
reach a child after its caller was told `unavailable`. Route both
through `duplicateIoRedisClient`.

The control window's capacity refusal ran before the store knew whether
it owned the task, so unrelated local load could veto a cancellation
bound for another replica. Establish that the task is local first and
leave a remote one to its owner's window.

`clearInterval` stops only future fence renewals. One already waiting on
the database could resolve after the release, read its own lifted fence
as expiry, and write a replacement that nothing remained to lift —
closing subagent admission for the account until it aged out. Track the
in-flight renewal, refuse overlapping passes, and await it before
releasing.

Every owner bounds its own task list, but the aggregation appended each
batch whole, so the model-facing list grew with the number of replicas
holding the scope. Cap the merged list while still reading every reply
for the stale-registration sweep.

The admission-fence prune commits independently of the fence that
follows it, so a refused or failed push left the cached auth document
describing entries the collection no longer held. Invalidate whichever
way the second write goes.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H1cCMDrTWaRNkmtKjpWELZ

* fix(agents): cap the merged task list the poll tool actually reads

Each owner bounds its own reply and the remote aggregation bounds their
sum, but `listTasks` merged that bounded remote list with however many
children this replica owns and returned it whole. `check_background_task`
could therefore still receive roughly twice the advertised cap. Bound the
deduplicated, sorted result and export the cap so both seams share one
number.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H1cCMDrTWaRNkmtKjpWELZ

* test: admit every task the merged-list cap test starts

The base store admits ten concurrent runs per scope by default, so
starting 150 at once left most refused for capacity and the assertion
never reached the merge it was written to check. Raise the cap for this
store only; admission is a different invariant with its own tests.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H1cCMDrTWaRNkmtKjpWELZ

* fix(agents): let a deletion notice its admission fence lapsing

Renewal failures were logged and swallowed, so a run of rejected writes
let the last confirmed `fencedUntil` pass while the deletion carried on
believing admission was still closed — long enough for another replica
to admit a child against conversations about to be removed. Track the
deadline only a confirmed write advances, and check it after the drain,
before anything is deleted: nothing has been removed at that point, so
the operation fails closed and the caller retries once the fence can be
held. A lapse detected after the rows are gone is logged instead, since
reporting failure there would invite a retry against conversations that
no longer exist.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H1cCMDrTWaRNkmtKjpWELZ

* test: raise both concurrency caps the merged-list test trips

Raising the per-scope limit left the store-wide `maxRunningTotal` at its
default hundred, so fifty of the hundred and fifty starts were still
refused. Verified against the base store directly this time: with only
the per-scope cap raised it admits a hundred, and with both raised it
admits all hundred and fifty.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H1cCMDrTWaRNkmtKjpWELZ

* fix(agents): close the fence renewal gap and keep running tasks listed

A renewal that started before its deadline but landed after it was still
credited with extending the fence from its own start time, so a window in
which admission stood open was papered over: a child could take a lease
the drain had already read past and the deletion would proceed without
cancelling it. The deadline now only advances when the write lands while
the previous one still holds; anything later records a lapse the fence
cannot be restored backwards over.

The model-facing cap sorted oldest-first and sliced, which dropped the
newest tasks — including children that had only just started running,
and which the poll tool offers no other way to discover. Bound by status
instead: running children first, then the most recent settled results.
Both caps share one helper, and the routed aggregation now bounds after
its loop so the choice is made across every owner's reply rather than by
whichever answered first.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H1cCMDrTWaRNkmtKjpWELZ

* fix(agents): finish the cap and the fence at the seams they still missed

The status-aware cap only reached the requester: an owner's own reply
still sliced positionally, so a replica holding more than the cap dropped
its running children before the requester could bound anything. Both
sides now share `boundedTaskList`.

A fence that lapsed during the deletion itself was only logged. The rows
are gone by then, so failing is still wrong, but the child another
replica admitted while the fence was down is not: the fence is retaken
and the drain repeated to cancel it.

A child's lease renewal had the same retroactive hole the admission fence
had — Mongo filters on the `now` captured before the call, so a write
landing after the lease expired still moves the row forward, while an
owner drain reading active leases in that gap saw the thread as free. The
lease now carries its own deadline and a late renewal stops the executor.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H1cCMDrTWaRNkmtKjpWELZ

* test: cover the lease lapse and the post-deletion re-drain

The owner-side cap shipped with a regression test; these two did not.
One drives a lease renewal that succeeds only after the lease it was
extending had expired and asserts the executor stops; the other lets the
fence lapse during the deletion itself and asserts a second drain runs
while the request still reports success.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H1cCMDrTWaRNkmtKjpWELZ

* fix(agents): close live-task lifecycle gaps

* test(redis): exercise cluster node discovery

* fix(test): type cluster discovery seam

* fix(ci): wait for orphaned apt processes

* fix(ci): reserve time for apt drain

* fix(ci): skip optional fonts in MCP jobs

* fix(agents): recover tasks after owner loss

* fix(agents): preserve local task discovery

* fix(agents): initialize fail-fast cluster publisher

* style(agents): sort routing test imports

---------

Co-authored-by: Claude <noreply@anthropic.com>
2026-08-19 02:16:35 -04:00
Danny Avila
e4d6bb71f9
📁 feat: Surface Stateful Workspace Downloads (#14984)
* feat: surface stateful workspace downloads

* fix: sort workspace change imports

* fix: reuse workspace button primitives

* fix: hide collapsed workspace actions
2026-08-18 22:05:13 -04:00
Ravi Kumar L
006e421cd2
💡 feat: add DB-backed admin insights (#14898)
* feat: add Mongo-backed admin insights

* feat: gate insights with environment variable

* fix: tighten insights access and activity metrics

* fix: preserve insights date selections

* perf: parallelize insights search aggregation

* test: wait for MCP conflict recovery

* test: satisfy strict MCP recovery typing

* fix: disable insights pagination while loading

* fix: localize insights range shortcuts

* fix: bound insights search input
2026-08-18 07:51:51 -04:00
Danny Avila
547bd8c4bf
🧵 feat: Persist View-Only Subagent Threads (#14957) 2026-08-18 07:41:42 -04:00
Danny Avila
a47ba7168f
🪢 feat: Custom Request Headers For Langfuse (#14945)
* ✨ feat: Custom Request Headers For Langfuse

Self-hosted Langfuse behind an authenticating proxy or gateway could not
be reached: every outbound Langfuse request hardcoded `Authorization` and
nothing else. Adds `langfuse.headers`, mirroring `endpoints.custom`
headers, and applies it to all four request surfaces — trace/media export
(via the agents run config), feedback scores, central project-identity
lookup, and admin credential verification.

Values resolve through the same pipeline as endpoint headers, so
`${ENV_VAR}` interpolation and header-safe encoding come along.
`extractEnvVariable` continues to refuse infrastructure secrets, so a
config cannot exfiltrate `MONGO_URI` through a header. A header whose
variable is unset is dropped with a one-time warning rather than sent as
a literal `${...}`, which a gateway would read as a wrong credential
instead of a missing one.

Headers merge beneath LibreChat's own `Authorization` on the REST
surfaces, matching `mergeHeaders`, so a custom header can never displace
the Langfuse credential.

These are deployment-level and documented as such: trace export batches
spans from every user through a single exporter, so unlike endpoint
headers they cannot carry per-user placeholders.

The central project-id cache key now includes the headers, so the
header-less module warm-up cannot record a proxy rejection against the
entry the request path later reads.

* 🔒 fix: Keep Langfuse Headers Out Of Stored Overrides

The generic admin config API accepts any field path inside an allowed
section, so `langfuse.headers` could be written through it. Unlike
`langfuse.secretKey`, headers are a map rather than one scalar path, so
the config secret registry cannot encrypt them at rest or mask them on
read — an admin-written map would sit in Mongo in plaintext and come
back in plaintext, widening exposure of what are gateway credentials.

Rejects them on both the dotted-patch and object-upsert routes, the same
way process-backed MCP servers are held to librechat.yaml. This is what
makes "deployment-level" true rather than merely documented.

* 🐛 fix: Wire Config Middleware And Header Collisions For Langfuse

Two codex review findings.

P1 — `api/server/routes/admin/langfuse.js` never mounted
`configMiddleware`, so `req.config` was undefined in production and
credential verification silently ran without the deployment's proxy
headers: exactly the deployments this feature targets could not save a
connection. The handler unit tests injected `config` into their mock
requests, so they stayed green. Mounts the middleware after the access
checks (unauthorized callers still short-circuit first) and adds
route-level tests that assert the handler actually receives a resolved
config — the composition root, not the component.

P2 — spreading custom headers under `Authorization` only replaced an
exact-case collision. A configured `authorization` survived alongside
the managed `Authorization` and fetch appends rather than replaces,
sending both credentials in one combined value. All four request sites
now use `mergeHeaders`, which already merges case-insensitively with the
override winning; tests cover the lower- and upper-case variants.

* 🔒 fix: Mask Langfuse Headers On Read And Harden Value Handling

Three codex round-2 findings.

P1 — the write guard blocked storing `langfuse.headers` in Mongo but did
nothing for the read path: `GET /api/admin/config/base` serves the
resolved AppConfig through `redactConfigSecrets`, which only knows
registered scalar secrets, so a yaml-configured gateway credential was
returned in full to any admin with Langfuse read access. Adds a
secret-map registry that masks values while keeping key names, so an
admin can still see which headers a deployment sets. Masking is safe
precisely because these are yaml-only — a masked read cannot be
round-tripped back over the real values. A malformed non-object value at
that path is dropped rather than serialized.

P2 — `mergeHeaders` indexes one spelling per lowercase name, so a config
holding both `authorization` and `AUTHORIZATION` had only one displaced;
the survivor was then appended by `Headers` into a combined value. Case
variants are now collapsed at resolution, before any consumer sees them.

P2 — `resolveHeaders` encodes only values it substitutes a user field
into, and no user is supplied here, so a literal or interpolated
character above U+00FF reached `Headers` unencoded and threw. Final
values now go through `encodeHeaderValue`; Latin-1 still passes verbatim.

* 🔒 fix: Keep Langfuse Header Credentials Out Of Logs And Validate Names

Three codex round-3 findings, plus a documented boundary for the fourth.

P1 — `loadCustomConfig` logs the parsed config at startup (`printConfig`
defaults true), so a literal gateway credential in `langfuse.headers` was
copied into application logs on every boot, undoing the masking the admin
read path had just gained. The printed copy now goes through
`redactConfigSecretMaps`, reusing the same registry. Scoped to map-valued
secrets so scalar-secret log behavior is unchanged; the live config keeps
its real values.

P2 — a nonempty but invalid field name (` X-Token`, `X Proxy Token`)
passed the emptiness check and then threw in the `Headers` constructor,
which would break export, verification, lookup, and feedback for the whole
deployment rather than that one header. Names are trimmed and validated
against the RFC 7230 token grammar, and dropped with a warning otherwise.

P2 — unresolved `${VAR}` detection tested the *resolved* value, so a
credential legitimately containing `${...}` was mistaken for a failed
substitution and dropped. Detection now inspects the configured text and
checks the referenced variables directly, which also drops references to
denylisted infrastructure secrets instead of forwarding them verbatim.

The fourth (fanout gateway forwards only `Authorization`, so a tenant
Langfuse behind its own proxy is not covered) is a real limitation in a
separate component. Documented on the schema field and in the example
config rather than left implied.

* 🔒 fix: Scope Langfuse Headers To Configured Origins

Three codex round-4 findings.

P1 — one header map was attached to every destination a run resolves to.
Under fanout that means a credential meant for an internal gateway was
also sent to the central destination, typically Langfuse Cloud: an
unrelated third-party origin. Headers are now attached only when the
destination's origin is one the deployment explicitly configured (a
self-hosted base URL, the fanout collector, or a tenant destination set
by env). The built-in `*.cloud.langfuse.com` defaults are excluded
precisely because nobody pointed at them. For trace export this also
means attaching after the export branch settles on a `baseUrl` rather
than before, since which destination wins depends on the branch.

P2 — `encodeHeaderValue` only encodes above U+00FF, so a newline, CR, or
NUL passed through and threw in `Headers`, breaking every request rather
than the one header. Values are trimmed (the common trailing-newline
case) then validated against the legal field-value bytes; an embedded
CRLF is a request-splitting attempt and is dropped, not stripped.

P2 — the write guard matched only the exact `headers` property, so
`{ langfuse: { "headers.X-Token": "..." } }` and root-level dotted
variants slipped through into the Mixed overrides document, where the
nested-map redactor never walks them and a later read returns them in
plaintext. All dotted spellings are now rejected.

* 🔒 fix: Bind Langfuse Headers To One Configured Origin

Four codex round-5 findings.

P1 — the round-4 allowlist still authorized every configured origin, so a
deployment with both a collector and an explicit central host sent the
same credential to both. `langfuse.headers` is one map with no way to say
which endpoint it authenticates to, so it is only unambiguous when the
deployment configures exactly one Langfuse origin. Iterating on which
origins to guess was the wrong axis; headers are now sent only when there
is a single configured origin and the destination is it, with a warning
when several make the intent unresolvable. That covers the self-hosted
case this feature exists for; multi-destination deployments need
per-destination headers the schema cannot yet express.

P1 — `fetch` defaults to following redirects, and Node strips
`Authorization` across origins but keeps arbitrary headers, so a redirect
off an allowed origin would hand the gateway credential to a host that
passed no check. Requests carrying custom headers now refuse redirects;
requests without them keep the default, so nothing changes for existing
deployments.

P2 — `extractEnvVariable`'s whole-string branch is anchored and greedy, so
`${CLIENT_ID}:${CLIENT_SECRET}` parsed as one variable name and the raw
template was sent as the credential. References are expanded here now, so
only literal values reach that path.

P2 — a valid token name is not necessarily usable: `Transfer-Encoding`
makes `fetch` throw and a fixed `Content-Length` misdescribes the body of
every other request sharing the map. Request-framing names are dropped.

* 🐛 fix: Expand Langfuse Header References Exactly Once

Codex round 6 (P2). After expanding `${VAR}` references myself I still
handed the result to `resolveHeaders`, which runs `extractEnvVariable`
over it again — so a credential containing `${PATH}`, or any other name
that happens to be set, was silently rewritten on export, verification,
lookup, and feedback. The round-3 test only used an *unset* embedded
name, which the second pass leaves alone, so it could not catch this.

Resolution no longer round-trips through `resolveHeaders`. The only part
still wanted from it was stripping `{{...}}` user placeholders, which is
now applied directly; expansion, encoding, and validation were already
local. Adds a test whose embedded variable is set, which fails against
the previous pipeline.

* 🐛 fix: Process Langfuse Header Templates Before Substitution

Codex round 7 (P2), the mirror of round 6. Having stopped re-expanding
the resolved credential, the placeholder strip was still running over it:
a token containing `{{LIBRECHAT_USER_ID}}` had that span deleted and
`abc{{...}}ghi` went out as `abcghi`.

Establishes the invariant the last two rounds were circling. Every
template operation — placeholder strip, unresolved-reference check,
expansion — now runs on the operator's configured text, and the
credential is substituted last and never touched again. Gateway
credentials are arbitrary strings, so none of their bytes are syntax.

Also moves the unresolved-reference check after the strip, so it no
longer reports a variable inside a `{{...}}` span that the strip removes.
2026-08-17 20:39:23 -04:00
Danny Avila
7ee9e4e363
🚏 fix: Preserve Endpoint Routing on HITL Resume Replay (#14948) 2026-08-17 18:02:16 -04:00
Danny Avila
aa35cd42b1
📬 feat: Add Durable Agent Trigger Delivery (#14925)
* feat: wire trusted agent trigger dispatch

* feat: add durable agent trigger delivery

* fix: annotate trigger envelope byte limit

* test: isolate trigger startup in server specs

* fix: fence trigger delivery during account deletion

* test: isolate trigger service in user controller specs

* fix: close trigger deletion admission race

* fix: harden account deletion fences

* fix: close durable trigger review gaps

* fix: require offline stale-fence recovery

* fix: type trigger lane sequence ids

* fix: fence admin user deletion triggers

* fix: make trigger deletion recovery durable

* fix: harden offline user deletion

* fix: serialize trigger lane publication

* style: sort trigger delivery imports

* fix: recover orphaned trigger publications

* fix: preserve trigger recovery ordering

* fix: fence trigger publication during purge

* fix: defer remote trigger deletion fences

* fix: close durable delivery cleanup races

* fix: drain CLI generation owners before deletion
2026-08-17 09:25:08 -04:00
Danny Avila
f8f118ef29
🛰️ feat: Execute Generic Agent Trigger Deliveries (#14921)
* feat: add generic agent trigger dispatch seam

* refactor: harden trigger dispatch contract

* fix: annotate envelope depth alias

* style: sort trigger envelope imports

* fix: reject unknown trigger dispatch modes

* fix: reject unknown trigger envelope versions

* refactor: validate complete trigger envelopes

* feat: add agent trigger execution host

* fix: enforce trigger delivery contracts

* fix: harden trigger admission path

* fix: finish trigger cancellation handling

* fix: retry strict steer rollout gaps

* fix: retry paused trigger steers

* fix: parallelize trigger admission setup
2026-08-17 08:45:49 -04:00
Danny Avila
57ea1137f6
🛡️ feat: Let Admins Restrict Stateful Workspace Scopes (#14910)
* feat: let admins restrict stateful workspace scopes

* fix: enforce stateful scope policy across agent paths

* fix: close stateful scope policy activation gaps
2026-08-17 01:29:19 -04:00
Danny Avila
485abef3fa
🥚 refactor: Default Agents to Preferred Stateful Workspace Scope (#14908)
* feat: add user default for stateful agent workspaces

* style: sort stateful workspace imports
2026-08-16 22:31:01 -04:00
Marco Beretta
fdc9c77f6e
🗄️ feat: Archive All Chats From Data Controls (#14885)
* feat: archive all chats from data controls

Adds an "Archive all chats" row under Data controls > Your data, next to
Shared links, with a confirmation dialog. It calls a new
POST /api/convos/archive/all endpoint backed by archiveAllConvos, which
archives every conversation currently visible to the user in a single
updateMany and refreshes the stats of every chat project the archived
conversations belonged to.

Temporary and retention-expired conversations are skipped: they are
already hidden from the chat list, so archiving them would only surface
them in the archived view. The update runs with timestamps disabled so
each conversation keeps its own updatedAt and the archived list stays
sorted by real activity.

Archiving a conversation now also drops the new-chat message cache alias
for it. A chat's first turn writes the same message array under both the
conversation key and the new-chat key, so without this the messages of a
just-archived chat kept rendering on the new chat screen until a reload.
Deleting already handled this; archiving did not.

* fix: keep archive-all state consistent

* fix: drop stale detail caches after bulk archive

* fix: harden archive-all request handling

* fix: reconcile archive batch failures

* Fix project stats refresh races and archive route boundary

* Fix archive-all review findings

* Fix archive scan index and partial-batch stats refresh

Reconcile project stats for already-committed archive batches when a later
batch fails, and index the archive scan as { user, _id } so non-tenant
pagination can use _id order.

* Fix Recoil reset after a partial archive-all failure

Refetch the submitted conversation on error and start a new chat only
when that conversation is still active and already archived.

* Fix archive recovery from resetting a newly opened chat

Re-read the active Recoil conversation after the archive-state lookup
resolves, so a slow getConversationById cannot start a new chat if the
user already opened another conversation.

* Fix project-stat reconciliation after archive races

Keep retrying optimistic project-stat writes instead of returning a
stale document after three lost CAS attempts, and retry destination
project discovery after a transient distinct failure.

* Fix archive reset and project-count increment races

Leave already-archived chats open after archive-all, recount new
project conversations instead of incrementing, and skip a delayed
increment when a concurrent refresh already recorded that chat.

* Recover destination projects after discovery retries exhaust

Keep committed conversation IDs when post-archive distinct fails, then
rediscover those projects in finally so a moved conversation's
destination still gets reconciled after the error is rethrown.

* Fix archive-all recovery batching and remount pending state

Recover destination projects in 500-id chunks so the final lookup
cannot exceed Mongo's command size, and share archive-all pending
state through a mutation key so Settings remounts stay disabled.

* Stamp bulk-archived chats and refresh the pinned cache

Bulk archive wrote only isArchived, so the archived table dated every
swept chat by createdAt and the default archivedAt sort dropped the whole
run into the legacy null group. Stamp one timestamp for the sweep; the
filter only matches unarchived chats, so an existing stamp cannot move,
and timestamps: false still preserves each updatedAt.

The pinned section fetches on its own key with a five-minute stale time,
so an archived pin kept rendering in the sidebar until that expired.
Invalidate it alongside the other lists on both success and failure.

Also drop the async from the failing-batch updateMany mock: its
Promise<never> is not assignable to the Query return type, while a plain
synchronous throw types as never.

* Bound archive recovery state with the sweep marker

Recovery held every committed conversation id for the life of the
request so the finally block could re-run project discovery after an
in-loop distinct gave up. Slicing that array into 500-id queries capped
the BSON command size but not the heap, so a very large history could
exhaust a worker mid-archive.

The archivedAt stamp already identifies exactly what this call
committed, so recovery is now one distinct scoped to it. That filter is
a prefix of the existing user/isArchived/archivedAt index, and the two
discovery call sites collapse into one filter-taking helper.

* Reconcile archive stats when a write outcome is unknown

A batch that commits but whose result never returns, a stepdown or a
connection drop between commit and acknowledgement, left archivedCount
at zero, so the finally block skipped both marker recovery and the stats
refresh. The chats were archived, so no retry could find them again: the
sweep filter no longer matches them and their projects kept stale
counts.

Both now key off the write attempt rather than the returned count.
Nothing else needs to change, because the marker is stamped by the same
write whose result went missing.

* Retry dropped project refreshes and guard stale pointer writes

Two ways a project could keep stale stats after archive-all.

A refresh that rejected was logged and dropped for good. Its chats are
archived, so no retry of archive-all can find them again to recompute
against, and the likeliest rejection is the recoverable one:
refreshChatProjectStatsForUser gives up when the project changed under
every compare-and-set attempt. Failures are now collected and replayed
once the rest of the run has stopped competing with them.

A save already in flight could also undo the sweep. Its conversation
document still said visible, so its tail took the pointer branch and
wrote lastConversationId back to a chat the sweep had just archived,
leaving the project advertising activity on a chat the workspace hides.
The pointer write now confirms the chat is still visible first, and
recomputes the project when it is not.

* Verify project pointers after the write, not before

Checking visibility before the pointer write only moved the race earlier:
a sweep landing between the check and the update still archived the chat
and cleared the project, and the write then restored it as
lastConversationId.

The check now runs after the write and repairs instead of preventing. A
sweep that lands earlier is caught here; one that lands later refreshes
the project itself, and refreshChatProjectStatsForUser compare-and-sets,
so it cannot commit a count it read before this write. Same single
indexed read as the check it replaces.
2026-08-16 22:11:57 -04:00
Danny Avila
1b7e2a4e6a
⚡ perf: Optimize First Load of Large Conversations (#14901)
* ⚡ perf: Index the Conversation Fetch and Trim the Client Message Projection

* ⚡ perf: Memoize the Message Tree per Cache Write

* ⚡ perf: Serve Message Reads via the Trimmed Projection and an Ownership Probe

* ⚡ perf: Defer Collapsed Disclosure Bodies Until First Expansion

* ⚡ perf: Progressively Mount Long Threads from the Scroll Anchor

* 🩹 fix: Address Codex Findings on Retention, Anchoring, and Cache Bounds

* 🩹 fix: Poll the Oversized Export Precondition Through the Progressive Mount

* 🩹 fix: Keep Video Results in the Client Message Projection
2026-08-16 19:45:21 -04:00
Danny Avila
7d850c308a
🧠 feat: Add Live Reasoning Labels (#14893)
* feat: add live reasoning labels

* fix: Stabilize reasoning label checks

* fix: Address reasoning label review findings

* chore: Bump Agents SDK for reasoning labels

* fix: Reset reused reasoning step evidence

* fix: Reconcile cleared reasoning labels

* fix: Fence reasoning label resets

* fix: Reset reasoning ownership before gap labels

* fix: Preserve THINK type through label reset

* test: Expect run-global reasoning revision
2026-08-16 18:11:55 -04:00
Marco Beretta
832bac39ad
🗄️ feat: Record When a Conversation Was Archived (#14863)
* feat: record when a conversation was archived

The archived chats dialog has a "Date Archived" column that was bound to
createdAt, so it showed when the chat was created rather than when it was
filed away. Nothing recorded the latter.

Conversations now carry archivedAt, set on archive and cleared on
unarchive, and the column reads it. Chats archived before the field
existed have no stamp and fall back to createdAt, which is exactly what
that column already showed for them.

The archive view sorts on the new field. archivedAt is absent on every
previously archived chat, so the missing-value group is the common case
here rather than an edge case: the cursor's null handling, written for
titles, now covers both, and an absent stamp survives the cursor as null
instead of collapsing to the epoch and replaying the whole archive.

* fix: address review findings on the archived-at stamp

- Protect `archivedAt` from saveMessageToDatabase's unset sweep. Any
  persisted field missing from endpointOptions is unset, so sending a
  message in an archived chat cleared the stamp while leaving isArchived
  true, silently dropping it into the legacy fallback group.
- Order the legacy group by the createdAt the dialog displays rather than
  by last activity. The cursor's secondary key is now chosen per sort
  field, so the fallback the cell renders and the order the server
  returns cannot disagree.
- Put that secondary key in the archive index too, so paging the legacy
  group does not fall back to a blocking sort.

* fix: keep archivedAt on a redundant archive request

Opening an archived chat and hitting the archive shortcut, or retrying
the POST, sent isArchived: true again and replaced Date Archived with
now. saveConvo now stamps only on the unarchived-to-archived transition
and still clears the field on unarchive.

* fix: make archive timestamp updates atomic

* test: type the archive race spy against the driver signature

* fix: archive without an aggregation-pipeline update

DocumentDB documents no support for pipeline-form updates on any engine version,
and the repository's compatibility assessment records that a prior P0 rewrote the
three that existed. Stamping archivedAt through a $cond pipeline reintroduced one,
which would have sent every archive and unarchive to the route's 500 handler on a
supported 5.0 deployment.

The conditional stamp is now a compare-and-set on isArchived, which keeps the
transition atomic without a pipeline: only the write that finds the chat unarchived
stamps it, so a duplicate or retried archive leaves the original date alone and an
unarchive that lands first is re-stamped. Schema defaults and createdAt-on-insert
go back to mongoose's own setDefaultsOnInsert and $setOnInsert, and tenantId is
once again stripped by the tenant-isolation plugin rather than by hand.

* fix: do not report a racing archive as a missing chat

Both conditional writes of the compare-and-set miss when the archive flag flips
between them: the chat was already archived when the transition write ran and
unarchived again before the already-archived write. saveConvo returned null for
a conversation that plainly exists, so POST /api/convos/archive answered 404.

Confirm the conversation is really gone before accepting that result, and retry
the pair when it is not. An unknown id still costs one existence read and falls
straight through to the 404.

* fix: resolve a fully contended archive to the chat's real state

Alternating archive and unarchive requests can split every attempt of the
compare-and-set: each transition write sees the chat archived and each
already-archived write sees it unarchived. Exhausting the retries therefore
proved nothing about whether the conversation exists, and the no-upsert archive
route turned a lost race back into a 404.

Read the conversation once more when the retries run out and answer with its
actual current state instead.
2026-08-16 17:07:44 -04:00
Marco Beretta
7857a99d63
⚡ perf: Flip the Pinned Flag Without a Full Conversation Save (#14862)
Pinning routed through saveConvo, which reads every message id for the
conversation and writes the whole array back just to set one boolean, and
can trigger a project-stats recompute on top.

None of that applies to a pin: it moves no chat between projects, changes
nothing the project workspace hides, and opens no retention window. A
dedicated setConvoPinned does the single findOneAndUpdate instead.

Measured against an in-memory MongoDB with the real message methods
wired in, on a 120-message chat: two driver commands and 3706 bytes
before, one command and 245 bytes after. The write scales with the
message count, so the gap widens on longer chats.

Archiving keeps using saveConvo, which it needs for exactly the project
stats and retention work a pin does not.
2026-08-16 17:07:27 -04:00
Marco Beretta
5fc05ac037
🕰️ fix: Stop Pinning and Archiving From Counting as Chat Activity (#14861)
* fix: stop pinning and archiving from counting as chat activity

Both routes went through saveConvo, which lets mongoose stamp updatedAt.
The sidebar orders chats by that field, so pinning hoisted an untouched
chat to the top of Today, and unarchiving a year-old chat dropped it
there too instead of back into its own date group.

saveConvo now takes preserveUpdatedAt, and both routes pass it. They also
pass noUpsert: with timestamps suppressed an upsert would insert a
conversation carrying neither createdAt nor updatedAt, so an unknown
conversation id is now a 404 rather than a silently created stub.

* test: pin a project's activity pointer against metadata-only saves

Review raised that preserving updatedAt could drag a project's
lastConversationAt back to the pinned chat's older timestamp, since the
incremental path $sets it outright.

That path is not reachable here: a pin carries no chatProjectId, so
previousChatProjectId stays null while the conversation has a real one,
projectMembershipChanged is therefore true, and saveConvo takes the full
recompute branch instead. This test holds that in place, with a newer
sibling conversation in the project so a regression to the incremental
path would fail it.

* fix: keep updatedAt through the retention backfill

Under RetentionMode.ALL a legacy chat with no stored isTemporary gets a
second write after the main update, and that one still had mongoose
timestamps enabled. The first archive of such a chat therefore bumped
updatedAt anyway and landed in Today, defeating preserveUpdatedAt on
exactly the old conversations it was meant to protect.
2026-08-16 16:20:20 -04:00
Marco Beretta
8a946290f6
📌 fix: Fetch Pinned Chats Independently of the Chats List (#14860)
* feat: give the pinned chats section its own fetch

The sidebar's pinned section filtered pinned chats out of the paginated
chats list, which only holds the 25 most recently updated conversations.
Once 25 newer chats existed, a reload hid the pin until the list was
scrolled far enough to fetch the page it lived on.

Pins are now fetched directly via GET /api/convos?pinned=true behind a
dedicated query, so every pin paints with the sidebar regardless of where
it falls in the chats list. Pin and unpin invalidate that query, and the
shared conversation cache helpers keep it in step so a rename, delete or
archive is reflected without waiting for a refetch.

Pins stay out of the date groups, which groupConversationsByDate already
handled.

* fix: address review findings on the pinned chats section

- Drain the cursor rather than capping the pinned request at 100. Since
  pins are kept out of the chats date groups, anything this query dropped
  was invisible in the sidebar entirely, not merely further down a list.
- Apply the active bookmark filter to the pinned request and key its
  cache by it, matching the chats list beside it.
- Move a pin to the top of the section when the caller asks for it, so a
  pin that just received a message leads the way it does in the chats
  list instead of waiting for a refetch.
- Invalidate the pinned list when a conversation is unarchived, since
  archiving removes it from that cache and nothing put it back.
- Index the pinned lookup: it filters on user + pinned and sorts by
  updatedAt, which no existing compound index covered.
- Protect `pinned` from saveMessageToDatabase's unset sweep. Any
  persisted field missing from endpointOptions is unset, so sending a
  message in a pinned chat silently unpinned it.

* fix: keep the pinned cache reconciled across the other convo mutations

Second review pass on the independent pinned query.

- Fall back to the pins already loaded in the chats pages when the
  dedicated request fails. Pins are stripped from the date groups, so an
  error otherwise emptied the section and hid them everywhere.
- Restore default focus and reconnect refetching, matching the
  conversations query. A pin changed in another tab is only reconciled by
  a refetch, since that tab's mutation never touched this cache.
- Invalidate the pinned list from the mutations that can produce or alter
  a pinned chat without going through pin itself: duplicate, fork,
  import, project assignment, and shared-link deletion.

* fix: invalidate pins on tag and project-deletion changes

Third review pass, same class as the last: the pinned query is keyed by
the active bookmark filter, so changing a chat's tags can move it in or
out of that filtered set, and deleting a project unsets chatProjectId on
its chats, pinned ones included.

* fix: cancel in-flight pinned fetches when deleting a conversation

Deletion cancelled the regular and archived queries but not the pinned
one, so a pinned GET issued before the delete could resolve after the row
was stripped and write the deleted conversation back, leaving a row that
navigates to a missing chat. Restoring default focus and reconnect
refetching in the previous commit made those in-flight fetches more
likely, so this widened rather than appeared.

Cancelled on mutate, and invalidated on success since cancelling a race
is best effort.

* test: make the SSE query-cache mock key-aware

The conversation cache helpers now run a second, pinned-keyed findAll
pass. This mock ignored its key argument and always returned an
allConversations entry, so those pinned writes were attributed to
allConversations and the write-count assertions saw three instead of two.

* fix: keep pins in sync through upsert and pin-only pages

Root-level SSE updates and resumable settlement call upsert rather than
update, so the independently cached pinned row never moved or refreshed.
An all-pin first page also left the chats virtual list empty, so
onRowsRendered never asked for the next cursor.

* fix: keep pins current through SSE recovery and project delete

Resumable SSE reconciliation invalidated conversation and allConversations
only, so an independently cached pin kept stale title and order.
Deleting a project-backed pin that lived only in that cache also skipped
the project query, because the mutation never read chatProjectId there.

* fix: keep pins current after bookmark edits and failed pages

Renaming or deleting a bookmark rewrote tags on conversations but left
the tag-keyed pinned cache pointing at the old filter. An all-pin page
whose next fetch failed also retried forever because the empty-list
effect had no memory of the attempt. Unpinning a pin that only lived in
the dedicated cache removed it from Pinned without inserting it into
Chats, and later cursor pages cannot recover a row whose updatedAt just
jumped ahead of the current cursor.

* fix: keep pins visible after a failed refetch

A failed pinned refetch left React Query holding the previous list, so
the nullish fallback never ran and a newly pinned chat vanished from
both sections. Unpinning an older pin also inserted it into every
cached chats variant, including bookmark and search results it would
not match. Drop the checked-in agent task prompt.

* test: type the pinned conversation fixtures correctly

The delete mutation takes a plain string conversationId, but reading it
back off a TConversation fixture widens it to string | null. Hoist the id
into its own constant so the call site passes the real string.

Type the tag fixture as TConversationTag so it carries the required _id
and user fields the mocked resolved value expects.

* style: sort the sidebar imports to the repo order

The new pinned-section imports went in out of the longest-to-shortest
order the import sorter enforces.

* fix: keep drained pins and empty chat caches from breaking the sidebar

A pinned page failing partway through the drain rejected the whole query, so
every pin already fetched was discarded and the section fell back to whatever
the chats cache happened to hold. Publish the accumulated pins before
rethrowing so the retry renders against the partial set.

Unpinning a chat that only lives in the pinned cache reinserted it into the
chats list by spreading the first page, which is absent once removal has
filtered out the last loaded row. Rebuild that page instead, matching the
upsert path.

* fix: order fallback pins by their timestamp

The merge kept dedicated rows in Map insertion order and appended the pins
recovered from the chats cache after them. A chat pinned while the dedicated
refetch is failing is the newest pin, so the server would return it first, yet
it landed last and could sit below the section's visible 30vh. Sort the merged
set newest-first so a fallback row takes the place the server would give it.

* fix: keep the shared badge and the move-to-top order on pins

The pin response has no isShared: the flag is derived per list request by
attachSharedFlags, which only runs for the list queries. Reinserting an unpinned
chat into Chats therefore dropped its shared-link badge, because unlike an
in-place update there is no existing row to carry the flag over from. Read it off
the cached pin before the update removes that row.

The chats cache refreshes updatedAt when it moves a conversation to the top, but
the pinned cache only reordered, leaving the previous turn's timestamp on the row.
Sorting the section newest-first then put it straight back. Refresh the timestamp
there too, so the move survives the sort and both caches agree.
2026-08-16 11:28:33 -04:00
Danny Avila
06bf324cf0
🛤️ feat: Per-Agent Code Execution Routing With Stateful Session Scopes (#14848)
* feat: route code execution per agent profile

* chore: sort execution profile imports

* test: preserve stateful environment literal types

* fix: isolate stateful code environments by user

* fix: preserve per-agent code routing end to end

* fix: route code priming by execution profile

* fix: isolate code profile lifecycle state

* fix: preserve mixed-profile code resources

* fix: complete stateful skill routing
2026-08-16 09:42:15 -04:00
Danny Avila
5e464bc930
📎 fix: Alias Shell Script MIME Variants to application/x-sh (#14817)
* 📎 fix: Alias Shell Script MIME Variants to `application/x-sh`

Chrome on Linux reports `.sh` files as `application/x-shellscript`
(freedesktop shared-mime-info) and libmagic reports `text/x-shellscript`.
Neither string appears anywhere in the source, so uploads were rejected
even though `application/x-sh` is in the default allowlist and
`codeTypeMapping` maps `sh` to it — `inferMimeType` only consults the
extension map when the client sends no type at all, so a non-empty
browser value passed straight through to the allowlist check.

Alias both variants to the canonical `application/x-sh`, matching the
existing treatment of `text/x-markdown` and `application/x-zip-compressed`.

Also attach `statusCode`/`body` to multer file-filter rejections. Without
them the error misses the `isCustomError` branch in `ErrorController` and
falls through to a bare `500 An unknown error occurred.`, so the rejection
reason was logged server-side but never reached the client. The upload
hook already surfaces `error.response.data.message`, so a rejected file
now explains itself instead of showing a generic upload failure.

* 🔁 refactor: Move Upload Error Contract Into `packages/api`

Addresses codex P1 on #14817.

The producer of the `statusCode`/`body` pair now sits beside its consumer:
`isCustomError` and `ErrorController` are already in
`packages/api/src/middleware/error.ts`, and `CustomError` is already in
`packages/api/src/types/error.ts` — only the construction of that pair was
stranded in legacy JS. `createCustomError` is exported from the same module
as the guard that recognizes it, and `multer.js` is back to a thin caller.

Also pins the `.sh` back-compat claim with tests: configs from the
documented workarounds (`application/x-sh` per #4660/#5689/#6297, and the
broad patterns from #14804) still accept a `.sh` upload after the alias
rewrites the type. A negative control confirms the endpoint config is
genuinely in play rather than falling back to the default allowlist.
2026-08-14 01:12:23 -04:00
Marco Beretta
d920328bfa
💬 style: Unify Message Row Layout and Edit Surfaces (#14770)
* style: Unify message row layout and edit surfaces

Route chat, share, and search messages through a shared MessageRow so
user turns render as right-aligned bubbles and assistant turns keep a
visible identity column.

Replace per-part text editors with one edit surface that keeps tools,
errors, and artifacts visible. Preserve non-text fields when saving
content parts, copy the full serialized message, and hide hover actions
that do not apply during streaming or errors.

* style: Align edit footer and lighten editor field in dark mode

Drop the divider above the user edit footer so both edit surfaces share
the same footer treatment.

Move the editor fields to surface-tertiary-alt. Light mode is unchanged
at #fff, while dark mode lifts from #0d0d0d to #2f2f2f so the field sits
above the #212121 panel instead of sinking into near-black.

* style: Drop focus border and ring from message editors

The editor fields changed border color and added a ring on focus. Keep
the border static and rely on the app-level focus handling instead.

* fix: Keep a triggered message action visible when the row is not hovered

Hover actions fade out on non-last rows, and mobile.css only restored
display and visibility for an active button, never opacity. Opening the
fork popover therefore left it anchored to an invisible trigger once the
pointer left the row. Skip the fade entirely while a button is active.

Extract the recipe the three toolbars repeated so the rule has one home.

Rework the streaming guard to the contract the toolbar now implements:
edit and fork are omitted from a streaming response rather than rendered
disabled, and the settled turn above keeps its own actions. It asserted
the removed disabled-and-transparent behaviour and its opacity check only
held because the growing response shifted the row out from under the
pointer.

* style: Trim message edit chrome and stabilize the status row

The edit surface was a titled card sitting inside the conversation: a
bordered panel with an "Edit message" heading wrapping bordered fields,
which read as a settings dialog rather than an inline editor. Drop the
card background, border and heading, and take the footer buttons down to
the small size so the editor reads as a field in the message flow. The
captured row goes from 253px to 187px.

Move "Unsaved changes" into the footer and merge the rerun hint into the
same slot. Both previously added their own row, so typing pushed the rest
of the conversation down. The slot is clamped to two lines, which stays
under the 36px button row, so the footer height holds at 36px regardless
of which message is showing.

* test: Cover message edit layout stability

Add a mock e2e spec that measures the edit footer and section boxes and
asserts they hold steady as the status text appears, for both the
single-part user editor and a multi-part response.

The multi-part case needs an assistant message with two editable parts,
so add an E2E_THINK_REPLY marker to the fake model. Its think tags are
parsed downstream by the agents stream pipeline, which yields a reasoning
part followed by a text part.

* fix: Read the fork popover open state from its store

Fork mirrored the popover state into its own useState and reset it from an
onClose prop. Ariakit 0.4 has no onClose, and React's DOM types accept the
name on any element, so it type-checked, landed on a div and never fired.
Closing by Escape or an outside click therefore left the button reading as
active until the trigger was clicked again.

Read the state from the store instead so every close path clears it.

* fix: Keep the whole toolbar visible while an action is open

Only the triggered button escaped the hover fade, so opening the editor or
the fork popover left the row as a single floating button once the pointer
moved away. Mark the active button and have every action in the toolbar key
off it, so the group stays opaque for as long as a surface is open.

The marker is a dedicated class rather than the existing `active`, which
HoverButtons pins to the edit button of every assistant message and would
hold those toolbars open permanently.

The existing guard pressed Escape to close the editor while focus sat on the
body, so the editor never closed and its assertion only held because the
sibling faded regardless. Close the editor through its own control, and drop
focus before measuring the fade now that Escape returns it to the trigger.

* fix: Withhold copy while a response is still streaming

Text-to-speech, fork and feedback were all withheld from a message that is
still generating, but copy was rendered throughout, so the button offered to
put half a sentence on the clipboard. Gate it on the same condition.

That empties the toolbar for the duration, and SubRow collapses an empty row,
so a streaming response now carries no actions at all until it settles. Both
guards encoded the old contract: the unit test asserted copy was present and
counted a single button, and the browser guard used copy as its proof that the
toolbar had mounted. The settled turn above takes over that role.

* fix: Move retry navigation to the outer edge of a user turn

A user turn is right-aligned, but its sibling navigation rendered ahead of the
actions, so the retry counter sat inboard of the icons instead of under the
edge of the bubble it belongs to. Order it last on user turns.

* fix: Ride the stream instead of chasing it

Following a generating answer went through a helper throttled at 145ms, so the
thread caught up in visible jerks rather than flowing. It now writes the scroll
position directly on each frame, which is what an answer arriving a few pixels
at a time actually needs, and glides only for the one long trip a turn makes,
when sending has to travel from wherever the reader was down to the newest
word.

Whether to follow at all is now answered by where the reader is and which way
they were going, rather than by the abort flag. `useMessageProcess` raises that
flag on any wheel at all, downward ones included, through a throttle whose
trailing call lands after the gesture has ended, so nothing timed to the
gesture could outlive it. Scrolling down to the newest word could therefore
never resume the ride, while the scroll-to-bottom button, which touches no
wheel, always could.

Arrival is judged on the scroll it produces rather than the wheel tick that
started it, because wheel scrolling is animated and at tick time the thread is
still far short of where the tick is taking it. Arriving also counts from
further out than leaving does: while an answer streams the end recedes between
the last tick and the frame that measures it, so judging arrival as tightly as
departure leaves a reader unable to catch it at all.

* fix: Reveal retry navigation on hover while an answer generates

Copy, edit, fork and read-aloud are all withheld from a response that is still
generating, which left the retry counter as the only thing rendering under a
half-written answer. It now reveals on hover there, like the actions it sits
with, and stays put on a settled turn.

* fix: Keep a refused rerun from discarding the edit

While a response is streaming, the edit action stays available on every earlier
row, and those editors see a per-message submitting flag that is false, so
Update and rerun is enabled. The send itself is still refused: ask() returns
false for the duration of the active submission. Both editors ignored that and
closed anyway, so the draft went with them and no rerun ever started.

Both rerun paths now check the result and leave the editor untouched when the
send is refused, so the work survives until the thread is free.

* fix: Let an upward gesture beat the pending send glide

Sending arms a smooth glide down to the newest word, and the landing re-pins the
thread to the bottom. The landing was scheduled two ways, on scrollend and on a
700ms fallback, and neither was ever cancelled. A reader who changed their mind
and headed up mid-flight was pinned again regardless, then dragged back by the
next streaming resize. The fallback fires for the whole window, so this held even
after the glide had visibly settled.

The gesture now marks the glide interrupted, wherever it lets go of the bottom,
and the landing stands down when it sees that. A glide the reader leaves alone
still re-affirms the ride.

* fix: Fade retry navigation on every streaming response format

Every other action is withheld from the row that is still generating, so the
retry counter is the only thing left under a half-written answer. The plain text
row already faded it to hover-only there; the structured rows did not, and left
it sitting on its own.

Both structured paths now apply the same condition, and the class string the
three of them share moves next to the hover action styles it belongs with.

* i18n: Correct the copy the edit surface rewrite left behind

The multi-part hint told the reader to save first and then rerun, but a save
closes the editor and reopening seeds the drafts from what was just saved, so
there is nothing left to rerun and the button stays disabled. Rerunning carries a
single edited section by design, so the hint now states that limit rather than
pointing at a step that is not there.

Drop com_ui_save_submit as well: the per-part editor that used it is gone.

* test: Make the message visual baselines opt-in

The suite asserts sixteen screenshots and the repository tracks none, so
Playwright's default treats every one as a miss and the mock e2e job fails on
Linux. Baselines only compare cleanly against the machine that produced them, and
nothing here can generate ones that match the runner image.

The flows keep running and asserting their structure, which is where their value
was; only the pixel comparison is now gated behind E2E_VISUAL_SNAPSHOTS.

* style: Restore import order in the reworked message files

The repository sorter and CI disagreed with what these files were left holding
after the edit surface rework. No behavior change.

* test: Follow the reworded rerun hint in the edit layout spec

The multi-part hint was restated in the previous commit; this assertion still
expected the old wording and would have failed the mock e2e suite.

* fix: Leave the send glide alone while the answer streams in

Every delta of an answer reruns the scroll effect, and the plain follow writes
scrollTop outright, which cancels an animation on its first frame. So the glide a
send starts was killed by the first token to arrive and the reader was snapped
down instead of carried.

The follow now stands down while a glide is travelling, which is what the hook
already documented but only enforced on the resize path.

* fix: Write a saved edit onto the thread as it stands

An earlier turn stays editable while the newest answer streams, and the save
captured the thread before the request but wrote it back after. Every delta that
landed during the round trip was overwritten. Most of the time the next delta
re-merged and the damage showed as a one-frame truncation, but a save that
resolved after the stream's final write left the cache wrong for the rest of the
session.

The thread is now read once the request has resolved, which is what the content
part editor already did.

The editor actions in this file also wrap again rather than hold one unbreakable
row, for the reason given in the following commit.

* fix: Let the editor actions wrap on a narrow row

At 320px an assistant turn gives the editor about 252px once page padding, the
identity column and the row gap are taken out, and Cancel, Save and Update &
rerun need more than that in English alone. The group was pinned with shrink-0,
so it ran past the edge of the row instead of wrapping. A longer translated label
makes it worse, and the user turn had no margin left either.

Both editors wrap again, which is what the footer did before the status row was
folded into it.

* fix: Catch up to the new bottom when the glide lands

Following stands down for the length of the glide, so an answer that arrives
while it travels moves the bottom past the target the glide aimed at. A short
response that finished before the glide reported landing left the thread a few
lines short of its own end, with nothing left to correct it.

Landing now closes whatever gap opened, unless the reader took over on the way.

* test: Follow the renamed rerun button in the edit flow specs

The button became 'Update & rerun' when the edit surfaces were unified, but two
edit-flow specs still located 'Save & Submit' and would have waited for it until
they timed out. A type comment named the old button too.

* fix: Judge the first thread scroll against a real position

The direction check seeded its last-position ref at 0, so the first scroll
event on an opened thread, which arrives carrying a large positive
scrollTop, read as a jump downward. Near the end that cleared the abort
flag and re-pinned a reader to the stream they were scrolling away from.

Take the first event as a baseline and judge direction from the next.

* fix: Hold the content part editor to what it replaced

EditContentParts took over from EditTextPart and left two of its behaviors
behind.

An emptied box now blocks Save and rerun instead of persisting a blank
part. EditTextPart refused the same edit through its form's required rule
and the sibling EditMessage still does, so both editors hold one line. The
keyboard shortcuts reach the save paths directly, so they are guarded
there too, and the footer says why the buttons are down.

The editor also follows the chat direction again, taking dir and text
alignment from the same setting EditMessage reads.

* fix: Hold the footer height while a response streams

Every action is withheld from the row that is still generating, and a lone
sibling counter renders nothing, so the footer measured zero until the answer
landed and then sprang to the height of the buttons. The transcript stepped
upward under the reader at the moment a response completed.

The placeholder that used to reserve this space went when the footer became
unconditional, so hold the height on the row itself instead.

* fix: Remember where the thread was put before judging a gesture

Direction is judged against the last sample, and the thread is placed at its
end without the reader touching it. With no record of where it was put, their
first gesture was spent taking the baseline instead of being obeyed: a single
PageUp cleared no flag of its own, so the next streamed resize rode the reader
straight back to the end they were leaving.

Every programmatic move now records the position it left the thread at, so the
sentinel stands only until something has actually placed it.

* fix: Spend the start of a turn only once it can be honored

A reader who scrolls away during one answer leaves the abort flag raised, and
nothing lowers it until the next connection opens, which is after this effect
has already seen the send. Marking the turn as started on that first pass spent
it against a closed gate: by the time the flag cleared there was no start left
to honor, the reader was still detached, and the answer they had just asked for
streamed on offscreen.

Record the turn as started only on the pass that acts on it.

* fix: Show the part edits that survived a refused save

The editor saves every changed part through one button, but the endpoint
takes a single part per call and nothing rolls a write back. A part the
server refused therefore left the earlier ones stored while the editor
reported that the message could not be saved, so cancelling from there
walked away from edits that were already live.

Record the writes that landed and reconcile the transcript with them
whichever way the save ended. The refused parts are the only ones left
holding a draft, so a retry no longer rewrites what already arrived.

* fix: Stop a shared transcript from calling the sharer the reader

The share row reused the chat view's user label, which reads "You". It is
the screen-reader heading for the user turn, so anyone opening a share
link heard every prompt the sharer wrote credited to themselves.

Use the neutral "User" label on this surface. It keeps the localization
the row gained, unlike the untranslated string it replaced.

* fix: Let go of the stream when an interaction settles over several resizes

Expanding a tool result mid-answer renders the container first and fills it
once its contents arrive, so one gesture produces more than one resize. Only
the first was credited to the interaction. The second read the reader as still
riding the stream and put them back on the bottom they had just left.

The suppressed resize now settles the ride as well as the near-bottom measure,
using the position the interaction actually left the reader at, so an
interaction that kept them on the end still streams.

* fix: Edit inside a structured text part instead of flattening it

A text content part holds either a string or a { value, annotations } object.
The Assistants thread sync persists the structured form with its file
citations intact, and the editor reads the part through the same union, so
saving an edit wrote a bare string over the whole object and took every
citation with it.

The same object was handed to the tokenizer, which measures length, so a part
that had been edited this way also stored a NaN token count. Write the edit
into value, keep the rest of the part, and count the text itself.

* fix: Keep a saved part's citations in the transcript it is written back to

A text or think part holds either a bare string or a { value, annotations }
object, and the editor already read both through getPartText. Writing the
draft back into the local message cache put the string over the whole value,
so a response carrying file citations lost them the moment it was edited and
did not get them back until a refetch.

Reading and writing now go through the same accessor, so an edit lands in the
shape it was read from and the rest of the part survives.

* fix: Let the message editor follow the chosen font size

Editing a message dropped the draft to a fixed 14px regardless of the
Font Size setting. On dev the textarea carried the markdown class, so it
read --markdown-font-size like the rendered message does; restyling it
into a bordered box replaced that with text-sm, and the new per-part
editor was written the same way. Anyone on Extra Small, Large or Extra
Large saw the text jump the moment they entered edit mode.

Share the .message-content typography with the editors through a
message-editor-text class so a draft is sized like the message it
replaces and keeps tracking the setting.
2026-08-13 19:30:39 -04:00
Danny Avila
1a3e2aebcb
🛰️ fix: Attach Request-Scoped MCP Servers (#14780)
* fix: attach request-scoped MCP servers

* fix: satisfy MCP static checks

* fix: format MCP runtime hint
2026-08-12 23:43:02 -04:00
Danny Avila
dccef82254
🪶 chore: Aggregate Empty MCP Tool Logs (#14767)
* fix: aggregate empty MCP tool logs

* fix: retain server names in MCP tool logs
2026-08-12 22:29:49 -04:00
Ravi Kumar L
9980b6221f
🪢 feat: add Langfuse session links (#14776)
* feat: add Langfuse session links

* fix: tighten Langfuse session link resolution

* fix: clear stale Langfuse session links

* test: verify tenant Langfuse session links

* fix: align Langfuse link with client conventions
2026-08-12 22:25:23 -04:00
Marco Beretta
92a8058f02
🛟 fix: Isolate Invalid Skills During GitHub Sync (#14735)
* fix: treat unrecognized SKILL.md frontmatter keys as warnings

An unknown key in one SKILL.md failed that skill outright, and because the
GitHub sync runner marks a source failed on any validation error, a single
stray key took down every other skill in the repository. Syncing
github.com/cloudflare/skills failed entirely because 2 of its 13 skills
carry a `references:` key.

UNKNOWN_KEY is now a warning, so the skill is stored (unknown keys and all)
and the issue is surfaced rather than fatal. `references` joins the allowed
set with a shallow JSON-safety check instead of a strict kind match: real
files use a string, a list of strings, a list of objects, and a map, and
pinning one shape would reintroduce the same failure.

Malformed frontmatter stays fatal: INVALID_TYPE, INVALID_SHAPE and the
non-plain-object check are unchanged.

* fix: skip individual skills instead of failing a whole sync source

Any error inside the discovery or commit loop reached the outer catch and
marked the entire source failed, so one unusable SKILL.md, one oversized
blob, or one duplicate name cost every other skill in the repository.

Each skill now runs inside its own boundary and a failure is recorded
against that skill. Errors that mean nothing else in the run can succeed
(lock loss, GitHub auth failures, rate limiting) still abort the source
rather than being charged to whichever skill hit them first. Skills are
marked seen before the attempt, so the reconcile pass cannot mirror-delete
the previously synced copy of a skill a later run can repair, and duplicate
names now drop the whole colliding group instead of letting tree order pick
an arbitrary winner.

Status gains `partial` (published some, skipped others) plus a capped
sample of the skipped skills with the reason for each. A run that publishes
nothing and skips something is still `failed`, carrying the first skip's
error. The skipped entries name repository paths, so they follow the same
visibility rule as owner/repo/paths; the bare count does not.

Sync warnings are logged too: a background run has no user-facing surface,
so the log is the only place a maintainer sees why an upstream SKILL.md
looks off.

* test: cover skill sync warnings reaching the log

An unrecognized frontmatter key no longer fails the skill, so a background
sync has nowhere to report it except the log. Every mock in this spec
returned an empty warning list, which left that path unexercised.

* fix: describe nested frontmatter values in the shared skill type

`SkillFrontmatterValue` allowed only scalars and string arrays, while the
server has always stored `hooks` and `metadata` as JSON-safe objects, and now
`references` too. A skill carrying any of them could not be represented by
`TSkill`, `TCreateSkill` or `TUpdateSkillPayload` without a cast.

The type stays free of `any` and `unknown`: values remain JSON-safe by
construction, and the server keeps bounding depth, string length and array
size when it validates them.

* fix: protect moved mirrors and rolled-back counts when a skill is skipped

Continuing past a failed skill exposed two problems that aborting the whole
source used to hide.

A moved skill's mirror keeps its old upstream id until the update lands, and
only the new path was marked as seen, so the reconcile pass read the mirror as
stale and deleted the very copy the skip path exists to preserve. The old id is
now marked as seen too.

Deletion counters were incremented when a stale name-conflicting mirror was
removed, but never undone when the following commit failed and the mirror was
restored. The run no longer stops there, so the status persisted a deletion
that did not happen and the reconcile pass counted the restored row again.
Counters are now rolled back when the restore succeeds.

* fix: bound unknown frontmatter values and keep moved mirrors through duplicates

Tolerating an unrecognized key meant its value skipped the shared JSON-safety
check, so a deeply nested or oversized payload was accepted and persisted under
a key nobody validates. The key stays non-blocking; the value is now held to
the same depth, array and string bounds as every structured key.

A skill that moves into a name another discovered skill also claims is dropped
with the rest of its duplicate group before the sync path can reuse its mirror,
which left the still-published copy unmarked and reconciled away. Both paths now
mark the moved mirror through one helper.

* fix: end the source when a skipped skill fails to roll back

A skill that fails and rolls back cleanly is just a skipped skill. One whose
restore or delete also fails leaves a mirror with half-rewritten files or a
half-created row, and the run now continues past it, so the source could report
partial success while that mirror stayed inconsistent and its pre-marked
upstream id kept reconciliation away from it.

Failed rollbacks now raise a source-fatal error carrying the original failure,
which stops the source the way a lost lock or a refused GitHub token does.

* test: cover a skipped skill discovered at the repository root

A repository-level SKILL.md is discovered with an empty path, so this pins
that a skip recorded against it still persists with the rest of the partial
status rather than taking the whole status row down with it.

* docs: describe unknown skill frontmatter warnings

* fix: preserve mirrors after partial skill sync

* fix: preserve skill validation details during sync

* fix: fail sync when mirror identity cannot be restored

* fix: harden skill sync failure boundaries

* fix: preserve skipped skills on fatal sync

* fix: surface skill sync diagnostics and rollback failures

* fix: preserve skill frontmatter extension keys

* fix: reject skill frontmatter keys that collide when normalized

Frontmatter keys are matched case-insensitively against the canonical
key list, so "Name" and "name" both resolve to "name". Every call site
normalized independently, and the last key in iteration order silently
won, meaning the effective value depended on YAML ordering rather than
on anything the author could see.

Centralize the normalization in normalizeSkillFrontmatterKeys and have
it fail when two recognized keys resolve to the same canonical key,
rather than picking one. parse.ts, deployment.ts and the agent handler
now surface that as a parse error; createSkill and updateSkill surface
it as a blocking DUPLICATE_KEY validation issue. Unrecognized keys are
still passed through untouched so extension frontmatter survives.

deriveStructuredFrontmatterFields and both write paths now run on the
normalized map, so a "Disable-Model-Invocation" key derives the same
column a lowercase one does.

* fix: harden github skill sync against dropped requests and failed cleanup

Three failure paths in the GitHub sync could leave a source looking
healthier than it was.

githubJson only handled HTTP-level errors. A fetch that rejected before
producing a response (DNS failure, socket reset, abort) escaped as a
raw TypeError, so the sync reported a generic crash instead of a typed
sync error. Wrap it as GITHUB_REQUEST_FAILED and add that code to the
fatal set, since a source whose requests never complete cannot be
partially synced.

When a synced file failed to persist, the orphaned upload was cleaned
up on a best-effort basis and the cleanup error was only logged. If the
cleanup itself failed, the source still ended with the original error
and left a real orphan behind. Promote that to a rollback failure so
the source reports SYNC_ROLLBACK_FAILED with the triggering error.

Skill warnings were logged inside commitRemoteSkill, before the file
sync and viewer setup that can still roll the skill back. A skill that
never survived publication therefore emitted warnings as though it had.
Return the warnings from the commit and log them once the skill is
fully published.

* fix: report skipped github skills before credential errors

serializeErrorMessage checked isCredentialError first, and that check
matches on the error text. A skipped skill whose path happens to
contain a credential-ish word, for example skills/credential-helper,
was therefore redacted to "GitHub skill sync credentials are not
available" for admins without credential-metadata access, hiding a
parse failure behind a wrong diagnosis.

Check the promoted skipped-skill case first, since it is identified by
error code rather than by text and is the more specific match. The
credential redaction still applies to everything else.
2026-08-12 13:22:06 -04:00
Danny Avila
88e08c91e8
🧷 fix: Preserve Elicitation Answers Across Aborts (#14745)
* fix: preserve elicitation answers across aborts

* chore: sort stream type imports

* fix: guard malformed resolved answers

* fix: close abort answer race gaps

* fix: retain exact answers across pauses

* fix: retain answers across resumed pauses

* test: satisfy HITL fixture types

* fix: retain legacy answers through approvals

* fix: preserve answers in reconnect snapshots

* test: type legacy answer fixture

* fix: bind legacy answers to paused content

* test: guard optional resume content

* fix: resume questions without streamed content

* fix: Preserve legacy answers through abort filtering

* test: Narrow reconstructed abort fixture

* test: Type abort transform fixture explicitly

* fix: Isolate answers with missing ask content

* test: Type missing-content ask fixture
2026-08-12 07:32:48 -04:00
Danny Avila
7347cfc195
🍡 feat: Batched User Questions With A Single Bounded Answer Form (#14737)
* feat: support batched user questions

* test: align batched question fixtures

* fix: harden batched question lifecycle

* test: submit batched HITL answers in e2e

* fix: address batched question review findings

* fix: preserve invoke return typing
2026-08-11 01:06:16 -04:00