Commit graph

736 commits

Author SHA1 Message Date
Ravi Kumar L
cd4cdf2ca0
🗄️ fix: support DocumentDB aggregations for insights (#15226) 2026-08-25 19:59:09 -04:00
Danny Avila
95e1a5c9fa
🧠 fix: Correct Memory Token Accounting (#15224) 2026-08-25 19:58:30 -04:00
Danny Avila
290b8664d9
🧾 feat: Track Authoritative Agent Event Outcomes (#15213)
* feat: track authoritative agent event outcomes

* fix: isolate agent event outcome types

* fix: declare agent event handler result

* fix: simplify agent event status selection

* fix: preserve authoritative event outcomes

* fix: preserve terminal event evidence

* test: use completed run-step envelope

* test: scope deferred HITL question locator

* fix: settle every agent event terminal path

* style: sort terminal host action imports

* fix: fence agent event terminal evidence

* fix: recover agent event terminal settlement

* fix: scope terminal retry hints by generation

* fix: settle terminal host actions exactly
2026-08-25 18:20:57 -04:00
Danny Avila
6360821470
🪢 feat: Coalesce Bound Agent Event Bursts (#15208)
* 🪢 feat: Coalesce Bound Agent Event Bursts

* fix: Restore API Package Build

* fix: Keep Trigger Batches Within One Window

* fix: intersect trigger batch windows

* fix: fence trigger batch lifecycle

* style: sort trigger service imports

* fix: make batch requeue crash-recoverable

* fix: fence concurrent batch requeues
2026-08-25 10:48:56 -04:00
Danny Avila
d641c398d5
🧳 fix: Port Subagent Control Receipt Writes to DocumentDB-Safe Operators (#15171)
* fix: harden subagent control receipt persistence

* fix: harden durable subagent control replay

* fix: await terminal control receipts on shutdown

* fix: close subagent control replay races

* test: type stale-owner transport fixture

* fix: quiesce durable subagent controls

* test: await subagent shutdown durability boundary

* fix: serialize durable subagent controls

* fix: fail shutdown on cleanup errors

* fix: report cancellable result availability accurately

* fix: fence subagent control receipt ownership

* fix: close distributed control receipt races

* test: type control receipt race fixture

* chore: require authoritative control receipts

* fix: close subagent control lifecycle races

* style: separate control reservation member

* test: harden subagent settlement wait

* fix: preserve authoritative control replay state
2026-08-24 20:05:39 -04:00
Danny Avila
a997275902
🧾 feat: Persist Authoritative Subagent Control Receipts (#15168)
* feat: persist subagent control receipts

* fix: require control receipt persistence

* fix: preserve authoritative control history
2026-08-24 11:39:56 -04:00
Danny Avila
fc2b8584c4
📇 feat: Surface Event Child Activity Through a Bounded Parent Index (#15142)
* feat: surface event-driven child activity

* fix: keep child task aggregation documentdb-compatible

* fix: address event activity review findings

* test: provide markdown message context defaults

* fix: report bounded child history truncation

* fix: preserve current child activity state

* fix: preserve durable event child activity

* fix: handle missing task timestamps

* fix: keep active event snapshots live

* fix: preserve event activity across valid anchors

* fix: close event child activity gaps

* fix: preserve event activity across resume
2026-08-23 18:50:00 -04:00
Danny Avila
d864597731
⚡ perf: Append Saved Message Ids Instead of Rebuilding the Conversation Array (#15141)
* ⚡ perf: Append Saved Message Ids Instead of Rebuilding the Conversation Array

Every saveConvo read every message id in the conversation (sorted) and
wrote the array back onto the document — twice per chat turn, O(n) in
conversation length, from a write path. The turn's savers know exactly
which message they just wrote, so they now pass it as
metadata.appendMessageIds and saveConvo $addToSet-s it, skipping the
read and the full-array rewrite. Every save without the option — titles,
archive, fork, import, threads — still rebuilds from the database, which
remains the heal point for the drift that message deletion has always
left behind (deletes never ran saveConvo).

The array's consumers read presence or length, or use it as an
optimistic cache placeholder, so incremental maintenance is
behaviorally identical; on traced turns the array stays exactly equal
to the messages collection.

Per-turn queries: 15 -> 13 (two Message.find gone), and the growing
array payload no longer crosses the wire twice per turn.

* 🎯 fix: Brand the Lineage-Only Resolved Conversation Instead of Guessing by Shape

The resolved-conversation files fast path treated an absent files
property as unresolved so the lineage-only partial from a bound
agent-event continuation could not silently hide a conversation's
uploads. But MongoDB never stores an empty files array, so nearly every
real conversation also lacks the property and the fast path never fired
— a follow-up turn on an upload-free conversation still paid the
getConvoFiles round trip.

The synthesized partial is the one object that cannot speak for the
database, so it now carries an explicit symbol brand
(PARTIAL_RESOLVED_CONVERSATION, non-serializing and invisible to key
iteration), and a stored document without files means what it means:
no files. Traced follow-up turns drop from 14 queries to 13.

* 🧪 test: Expect the Appended Message Id in the Route's saveConvo Metadata

messages-get.spec.js pins the exact metadata POST /api/messages passes to
saveConvo; the route now forwards the saved message's _id as
appendMessageIds, which is the behavior the append path depends on.
2026-08-23 16:52:44 -04:00
Danny Avila
f3c6e24f84
⚡ perf: Cut Serial Round Trips and a 100ms Admission Stall from Chat Turns (#15138)
* ⚡ perf: Stop Awaiting the Conversation Access Marker Write

Without Redis the CONVO_ACCESS violations namespace is backed by keyv-file,
whose debounced write resolves after ~100ms. validateConvoAccess awaited
that write before calling next(), so the first message to any existing
conversation waited ~100ms before the request was even admitted — once
per conversation per ten-minute window, on every default deployment.

The marker only short-circuits the next check, so the write no longer
gates the request. The same read now stashes the full document on
req.resolvedConversation (null when absent) for downstream consumers.

First-turn ack on an existing conversation: 109ms -> 5ms.

* ⚡ perf: Read the Conversation Once per Chat Turn

A chat turn read the same conversation document four times: the access
check (two fields), the subagent thread guard (full document), agent
initialization (the files field), and the first save. The access check
now reads the full document and leaves it on req.resolvedConversation,
the guard accepts that pre-resolved document instead of re-reading, and
initializeAgent takes the conversation's file refs from it rather than
issuing a separate findOne.

Two serial round trips removed from every turn; the same document still
serves the first save as before.

* ⚡ perf: Remove Duplicate JWT Authentication on Agents Routes

routes/agents/index.js applies requireJwtAuth and then mounts the v1
router at '/', which applied requireJwtAuth again. Every request through
the agents router — chat turns included — ran the passport strategy
twice: two signature checks and two user document reads. The v1 router
is mounted nowhere else; its separately exported avatar router carries
its own auth in files/index.js.

* ⚡ perf: Skip the History Read for Root-Parent Turns and Walk the Tree in O(n)

loadHistory fetched every message in the conversation and then walked the
parent chain from the request's head. For a new conversation — or a new
branch from the root of an existing one — the head is the root sentinel,
which no message carries as its id, so the walk was empty by construction
and the fetch was wasted. It now returns early.

getMessagesForConversation found each ancestor with Array.find inside the
walk, O(n^2) on a linear conversation (~5ms at 1000 messages). A Map by
messageId makes it O(n); first-match semantics are preserved.
2026-08-23 15:17:35 -04:00
Marco Beretta
44d97f859d
⏰ feat: Custom Cron Cadence for Scheduled Chats (#15084)
* feat: custom cron cadence for scheduled chats

Scheduled chats could only be built from four fixed presets, each pinned to a
single hour and minute, so anything outside that shape (twice a day, every 15
minutes, the 1st of the month) was not expressible. This adds a Custom cadence
that takes a raw five-field cron expression.

The cadence schema becomes a discriminated union on `frequency`. A cron row
carries `expression` instead of the hour and minute it cannot represent, since
there is no single hour for `0 9,17 * * 1-5`, and the Mongo schema requires each
field only for the shape that has it: a blanket `required` would reject every
cron write, and dropping it entirely would let a structured cadence silently
fire at 00:00 with a missing hour.

Five fields only. croner also reads a six-field form carrying seconds and a
seven-field form that pins a year, and both are refused. Seconds would promise a
precision the engine does not keep, since it polls on a thirty-second tick and
offsets each schedule by up to two minutes of jitter. A pinned year makes a
cadence that runs out, and every place that computes a next run reads "no next
occurrence" as a cadence it cannot read.

Compilation, validation, next-run previews and interval measurement live in
packages/data-provider so the dialog and the engine share one parser and cannot
drift. The dialog previews the next occurrences, enforces the admin interval
floor and disables its own submit from the same functions the server validates
with, so it cannot offer a Create the API answers 400 to.

The interval floor now covers cron, and measures it twice, taking the smaller.
The nominal gap is probed in UTC and discounted by the same worst-case DST
allowance the structured branches carry, which keeps `0 9 * * *` reporting
exactly what the Daily preset reports. Real elapsed time is then measured in the
schedule's own zone across each of that zone's transitions, because
spring-forward compresses a gap that straddles one: `0 0,12 * * *` in
America/New_York is 11 hours that day, not 12, and a floor between the two would
otherwise be bypassed. The floor ships with the schedules list so the dialog can
mirror it rather than surfacing it as a 400 after submit.

Radio gains a wrap variant, since five frequency segments no longer fit one row
in a phone-width dialog and a translated label can push even a desktop one over.
Its indicator follows the selection across rows; the single-row default is
unchanged.

* fix: mark the cron input invalid when the interval floor rejects it

A floor-violating expression disabled Create and rendered the cadence
message, but the input itself still said aria-invalid=false and its
aria-describedby never reached that message, leaving a screen reader
user with a disabled Create and no stated reason.
2026-08-23 14:35:18 -04:00
Danny Avila
8f7864ba2a
⏱️ test: Stabilize Meilisearch Retry Cleanup Assertion (#15129) 2026-08-23 02:37:57 -04:00
Danny Avila
1de88e7e91
📨 feat: Continue Bound Child Agents from Events (#15112)
* feat: add authenticated agent event ingress

* style: sort agent ingress imports

* fix: harden agent event ingress

* fix: bind event provenance to API keys

* fix: inspect event input with legacy PII filters

* fix: scope event status reads to source keys

* fix: bind event status reads to remote sources

* feat: add bound event-driven child turns

* fix: harden event-bound child continuations

* fix: satisfy event binding type contracts

* fix: close event actor lifecycle races

* fix: harden event actor dispatch continuity

* fix: fence event actor resume lifecycle

* fix: bind event actor state to lifecycle

* fix: preserve cascade write outcomes

* test: type cascade failure injection

* style: sort cascade test imports

* fix: harden event child lifecycle boundaries

* fix: make event cleanup retryable

* fix: annotate event retention clock

* fix: reconcile partial cascade metadata

* fix: recheck event binding expiry on resume

* fix: fence event actors by retention deadline

* fix: close event actor lifecycle races

* fix: harden event child lease acquisition

* fix: lazy-load event child lease adapter
2026-08-23 01:15:57 -04:00
Joseph Licata
fc7d56dffe
🪙 chore: Update GPT-5.6 Sol Token Cost Rates (#15114) 2026-08-22 11:02:29 -04:00
Danny Avila
3ebef4c84e
📨 feat: Add Authenticated Agent Event Ingress (#15110)
* feat: add authenticated agent event ingress

* style: sort agent ingress imports

* fix: harden agent event ingress

* fix: bind event provenance to API keys

* fix: inspect event input with legacy PII filters

* fix: scope event status reads to source keys

* fix: bind event status reads to remote sources
2026-08-22 01:09:04 -04:00
Danny Avila
67b7b441b2
🛂 feat: Filter Model-Bound Content by Source (#14425)
* feat: introduce optional content protection seam

* feat: enforce source-aware content filters

* feat: complete source-aware content enforcement

* test: activate skill file-text fail-close fixtures

* fix: harden source-aware content filters

* fix: harden model-bound content filtering

* fix: preserve legacy filters and generated files

* fix: inspect shared scalar metadata

* test: align mocks with current dev dependencies

* feat: add persisted content filter safeguards

* feat: complete source-aware content filter enforcement

* fix: move resume content preflight into TypeScript

* fix: close content inspection edge cases

* fix: harden content protection boundaries

* fix: complete content protection safeguards

* test: align persisted memory filter coverage

* fix: reconcile content protection with current dev

* fix: reconcile content protection with latest dev

* fix: close content protection review gaps

* fix: enforce source-aware provider boundaries

* fix: preserve legacy PII preflight semantics

* test: stabilize stored branch preflight fixture

* fix: defer agent writes until protected model admission

* perf: harden source-aware model-bound filtering

* fix: canonicalize provider lineage before validation

* fix: satisfy model-bound callback type checks

* perf: Bound content protection filtering work

* fix: Bound submission array traversal

* fix: Stabilize bounded content snapshots

* fix: Scope model-bound traversal overflows

* fix: Preserve scoped content inspection

* fix: Accumulate aggregate traversal scopes

* fix: centralize content policy boundaries

* test: align deferred tool policy context

* test: align controller policy mocks

* style: normalize content protection imports

* fix: close content policy review gaps

* fix: narrow active skill policy config

* fix: address content protection review boundaries

* fix: retain exact provenance overflow sentinel

* fix: preserve literal and scoped provenance updates

* fix: narrow persisted edit provenance

* fix: isolate exact overflow attribution

* fix: centralize stored prompt protection

* fix: fail closed on incomplete transcript evidence

* fix: align canonical transcript routing

* refactor: centralize content policy preflights

* fix: isolate upload policy error typing

* style: sort policy preflight imports

* refactor: centralize content policy boundaries
2026-08-21 22:43:32 -04:00
Danny Avila
10f95c0ce9
🐇 refactor: Decouple Meilisearch From Document Saves (#15109)
* 🐇 fix: Decouple Meilisearch from document saves

* 🧭 fix: Preserve detached indexing invariants

* 🧬 fix: Version detached Meilisearch writes
2026-08-21 22:26:44 -04:00
Danny Avila
08c9cc3d3d
🖼️ fix: Restore Shared Subagent Activity as a Read-Only View (#15108) 2026-08-21 20:58:14 -04:00
Danny Avila
749eed0d60
🪟 feat: Unify Subagent Activity Panel (#15106)
* feat: unify subagent activity panel

* fix: fence durable activity to selected task

* fix: preserve exact panel activity semantics

* fix: scope panel identity to parent turn

* fix: keep detached readiness status neutral

* fix: harden subagent activity invariants

* test: support backend TypeScript target

* fix: preserve subagent invocation identity

* fix: bound subagent activity correlation

* fix: drain exact-parent subagent updates
2026-08-21 19:51:11 -04:00
Oliver Faust
b399ad8370
📄 fix: Serve Stored Text for "Upload as Text" File Downloads (#14723)
* 📄 fix: Serve Stored Text for Text-Source File Downloads

"Upload as Text" attachments store extracted content in the DB with
source 'text'; OCR uploads persist the OCR strategy name (e.g.
'mistral_ocr') as a filepath placeholder since no backing file exists.
The download route resolved these records to the local strategy and
passed the placeholder to fs.createReadStream, which failed with
ENOENT — and the response was never ended after the stream error, so
the request hung until the client timed out.

Serve the stored text directly as a .txt download for text-source
files (re-fetched by _id, as getFiles excludes 'text' by default),
and end the response on stream errors: 500 without the download
headers before headers are sent, otherwise abort the truncated
response so clients detect the failure.

* fix: Preserve text-source preview semantics

* fix: Complete text-source download coverage

* fix: Tie text downloads to blob lifecycle

* fix: Isolate preview downloads and share text snapshots

* fix: Keep shared previews in share scope

* style: Sort text download imports

---------

Co-authored-by: Danny Avila <danny@librechat.ai>
2026-08-21 14:24:43 -04:00
Marco Beretta
33e42e6d5d
🎛️ feat: Configurable SearXNG Search Options (#14987)
* feat: configurable SearXNG search options

SearXNG queries were hardcoded to google,bing,duckduckgo with no way to
change the engine list, the result language, or the request timeout. Most
self-hosted instances get served CAPTCHAs by DuckDuckGo, so a third of
every query silently returns nothing and operators have no lever to pull.

Add a searxngSearchOptions block to the webSearch config that accepts
engines (as a comma-separated string or a list), language, timeRange, and
timeout, and thread it through to the search tool. Engines are normalized
to the comma-separated form SearXNG expects, with blank entries dropped so
a stray comma cannot produce an empty engines parameter.

Refs #14117

* fix: normalize SearXNG engines on the runtime config path

The engines transform lived only on the zod schema, but loadCustomConfig
returns the raw YAML object rather than result.data, so nothing downstream
ever saw the transformed value. A YAML list reached the SDK as an array and
threw "options?.engines?.trim is not a function" when the search tool was
built, taking web search down entirely for the exact block the example yaml
documents. An untrimmed string reached SearXNG with spaces still in it.

Extract the normalization into normalizeSearxngEngines and apply it in
loadWebSearchConfig as well as the schema, so both the parsed and the raw
path produce the same comma-separated value. Widen the loader's parameter to
TWebSearchConfigInput, which models engines as the list or string an operator
actually writes, and cover the raw path with tests that call the loader rather
than the schema.

* chore: drop unused RerankerTypes import in web config loader
2026-08-21 11:36:23 -04:00
Marco Beretta
29f6ec6eae
🙈 feat: Config Option to Hide Response Feedback Buttons (#15085)
* feat: add interface option to hide response feedback buttons

Adds `interface.feedback` to librechat.yaml. When set to false, the
thumbs up/thumbs down buttons are removed from the message action row
and the feedback endpoint rejects writes with 403, so deployments that
do not consume the data can stop collecting it. Defaults to true.

* refactor: hoist the feedback gate out of the message row and into typed middleware

Reading startup config inside HoverButtons put a query observer and two Recoil
subscriptions on every message row, and rows never unmount, so the cost grew
with the conversation. Resolve the flag once per chat in useChatHelpers and
carry it on TMessageChatContext; useMessageActions withholds handleFeedback
when it is off, which the action row already treats as "no feedback controls".
The flag now stays false until the config resolves, so a disabled deployment
never flashes controls whose writes are rejected.

Move the server-side policy into requireFeedbackEnabled under packages/api so
the route keeps no policy of its own.

* test: stub the feedback gate in specs that replace the api package

The messages router now imports requireFeedbackEnabled, and express rejects an
undefined handler at require time, so every spec that mocks @librechat/api
wholesale has to carry the export.
2026-08-21 11:16:44 -04:00
Danny Avila
634432b2ae
🪟 feat: Read Child Threads Through Their Parent (#15073)
* feat: add parent-scoped subagent thread reads

* fix: tighten child thread read bounds

* perf: project child activity messages

* test: update child activity route fixtures

* test: satisfy response mock types

* fix: bound child activity reads at storage

* style: sort child activity imports

* fix: bound child activity storage reads
2026-08-21 11:10:26 -04:00
Danny Avila
8c14f03432
🗂️ feat: Scope Scheduled Chats to Chat Projects (#15056)
* 🗂️ feat: Scope Scheduled Chats to Chat Projects

Adds an optional chat-project destination to a schedule, plus the operator
config to require one — or to pin every scheduled run to a specific project.

Feature:
- `chatProjectId` on the schedule row, accepted on create/update, projected on
  the wire, and carried into the run's conversation through the durable trigger
  envelope's `run` context.
- `interface.schedules.requireProject` refuses schedules that are not filed
  under a project; `interface.schedules.projectId` pins every run to one
  project and implies the requirement.
- Dialog gains a project picker (required when configured, a read-only row when
  pinned); the card shows the destination and the new disabled reasons.

Invariants:
- ONE resolver (`resolveScheduleProjectId`) decides the destination for the
  write handler, the fire path, and the wire projection alike, and an operator
  pin OUTRANKS the stored id in all three. Tightening the config therefore
  redirects — or stops — existing schedules instead of grandfathering where
  their runs land. A pin implies `requireProject` for the same reason: without
  it, a row created before the pin would keep firing with no project at all.
- Create/fire precheck symmetry, mirroring `resolveAgentFireAccess`: a write
  this handler accepts is one the next fire also accepts. Any edit leaving a
  schedule ENABLED re-validates its EFFECTIVE (possibly stored) project, like
  the existing stored-agent and cadence-floor rechecks. A DISABLING edit skips
  the requirement, or a schedule auto-disabled for `project_required` could
  never be turned off.
- Fire-time enforcement auto-disables rather than filing runs loose, matching
  agent_deleted: new `project_required` (requirement raised after creation) and
  `project_deleted` (gone, or pinned to a project this owner does not have)
  reasons, both refused BEFORE a billed generation is dispatched, and both
  advancing so a schedule can never wedge on the occurrence.
- `computeCreateDigest` appends the field only when present, so a payload
  without a project digests byte-identically to one from before this change —
  a create retried across the upgrade still matches its own row instead of
  reading as key reuse.
- Project reads are scoped to the owner, so ownership and existence are the
  same lookup; a read ERROR propagates instead of failing closed, so a Mongo
  blip retries the fire rather than auto-disabling the schedule.

The trigger idempotency key hashes principal/event/target and never
`envelope.run`, so the added run field cannot destabilize delivery identity.

* 🗜️ fix: Keep the Schedule Dialog Inside Its Height Budget

The project picker landed as a new ROW in the schedule dialog, which broke the
e2e edit spec: `md:overflow-visible` turns off the template's scrolling from
`md` up, so the dialog's content must fit the viewport. The extra row pushed the
footer's Save button below a 720x1280 window, where Playwright reported a
visible, enabled button it could never click — 226 scroll-into-view retries and
a 2-minute timeout, on all three attempts.

Measured against `dev` at 1280x720 (Save button's bottom edge, viewport 720):
  dev              688   (32px slack)
  project row      ~790  (off-screen, CI failure)
  3-column row     704   (16px slack — half the budget spent)
  this commit      686   (34px slack, 2px better than dev)

The identity row is now three columns — name, agent, project — and its caption
moved out of the agent cell to sit full width beneath the row: at a third of the
dialog that sentence wraps an extra line, and the row is the tallest thing
competing for the budget. The caption is grouped with the row rather than left
to the form's own 4-unit rhythm, which spent more height on the gap than the
caption occupies.

The e2e spec now asserts the button is in the viewport before clicking it, so
the next field that overflows this dialog says so in one line instead of a
two-minute timeout on a visible element. FOLLOWUPS.md records what the planned
dialog controls (multi-day weekly, timezone, attachments) need first: give
`ControlCombobox` the `portalElement` prop `Dropdown` already has, portal the
popovers into the dialog content, and let the form scroll again.

* 🧹 fix: Address Codex Review on Scheduled Chat Project Scope

Four P2 findings, all real.

Unreachable clearing path (ScheduleDialog). The picker only held live projects, so
`com_ui_schedule_project_none` was a PLACEHOLDER — nothing selectable. Once a
schedule had a project the owner could never take it away, leaving the server's
`chatProjectId: null` path reachable only by API. The picker now carries a real
"No project" option whenever a project is optional, and omits it when one is
required, where there is nothing valid to select.

Placeholder shown for a real project (ScheduleDialog). A stored or pinned project
outside the first loaded page had no name in the paged map, and the combobox
renders its placeholder for an empty display value — telling the owner a scoped
schedule had no project. That one project is now read by id, with the raw id as a
last resort: a poor label, but an honest one.

Project policy skipped at the resume boundary (service.ts). `claimScheduleResume`
re-applied the schedules gate, the revision fence, the kill switch and
SCHEDULES:USE, but not the project policy this PR added. Approving a paused run
whose project was deleted — or whose owner now sits under a requirement or a pin
it no longer satisfies — billed a continuation the very next scheduled fire would
refuse and auto-disable the schedule for. The effective-project resolution now
runs there too, refused before the lease and the capacity slot so a policy refusal
costs nothing and leaves no state to unwind. NOTE: agent access and balance are
still not rechecked on resume; that gap predates this PR and is left alone.

Per-card project derivation (ScheduleCard). Every card ran the projects hook and
rebuilt the full option array, name map, and one icon element per project, to use
a single name — O(schedules x projects) per render and per project-list refresh.
The hook is split: `useChatProjectNames` (map only, for the panel, which resolves
every card's name once and passes it down) and `useChatProjectPicker` (options and
pagination, for the dialog's one combobox). The panel skips the query entirely
until some schedule actually has a scope.

Tests: four at the resume boundary (verified failing without the gate) and four in
the dialog spec. The picker selection in one existing test now goes through the
search field — the popover's VIRTUALIZED renderer sizes its window from a scroll
height jsdom always reports as 0, so with three options it materialized only two.
Full schedules e2e re-run green against the rebuilt client.

* 🎯 fix: Settle Project-Policy Refusals and Keep Create Retries Idempotent

Second Codex round, four P2s. Three were consequences of the resume gate added in
the previous commit, which was half-built: it admitted where it should not and
stranded the run where it refused.

Project policy moves from `claimScheduleResume` into `isScheduleLive`'s `policy`
branch. Both entry points consult that branch FIRST, and both already route its
refusal through abort-and-settle — so a policy stop now settles the occurrence
instead of answering a bare 409 while the job stays `requires_action`, the card
keeps reading "Needs approval", and every retry repeats the same 409 until expiry.
No change to resume.js: the existing branch does the work.

The rule is deliberately NARROW. It refuses only where no valid destination is
left — the requirement is on with nothing satisfying it, or the schedule's own
project is gone (which also unset it on the conversation). It does NOT refuse
because an operator's pin moved: the paused conversation cannot be rebound
(`chatProjectId` is excluded from the resume context and the continuation reuses
the same conversationId), so refusing would strand a pending approval over a
destination it can never reach, for a pin that governs only where the NEXT run
lands — which the fire path already redirects.

Create retries are idempotent again. Project policy had been applied BEFORE the
`clientRequestId` replay lookup, so a raised requirement, a deleted project, or a
moved pin could answer 400 for a create that already committed — pushing the
client to rotate its key and create a DUPLICATE schedule, the exact failure the
key exists to prevent. Policy now applies only to a genuinely new insert, and the
digest is computed from the CLIENT's payload rather than the resolved destination,
so today's policy can no longer re-digest a genuine retry into a mismatch.

An explicit `chatProjectId: null` under a pin is refused rather than silently
resolved to the pin. The payload contract defines `null` as clearing the scope;
answering 201 while filing under the pin reported success for the opposite of what
was asked. Only an OMITTED field takes the pin silently.

Tests: five on the policy branch (both refusals verified failing without it, plus
guards that a moved pin and a live project still admit) and two on the handlers
(the pinned explicit clear, and a committed create recovered by retry after the
policy tightened — verified answering 400 without the reordering).

* 🧭 fix: Converge the Stored Project on the Destination a Fire Resolved

Third Codex round, four P2s.

The root confusion behind the resume findings: an operator pin outranks the stored
id at fire time, `fireSchedule` sends the pin in the trigger envelope, and the row
keeps its old value. The row therefore LIED about where that occurrence's
conversation went, and every later re-validation — the resume boundary above all —
checked a project the conversation was never filed under. A schedule storing A,
pinned to B, with B later deleted and A still live, was admitted for resume into a
conversation that had just been unscoped.

Fixed at the source rather than at each reader: a fire that resolves a destination
different from the stored one writes it back, claim-token fenced like every other
worker-side write and deliberately WITHOUT a configRevision bump — this is the
server reconciling itself to policy, not an owner edit, and a bump would fence an
in-flight occurrence off its own run. Written only AFTER the destination validates,
so an unusable pin never lands in the row, and best-effort: the envelope already
carries the right destination, so a failed write costs accuracy on a later recheck,
never the run itself. The wire projection already reported the pin, so this also
stops the row and the UI disagreeing.

An explicit `chatProjectId: null` under a pin is now refused on the DISABLING edit
path too. The pin check and the requirement are independent rules, and folding them
together let `{enabled: false, chatProjectId: null}` skip the pin check entirely,
unset the row, and answer with a wire projection still naming the pin. Only the
requirement is waived for a disabling edit.

The dialog no longer requires a project for an edit that leaves a schedule DISABLED.
The server waives the requirement there precisely so a row auto-disabled for
`project_required` can still be renamed or tidied up; requiring it in the form made
that unreachable, and an owner with no projects could not edit the stopped schedule
at all.

FOLLOWUPS.md records the two residual gaps with their exact triggers: the sub-second
deletion race inside the resume claim window (which needs the effective project
persisted per OCCURRENCE plus a distinct policy conflict routed through
abort-and-settle), and the fact that convergence happens only when a schedule fires.

Tests: four on convergence (pin written, no write when unchanged, no write for a
destination that failed validation, fire survives a failed write), two on the
disabling-edit pin rules, two in the dialog. Schedules e2e re-run green.

* 🔑 fix: Keep an Explicit Project Clear Out of an Omitted Field's Digest

`computeCreateDigest` appended `chatProjectId` on `!= null`, so an OMITTED field and
an explicit `null` produced the same digest. Because the replay lookup and
`matchesCreateIntent` deliberately run before project policy, a request could reuse a
pinned create's `clientRequestId` while explicitly sending `chatProjectId: null` and
receive 201 describing the pinned row — success reported for the opposite of what it
asked, and the pinned-clear refusal the normal create path applies never reached.

Now `!== undefined`: an omitted field still digests byte-identically to a payload from
before project scope existed, so a create in flight across the upgrade still matches
its own row, while an explicit clear is a distinct intent and digests differently. A
pre-scope client never sent the field at all, so nothing legacy can carry an explicit
null.

* 📍 fix: Validate a Paused Run Against the Project Its Own Occurrence Used

The schedule-wide convergence from the previous commit was not enough, and the
reason is the single-active run index: it covers `status: 'started'` only, so a
PAUSED run does not block the next occurrence. While run 1 sat paused in project A,
a pin move plus one later fire rewrote the schedule row to B — and the resume
policy then validated B while run 1's conversation was still filed under A. Delete
A and the continuation was admitted into a conversation that had just been unscoped.
That window lasts as long as the pause, not the sub-second race the previous commit
documented.

The reservation now records the destination THIS occurrence used, and
`isScheduleLive` validates that record when given the occurrence's `scheduledFor` —
which `resume.js` already reads two lines above the call. No new conflict type and
no settlement-path surgery: the refusal rides the abort-and-settle branch that check
already has.

An ABSENT record falls back to the schedule-level resolution rather than reading as
unscoped. A pre-scope occurrence, or one whose row is gone, must never be treated as
evidence to stop a run — the fallback keeps legacy paused runs behaving exactly as
they do today.

Schedule-wide convergence stays: it keeps the row honest for the UI and for every
check that has no occurrence in hand.

FOLLOWUPS.md now describes the one remaining gap accurately — a deletion inside the
claim window, which needs a distinct policy conflict routed through abort-and-settle
rather than the bare 409 an `inactive` conflict produces.

Tests: three on occurrence-vs-row precedence (the decisive one verified failing
without the lookup), two on what the reservation records, and the resume controller
spec now pins `scheduledFor` in the policy call. Schedules e2e re-run green.

* 🏷️ fix: Tell a Deliberately Unscoped Occurrence From an Unrecorded One

The occurrence fallback added in the previous commit conflated two different
absences. A post-upgrade run that deliberately went unscoped omitted the field
exactly like a row written before the field existed, so both took the fallback —
and a paused unscoped run was then validated against the schedule's CURRENT
project. Under a requirement or a pin added while it sat paused, that admitted a
billed continuation into a conversation satisfying no present policy.

The reservation now ALWAYS records its decision, `null` for unscoped, and the read
reports `recorded` from key PRESENCE rather than truthiness. Only an unknown record
— a pre-scope row, or no row at all — falls back to the schedule-level resolution;
a recorded null is the genuinely unscoped occurrence it says it is, and is refused
once a project becomes required.

The distinction rests entirely on a stored `null` surviving as a present key while a
never-written field stays absent, so that is asserted against real Mongo rather than
assumed: if it ever stopped holding, unscoped runs would silently start being
validated against the schedule's current project again.

Tests: two against mongodb-memory-server (recorded null vs never-written vs missing
row), plus refusal of a recorded-unscoped occurrence under a new requirement and the
preserved fallback for a pre-scope one. Schedules e2e green.

* 🔁 fix: Validate an Initial Scheduled Start Against Its Own Occurrence

The initial-start policy check in `request.js` called `isScheduleLive(..., { policy:
true })` without `scheduledFor`, so it fell back to the schedule-level resolution
even though the run row already carries the occurrence's recorded scope. An
occurrence reserved unscoped, with a pin introduced while its loopback request sat
queued, was therefore admitted against the new pin — producing a billed unscoped
conversation under a requirement it does not satisfy, from an envelope already built
without a project.

`scheduledFor` was already in scope there. Passing it makes the initial start and the
resume validate the same way: against the destination the occurrence itself recorded.
2026-08-21 11:09:04 -04:00
Danny Avila
757fbebc37
🛟 fix: Report Skill Sync Files Whose Paths Cannot Be Mirrored (#15067)
* 🛟 fix: Report Skill Sync Files Whose Paths Cannot Be Mirrored

`discoverSkills` dropped any file whose path failed `isSafeRelativePath`
with no warning, no count, and no record. The skill published, reported
`succeeded`, and was missing files — invisible from the mirrored copy.

Real case: NVIDIA/skills has two such files (spaces in the filename), so
that repository syncs "cleanly" while silently losing them.

Matches what zip import already does — record the file, keep the skill —
and mirrors the existing `skippedSkills` shape into `skippedFiles` /
`skippedFileCount` on the sync status. Dropped files now make a run
`partial`, since a run reporting `succeeded` while dropping content is
the bug.

Only dropped *skills* can still make a run `failed`: a source that
published everything it found is a real mirror even if a file inside one
skill could not come along.

* 🔧 fix: Charge Dropped Files to the Skill That Published Them

Codex review: the up-front accounting counted a skill's unsupported files
whether or not that skill went on to publish. Two consequences — the status
described a skipped skill as published-but-incomplete, and enough failed
skills could consume the 20-entry sample and crowd out drops from skills
that actually published, which is the case the record exists for.

Now recorded at the two points a skill is counted as synced, matching what
`ISkillSyncSkippedFile` already documented ("the skill itself is live").

Also replaces `Array.prototype.at` in the new tests: it is outside this
package's lib target, so `tsc` rejected it even though jest ran it fine.
2026-08-21 03:36:27 -04:00
Danny Avila
6d09a6ccee
🧾 feat: Sibling Task Manifest for Resumed Parent Runs (#15063)
* feat: add bounded subagent orchestration snapshots

* fix: harden orchestration snapshot selection

* fix: close snapshot settlement race

* fix: preserve retry lease uncertainty

* fix: classify bounded sibling leases

* fix: classify captured terminal leases

* fix: enforce snapshot byte budget

* fix: retain terminal lease evidence
2026-08-21 03:35:46 -04:00
Danny Avila
d0f9d5625e
🧵 fix: Close Child-Thread Read and Search-Cleanup Gaps (#15055)
* fix: close child thread read and cleanup gaps

* fix: preserve child search cleanup invariants

* test: complete mocked update result

* perf: parallelize scoped message reads

* fix: close child thread compatibility gaps

* test: expect preserved cleanup failure

* fix: reconcile legacy Meili cleanup markers
2026-08-21 00:44:37 -04:00
Danny Avila
9c9696de8b
🧵 fix: Hide Child Threads From Navigation (#15041)
* fix: hide subagent threads from conversation lists

* fix: exclude child threads from search indexes

* fix: fence child search index cleanup

* fix: preserve private search exclusion markers

* fix: wait for cleanup through Meili client

* fix: preserve lean message update results

* fix: make search cleanup acknowledgments retryable

* fix: complete child search cleanup migration
2026-08-20 18:33:25 -04:00
Danny Avila
868cfbb343
🎡 fix: Scheduled Chat Slot Accounting and Reconciliation Rotation (#15040)
* fix(schedules): harden limits and reconciliation

* docs(schedules): align reconciliation invariants

* fix(schedules): preserve prompt editor contract
2026-08-20 17:18:41 -04:00
Danny Avila
c5276fc63d
⏱️ feat: Run Scheduled Chats Through Durable Agent Triggers (#14939)
* feat: Scheduled Chats — agent-centric scheduled runs creating real conversations

Squash of the full review-hardened branch (PR 14540, supersedes 14373) onto
latest dev, preserving the exact verified tree. History prior to this commit
lived on the pre-squash branch; every invariant below survived 25 Codex review
rounds plus two external audit rounds (R26) with regression tests that fail
without their fixes.

Feature:
- Schedules CRUD + side-panel UI (cadence dialog, run cards, Run Now), roles/
  permissions (SCHEDULES:USE), interface.schedules availability, per-user limits
  and capacity slots, timezone-aware cadence with DST-conservative floors and
  misfire grace.
- Engine: single-process claim/fire loop with leases, loopback POST dispatch
  (signed schedule-fire JWT claims, per-occurrence idempotency key inside the
  route's clientRequestId charset), overlap/balance/capacity/duplicate skip
  policies with auto-disable streaks (too_many_failures, insufficient_balance),
  reconciliation from retained terminal-job evidence, erasure sweep.
- Scheduled runs create real conversations through the resumable agents chat
  path: HITL pauses surface on the card (requires_action), resumes re-apply the
  fire boundary's admission policy (revision fence, enabled, global kill switch,
  SCHEDULES:USE, availability) before continuing a billed generation.

Correctness invariants (the audit surface):
- Settlement discipline: every persistence-producing write happens-before a
  run's terminal outcome write; Stop/complete/pause race through single-winner
  terminal CAS claims (dev's TerminalJobClaim substrate) with retained,
  completedAt-less evidence for scheduled fires plus an owner-intended outcome
  stamp (scheduleOutcome) the reconciler prefers over re-derived success —
  round-tripped through the Redis hash mapper.
- Swallowed generation failures (client error content parts) classify to
  error/skipped_balance instead of success on both initial and resumed paths;
  stale stamps are refreshed evidence-first when persistence plus the Mongo
  outcome write both fail.
- Abort honesty: delivery judged by generation ownership and the CAS's actual
  from-status; republication escalates to the transport's acknowledged variant;
  the Stop route settles only genuinely paused runs.
- Account deletion: one-way barrier (deletionRequestedAt) with auth-cache
  tombstone-before-stamp, boundary rechecks across all auth strategies, quiesce
  of scheduled + interactive work with durable per-stream abort fences
  (positive-evidence acknowledgement only), owner-side finalization markers for
  post-terminal billed writes, deferred-deletion sweep (explicitly ensured
  partial index) that completes cascades autonomously. Remote OpenAI-compatible/
  Responses requests are documented as outside the quiesce and tracked in issue
  14594.
- Store compatibility: finalization markers optional on the legacy IJobStore
  contract with coherent degradation (registration fails -> synchronous title
  fallback; count reads 0).

* fix: fence terminal response persistence from deletion; retain stale-pause evidence

Two blockers from the third external review round.

Terminal persistence visible to account deletion:

- The finalization marker was registered only when post-terminal TITLE work was
  possible, but every persistence-owning terminal CAS opens the same window: the
  claim drops the job out of the active set BEFORE the response save (and the
  background user-message/convo saves), so a deletion quiesce landing there saw
  neither an active job nor a marker and could cascade while the admitted request
  could still recreate messages. Both controllers now register the marker before
  every persistence-owning claim — the fresh-turn path and the HITL resume — and
  release it only after their pending saves (and any post-terminal title) have
  landed, on success and failure paths alike. The TTL bounds a crash.
- settleAbortFence no longer clears a complete/error fence while
  `terminalPersistencePending` is set: terminal at the CAS is not settled while
  the owner is still persisting. The job facade now surfaces the flag.
- The marker trio is REQUIRED by the runtime store contract (assertJobStoreV2
  refuses a store without it at configure time, keeping the failure loud and
  deterministic) while remaining optional on the legacy public IJobStore type for
  source compatibility. The silent degrade path from the previous round is gone —
  it was not deletion-safe.

Stale-pause recovery retains scheduled evidence:

- Three crash/timeout recovery paths — ApprovalLifecycle.failStalePausePersistence
  and the InMemory/Redis stale-pause cleanups — unconditionally stamped
  `completedAt`, putting a scheduled fire's failed-pause error terminal on the
  short completed TTL. A Mongo outage longer than that TTL erased the evidence and
  the reconciler recovered the run as `interrupted` instead of `error`. All three
  now follow the controller-observed path from the previous round: scheduled jobs
  omit `completedAt` (retained-evidence TTL) and stamp the error outcome.

The PR description now explicitly narrows the deletion guarantee for the remote
OpenAI-compatible/Responses paths (tracked in issue 14594).

* fix: deletion-fence marker protocol — generation-scoped, atomic, fail-closed, all terminal paths

One consolidated pass over the finalization-marker mechanism, per the fourth
external review round. The invariant it establishes: NO persistence-owning
terminal CAS runs without a durable, generation-qualified marker covering the
window it opens, and every consumer treats a pending terminal as unsettled.

- Generation-scoped markers. Entries were keyed (userId, streamId), so a
  Stop-superseded generation finishing late could clear the marker its
  replacement registered on the same conversation. Marker fields are now
  qualified by the generation's createdAt; clears must present the same
  identity, and an unqualified legacy clear cannot drop a qualified entry.
- Atomic Redis registration. HSET-then-EXPIRE loses the fresh marker when the
  user's existing hash expires between the two commands (or the process dies
  there); registration is now a single Lua script carrying both.
- Fail closed everywhere. Registration failure (after one retry) now REFUSES
  the terminal CAS instead of proceeding uncovered: the completion claim throws
  into the error path, the error path skips completeJob and leaves the job
  ACTIVE — deletion-visible by itself, recovered by the stale-running reaper —
  and abortJob returns a new retryable `fence_unavailable` failure the Stop
  route answers with 503 and the deletion quiesce treats as an unacknowledged
  stop (fence kept). The previous round's log-and-proceed is gone.
- Every terminal path enrolled. abortJob now owns its window (register before
  the abort CAS, clear in its finally — the Stop route's checkpoint prune and
  partial save run inside beforePublish, between CAS and publication); the
  interactive and background generation-error paths register before their
  completeJob; the resume controller's error finalization registers before its
  completeJob. A lost or thrown claim releases the marker after pending saves
  flush instead of holding the user's deletion behind the TTL.
- Every pending terminal unsettled. settleAbortFence defers on
  terminalPersistencePending for ALL statuses — including `aborted`, whose
  route-side persistence the previous guard missed.

Also: a direct Redis regression for scheduled stale-pause retention (the P2
test gap), and the PR description no longer claims to carry every commit.

* fix: lease-token lifecycle fences — same-generation isolation, admission fence, undelivered-Stop retention, legacy abort enrollment

Fifth external review round; four P1s, handled as the requested consolidated
lifecycle-fence pass.

- Lease tokens. Marker fields were (streamId, createdAt), shared by every
  contender on the same generation — completion and Stop, or two racing Stops —
  so a losing contender's cleanup erased the winner's still-live marker. Every
  registrant now carries a unique lease token in the field and may only ever
  clear its own lease; unqualified legacy clears cannot touch qualified entries.

- Admission fence. Authentication can pass before the deletion barrier goes up,
  and the durable createJob is several async steps later — a deletion quiesce in
  that window saw neither an active job nor a marker and could cascade before
  the admitted request created its job. The controller now registers an
  admission lease and THEN rereads the deletion barrier: the ordering guarantees
  either this request observes the barrier (403, lease released, slot/claim
  cleanup) or the quiesce observes the lease and defers. Held until createJob is
  durable; released on every refusal and initialization-error path. Fail closed
  when the lease itself cannot be registered (503 retryable).

- Undelivered-Stop retention. abortJob released its lease in a finally even when
  delivery AND publication had provably failed — the job reads terminal
  (invisible to active-set scans), a user Stop writes no durable abort fence,
  and the remote owner keeps generating and will persist its abort-catch writes
  whenever the signal finally lands. The lease is now retained in exactly that
  case, and each resignal attempt heartbeats a fresh lease so the fence outlives
  the TTL for as long as delivery is still being driven. The abort-winning
  turn's own loser-side pending saves are additionally fenced in the controller
  catch (best-effort — those writes are already in flight).

- Legacy abort enrollment. abortMiddleware (assistants abort route fallback for
  non-assistants endpoints) awaited abortJob and then spent usage and saved the
  stopped response AFTER the abort's lease was released. Both writes now run
  inside `beforePublish`, between the abort CAS and publication, covered by the
  same lease as every other abort.

Barrier tests, each verified to fail without its fix: same-generation lease
isolation (store), racing two-Stop loser cleanup (manager, stale-read forced
CAS race), undelivered-Stop lease retention, resignal heartbeat, admission
refusal with lease-before-reread ordering plus release-on-durable-create, and
legacy-abort persistence inside beforePublish.

* fix: heartbeat-backed owner-lifecycle leases close the settlement handoff races

Sixth external review round: the remaining P1 interleavings were one structural
problem — lease handoffs that were not atomic — resolved as the requested
consolidated lifecycle-lease pass.

- Quiesce reads leases BEFORE the active-job scan. The admission-lease -> durable
  -job handoff is only atomic against a reader in the OPPOSITE order of the
  writer: writers hold the lease strictly until the job is active-set visible,
  so leases-first shows every interleaving either the lease or the job.
  Jobs-first allowed a request to create its job after the scan and release its
  lease before the count — hiding both, cascading, and letting the new
  generation persist into a deleted account.

- The abort acknowledgement is fenced by the owner-lifecycle lease. Redis ACKed
  the moment the owner's AbortController tripped; the stopping side released its
  lease on that ACK while the owner's asynchronous abort-catch persistence was
  still ahead. The transport now awaits a manager-installed pre-ACK hook that
  registers a DETERMINISTIC owner lease (exactly one owner exists per
  generation, and determinism is what lets the signal-time registrant and the
  owner's catch-side release agree across processes) before the acknowledgement
  is persisted or published; an owned same-replica abort bridges to the same
  lease before tripping its local controller. Both generation-owner catches
  (fresh turn, resume) release it once their writes land.

- A failed replacement handoff no longer orphans the predecessor. The atomic
  replacement removes it from active storage, and an unconfirmed handoff
  terminalizes the replacement too — leaving nothing a quiesce could discover
  while the predecessor's provider may still be generating. Its owner lease is
  now retained at the point the receipt fails delivery; the owner replica renews
  it through the pre-ACK fence when the signal finally lands.

- Leases HEARTBEAT while held. The five-minute store TTL only bounds a crashed
  holder; live persistence — a stalled save, a long deferred title — must never
  outlive its own fence. holdUserFinalization registers and renews every minute
  until released; the controllers' completion/error/admission leases all hold.
  The undelivered-Stop retention moved to a deterministic `stop` lease that
  every resignal attempt renews and the first successful one clears (no more
  opaque leases accumulating to TTL), and a THROWN abort transition releases the
  contender lease instead of leaking it.

- The user-document abort-fence mutations now invalidate the auth user-doc
  cache, matching every other user-doc write.

Barrier tests, each verified to fail without its fix: quiesce lease-scan
ordering (plus the observed-lease defer), pre-ACK fence ordering at the
transport, owned-abort owner-lease bridging, replacement-handoff predecessor
retention, held-lease heartbeat past the TTL, and failed-then-successful
resignal reaping the retained stop lease.

* fix: one manager-owned owner-lease span across every abort delivery path

Seventh external review round; four lifecycle-fence gaps, closed by making the
owner-lifecycle lease a single manager-owned, heartbeat-held span.

- Fail-closed acknowledgements. The pre-ACK hook registered a one-shot lease and
  the transport ACKed even when it failed; a same-replica owned abort likewise
  swallowed registration failure. The hook now acquires a HELD owner lease
  (heartbeat until the owner's catch releases it via releaseOwnerLease) and a
  rejection SUPPRESSES the acknowledgement — the stopping side stays retryable
  behind its retention lease, and every resignal re-drives the handler. The hook
  also stopped gating on `job.createdAt === generationId`: during a replacement
  handoff the store holds the replacement while the abort targets the
  predecessor, and that gate silently skipped exactly the generation being
  acknowledged (the store job is owner identity, never a generation gate). A
  local owned abort acquires the same held lease before tripping its provider;
  post-CAS the trip cannot be withheld, so acquisition failure downgrades
  delivery and the retention handoff keeps the user fenced.

- Committed-but-lost-reply disambiguation. A thrown abort transition released
  the contender lease as if nothing had happened, but a Lua CAS can commit and
  lose its reply — an aborted job invisible to active-set scans whose provider
  was never signalled, with no fence left. The throw path now re-reads the exact
  generation: only a job still live under the caller's identity proves no
  commit; committed or ambiguous outcomes hand the fence to the deterministic
  stop lease (kept on the contender lease if even that fails) before rethrowing.

- Replacement handoff covered end to end. A LOCAL replacement abort acquires the
  predecessor's held owner lease before the trip (failure reports the receipt
  undelivered, engaging retention). Failed-handoff retention is no longer a
  swallowed one-shot: it heartbeats with the durable acknowledgement proof as
  its renewal predicate — acquisition failures keep retrying for as long as the
  fence is needed, and the retainer stands down (without clearing the shared
  field) once the remote owner ACKs and thereby holds its own lease.

- A LOCAL resignal delivery hands off to the owner lease BEFORE clearing the
  retained stop lease, and keeps the stop lease when that handoff fails.

Barrier tests: hook rejection suppressing the ACK, commit-then-lost-reply
retention with its provably-uncommitted counterpart, local-resignal owner
handoff ordering, pre-ACK owner lease held past the store TTL until release
(and provably stopped after), and failed-handoff retention retrying on its
heartbeat — verified fail-before/pass-after by stashing the fixes.

* fix: finish scheduled chat lifecycle hardening

* fix: close scheduled chat review follow-ups

* fix: generation-fence abort recovery evidence

* test: wait for settled approval tool output

* test: preserve scheduled init reconciliation option

* refactor: rebuild scheduled chats on durable agent triggers

* test: reset MCP cache mock between cases

* test: isolate scheduler startup in server specs

* fix: harden scheduled run lifecycle

* fix: normalize schedule capacity conflicts

* test: type schedule collision fixture

* fix: re-fence scheduled resume and expiry

* fix: fence scheduled resume handoffs

* fix: release superseded manual schedule leases

* fix: release failed run-now claims

* fix: release superseded engine claims

* fix: repair schedule dialog interaction and rework its form

The agent picker was unusable: ControlCombobox portals its popover to the
body by default, which lands it outside the dialog's Radix focus trap. Clicks
passed through it, it could not be tabbed into, and the trap fighting Ariakit
for focus locked the page up on selection. The prop is documented for exactly
this case — pass `portal={false}` and give the dialog `overflow-visible`, as
ProjectButton already does. The time and day dropdowns defaulted the same way.

Alongside that:

- Extract the agent builder's instructions editor (special-variable menu plus
  expand-to-fullscreen) into a controlled `VariableEditor` and use it for the
  schedule prompt. Insertions now route through `onChange`, so react-hook-form
  sees them — the schedule PATCH is built from `dirtyFields`, and a `setValue`
  that skipped dirty tracking would have dropped an inserted variable silently.
- Replace the hand-rolled frequency buttons with the shared `Radio`. They marked
  the selection with `bg-surface-hover` on an outline button whose hover is the
  same token, so the selected option was indistinguishable from a hovered one;
  `Radio` is also a real radiogroup rather than four `aria-pressed` toggles.
- Wrap the fields in a real `<form>` and associate the footer button by id, so
  Enter submits. Group the frequency, day and time controls in fieldsets.
- Add placeholders for name and prompt, match the textarea fill to the other
  fields, and label the hourly case as minutes past the hour.
- Widen the dialog to `md:max-w-3xl` and pair name with agent so the form fits
  without scrolling on desktop.
- Move scheduled chats below skills and above prompts in the side nav.

The new dialog spec fails when the portal fix is reverted.

* test: cover scheduled and subagent deletion drains

* refactor: own the form-control appearance in the client primitives

Addresses the codex finding on ScheduleDialog: a feature-local `FIELD_CLASS`
restated the `Input` primitive's border, radius, height and background so it
could be pasted onto the schedule dropdowns, leaving those controls with no
connection to the primitive they were imitating.

Move that appearance into `packages/client/src/components/Field.ts` as the
single source `Input` and `Textarea` now compose, and give `Dropdown` and
`ControlCombobox` a `variant="field"` that applies it. The schedule dialog
passes the variant and carries no class strings of its own.

This also repairs a break the dev merge would otherwise have introduced: the
newer `Dropdown` splits `className` (wrapper) from `triggerClassName`, so the
old pasted classes would have landed on the wrapper and left the triggers
unstyled.

The semantic-token guard now watches the shared module and asserts each
primitive still composes it, which covers more than the two files it read
before.

* fix: keep an explicit schedules disable from becoming an opt-in

`use` is two things at once for a dual-purpose runtime interface field: a
permission bit, which DB overrides strip, and the runtime disable signal that
`getLimits` reads. Stripping it from `{ use: false, maxPerUser: 2 }` leaves an
object, and `getLimits` treats any object without `use: false` as enabled — so
an admin override written to stop scheduled billing for a role or user started
it instead.

Collapse an explicit disable to the boolean form before the strip, on both
paths that accept it: the `interface.schedules` field patch, which admitted the
object wholesale because bare runtime paths deliberately bypass the permission
gate, and the overrides merge, which reached the composite-field branch and
kept `maxPerUser`. Objects that only narrow limits are untouched, so a
principal can still be given a smaller cap.

Both regressions fail without the normalizer.

* fix(schedules): Wave A — null-balance CAS, atomic paused-card clear, clustered erasure sweep

Slice 1 (thread r3804518381): route the existing-null balance initialization
through a { user, tokenCredits: null } compare-and-set instead of a blind $set,
so a concurrent initializer/charge landing between the preflight read and the
write is never handed back its spent starting balance. On a CAS miss the
preflight re-reads the winner. The absent-record $setOnInsert path and the
credited-record refill-config sync are unchanged. Adds the initializeNullBalance
adapter (no upsert) and regression coverage for winner/miss/sync cases.

Slice 2 (thread r3804518388): updateScheduleById now drops a `requires_action`
lastRun projection atomically with the configRevision bump. Any pause present at
edit time was projected under the pre-edit revision and can never be replaced by
its own revision-fenced terminal outcome — a disabling edit would strand the
card on "Needs approval" forever. Implemented as classic-operator CAS branches
(DocumentDB rules out a conditional pipeline $unset), fenced on the card STILL
being the pause so a terminal outcome or newer occurrence that races in is
preserved. Terminal history survives untouched.

Slice 4 (thread r3803826204): expose initializeScheduleErasureSweep from the
schedule runtime facade and start it in every clustered (experimental) worker
after Mongo is up. It re-drives eraseScheduleIfDrained for soft-deleted rows so
a hidden prompt cannot outlive its drain when the delete/erase-on-settle
attempts miss. It arms nothing else and never infers owner death from a
process-local missing job (isTopologySafeToArm gates that).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq

* fix(schedules): Wave B.1 — reversible account-deletion schedule suspension

Thread r3804518383. Account-deletion quiesce marked every schedule `deleting`,
disabled it, and cleared nextRunAt destructively. When a later cascade step (or
the drain itself) failed, the controller cancelled the user-deletion fence —
restoring the user — but the schedules stayed `deleting` and were erased by the
sweep, silently losing all of a live user's scheduled prompts.

Replace the destructive marking with a REVERSIBLE, token-fenced suspension:

- suspendUserSchedulesForDeletion(userId, token) snapshots each schedule's prior
  enabled/nextRunAt under a per-attempt token, then fences firing (disable, clear
  nextRunAt, rotate claimToken). It never sets `deleting`, so a suspended row is
  not erasure-eligible. Snapshotting reads then bulkWrites (a classic update
  cannot copy field values under DocumentDB), fenced per row so an already-
  suspended/soft-deleted/edited row is left alone; idempotent per token.
- restoreUserSchedulesFromDeletion(userId, token) reverses it, re-enabling and
  re-arming only rows still carrying the exact attempt token and not independently
  deleted — so an owner-deleted or newer-attempt-suspended schedule is never
  resurrected.
- deleteUserController generates the attempt token, passes it to quiesce, and on
  any failure that cancels the user-deletion fence restores the suspended rows. A
  successful deletion hard-deletes them (and their snapshots) via the existing
  cascade and never restores.

Adds a `deletionSuspension` embedded field (excluded from the wire schedule),
data-method regression tests (suspend/restore/fence/idempotency/no-resurrect),
and controller tests (restore on drain-false and post-quiesce cascade failure,
no restore on success).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq

* fix(schedules): Wave B.2 — wire the interactive Stop persistence protocol

Thread r3804255932. The durable Stop primitives (requestRunAbort 'stop',
getScheduleRunAbortState, markRunAbortPersisted) already existed but production
only used requestRunAbort(..., 'deletion'). An interactive Stop flipped the job
to `aborted` and then persisted its partial message + checkpoint inside
`beforePublish`, without ever stamping the schedule Stop or acknowledging it —
so reconciliation, the generation owner, or a concurrent schedule/account
deletion could terminalize the run and release its capacity (and erase data)
mid-write.

Wire the request -> persist -> acknowledge -> settle barrier through the
schedule runtime (the route never touches raw Mongo):

- Expose beginScheduledStop / acknowledgeScheduledStopPersistence on the service.
- The abort route stamps the Stop BEFORE signalling abortJob (a serialized
  'in_progress' loser returns 409 STOP_IN_PROGRESS without a second abort),
  acknowledges only after beforePublish persistence succeeds, and on a
  persistence failure leaves the barrier unresolved so the run stays preserved
  (client retries; stale-owner timeout is the bounded recovery). A failed abort
  releases the stamp it placed so a replacement/retry is never blocked through
  its predecessor.
- recordScheduleOutcome (the owner settlement path) now waits, bounded, for the
  Stop acknowledgement before terminalizing; a resolved/non-stop/stale marker
  proceeds immediately. The paused Stop settles only after its own ack.

Adds service-layer barrier tests and route-level ordering/persistence-failure/
in-progress tests; the data-layer serialization is already covered.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq

* fix(schedules): Wave C.1 — reconcile durable trigger delivery with the reservation

Threads r3803826192 (manual limiter) and r3804255924 (PII/moderation), plus the
independently-found long-Retry-After race. fireSchedule reserves a `started` run
and a global capacity slot BEFORE the durable trigger delivery reaches the chat
route, where an interactive limiter (manual Run Now), PII, or moderation can
reject it before any generation job exists — dead-lettering the delivery while
the run sat `started` until the 30-minute orphan sweep mislabeled it interrupted.
And a valid delivery deferred by Retry-After (up to 24h) could be orphan-settled
and have its capacity released, then fire anyway.

Translate durable delivery state into the schedule outcome:

- Store the deterministic trigger deliveryKey on the ScheduleRun reservation
  (computed from the envelope BEFORE enqueue, so an ambiguous commit still has it).
- Add a getTriggerDelivery engine dep (wired to the merged trigger service's
  getDelivery) that reads the durable delivery by key.
- Schedule reconciliation, for a jobless `started` run: staging/pending/leased →
  admission is live, never orphan; dead → record `error` from the durable
  lastError and release capacity promptly (no 30-minute wait), through the
  ordinary outcome/auto-disable path; succeeded or no record → the existing
  legacy orphan policy (interrupted only past the cutoff); a delivery lookup
  failure defers rather than orphaning a possibly-live delivery. Limiter/PII/
  moderation middleware writes no schedule state.

Adds reconcile state-mapping tests (dead/pending/leased/staging/succeeded/none/
lookup-failure) and a fire test that the reservation's deliveryKey equals the
enqueued delivery's idempotency key.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq

* fix(stream): Wave C.2 — durable retry/ack for terminal host lifecycle actions

Thread r3804518375. Approval expiry won the `requires_action → aborted` CAS and
then invoked the host hook best-effort: `runApprovalExpiredHandler` swallowed a
failure, and because later sweeps enumerate only `requires_action` jobs, the now-
aborted job was never offered again. In the clustered entrypoint (no schedule
reconciler) the ScheduleRun stayed `requires_action` and its retained job
persisted indefinitely.

Make the host lifecycle work durable rather than schedule-specific:

- Add a generic `terminalHostActionPending` marker, set ATOMICALLY in the same
  terminal transition (ApprovalLifecycle.expireWithIdentity), only when a host
  adapter is installed.
- Retain and index such jobs: both stores keep them out of terminal reaping and
  expose getTerminalHostActionJobs(); Redis adds a set + extended (24h-bounded)
  TTL, in-memory a bounded 24h retention so a permanently-failing hook cannot leak.
- The manager clears the marker only after the adapter acknowledges success,
  fenced by generation identity (clearTerminalHostAction), so a replacement
  generation can neither clear nor execute its predecessor's action.
- cleanup()/expireStaleApprovals() enumerates unacknowledged terminal host actions
  across restarts and replicas and retries the idempotent hook; the relay only
  re-invokes while the marker is unacknowledged, so a successful ack prevents
  duplicate work. Store-won expiry marks it too, so a loser-replica relay still
  crosses the hook.
- Terminal SSE notification continues regardless of host-hook outcome.

Covers in-memory behavior (retry after failure, restart/other-replica retry, ack
prevents duplicates, identity fence, terminal notification on failure, no marker
accumulation for non-scheduled jobs) and updates the Redis cluster-membership
contract test for the new index.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq

* style(schedules): satisfy import sorting in fire.ts and fire.spec.ts

CI "Static checks" failed on IMPORT_SORT for the two files Wave C.1 added imports
to (the AgentTriggerEnvelope type import and getAgentTriggerIdempotencyKey).
Applied scripts/sort-imports.mts to exactly those files — imports-only reordering,
no behavior change. Other files reported by a repo-wide check are pre-existing on
dev and deliberately left untouched so this PR is not widened.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq

* fix(schedules): repair the deletion CLI and order restore before the fence release

Addresses three findings from the fresh Codex review.

P1 — config/delete-user.js called methods.disableUserSchedulesForDeletion, which
Wave B removed in favor of suspendUserSchedulesForDeletion. The file is
@ts-nocheck and its spec mocked the removed name, so neither typecheck nor tests
caught it; the real CLI would throw a TypeError before deleting anything and then
only unwind the fence. The CLI now uses the tokenized protocol: it mints a
suspension token, suspends with it, and restores that exact attempt's rows in its
finally block when the deletion does not commit. Its spec mocks the real methods,
so the breakage can no longer hide.

P2 — both the HTTP controller and the CLI released the user-deletion fence BEFORE
restoring schedules. That fence is what refuses new schedule writes/claims, so the
gap let an owner PATCH edit a still-suspended row and have its enabled/next-run
state overwritten by the older snapshot, and let a second deletion attempt
re-suspend under a new token — making the first restore a no-op and stranding the
disabled snapshot permanently. Restore now runs first, while writes are still
fenced.

Hardening for the same defect class across a crash: suspendUserSchedulesForDeletion
now ADOPTS an existing suspension's snapshot when re-suspending a row abandoned by
an earlier attempt, instead of re-capturing the row's current (already-suspended)
state. Without this, an attempt that died before restoring would have its
successor snapshot "disabled, no next run" and permanently strand the schedule.

Tests: CLI restore-before-fence ordering and no-restore-on-success; the same
ordering assertion on both controller post-quiesce failure paths; a data-method
regression that a second attempt adopts the abandoned snapshot and restores the
original enabled/next-run state.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq

* fix(schedules): converge dead deliveries in topology-safe maintenance

Codex finding: the `dead` delivery mapping added in Wave C.1 lives only inside
startScheduleEngine's reconciler, but the clustered entrypoint arms no engine — it
runs erasure-only maintenance. A delivery queued before a restart into clustered
mode and then rejected before generation creation (interactive limiter, PII,
moderation) dead-letters while its ScheduleRun stays `started`, holding a global
capacity slot indefinitely for an ordinary non-deleting schedule.

Add a dead-delivery convergence pass to the erasure sweep, so every topology that
runs schedule maintenance settles it. The pass is POSITIVE-EVIDENCE-ONLY and is
therefore safe where absence-based reconciliation is not: a `dead` delivery is
durable shared state proving no generation owns the reservation. It settles only
when the job is confirmed absent or identity-mismatched (an identity-matched job
still owns the run), defers on an unknown job lookup, on an in-flight abort, and
on an in-flight resume hand-off, ignores legacy reservations with no deliveryKey,
and applies a short grace so an accepted delivery still creating its generation is
never settled mid-handoff. Auto-disable policy is deliberately left to the armed
engine; this path records the failure and frees the slot.

Deliberately does NOT touch api/server/experimental.js — the clustered entrypoint
already starts this sweep, so the convergence arrives through the existing
initializer and the shared-file footprint stays as-is.

Tests: settles a dead delivery as error under an explicitly UNSAFE topology,
leaves live deliveries alone, never settles under an identity-matched running
generation (delivery is not even consulted), defers an in-flight abort, and
ignores a reservation with no deliveryKey.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq

* fix(stream): refresh host-action retention on each retry attempt

Codex finding: unacknowledged terminal host-action evidence was capped at the
24h pause TTL, so a host dependency (Mongo) unreachable for longer than that let
the Redis key — and its pending marker — expire with no generation-fenced
acknowledgement, stranding the ScheduleRun where no reconciler is armed.

Measure retention from the LAST retry rather than from the terminal transition:
enumerating a pending host action IS the retry attempt, so both stores refresh
its retention as they hand it to the hook (Redis re-EXPIREs the job key; the
in-memory store stamps terminalHostActionRefreshedAt and bounds from it). Evidence
therefore survives as long as some replica is still actively retrying, while a
deployment that stops sweeping entirely still lets it age out — so this does not
reintroduce the unbounded leak the cap existed to prevent.

Test: after a failed hook, a later cleanup pass keeps the marker pending and moves
its retention basis forward.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq

* fix(schedules): defer settlement when the Stop barrier times out

Codex finding: waitForStopPersistence returned after its 5s poll budget even when
the Stop was still fresh and unacknowledged, and recordScheduleOutcome then went
straight on to terminalize the run — releasing its capacity, deletion, and erasure
barriers while beforePublish may still have been writing. Slow checkpoint cleanup
is indistinguishable from a dead route on that signal, so the timeout was being
treated as if the barrier had been satisfied.

The poll budget now means "undecided", not "clear". On timeout with a fresh,
unacknowledged Stop the barrier DEFERS: recordScheduleOutcome returns false
without recording, leaving the run active/preserved. Settlement then happens
either when the route acknowledges, or once the existing stale-owner cutoff
(ABORT_OWNER_PRESUMED_ALIVE_MS) authorizes a later attempt — which the loop
already treats as clear-to-settle. Callers with durable retry (the approval-expiry
host action, reconciliation) re-drive it, so a deferral converges rather than
stranding the run.

Test: a fresh Stop that never acknowledges within the budget reports not-settled
and records no outcome.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq

* fix(schedules): require definite delivery failure, extract its message, converge deferred Stops

Third Codex round. Three findings in code from this closeout, plus one pre-existing
P1 that is a one-line operator-safety fix.

P1 — dead-delivery settlement demanded too little evidence. `dead` does not prove a
request was rejected: the trigger host marks response timeouts and invalid success
responses `certainty: 'ambiguous'`, and the engine dead-letters those once retries
are exhausted. The erasure sweep treated every dead letter as positive evidence, so
an ambiguous one sitting over a generation a peer had accepted could terminalize the
run and release its capacity mid-flight. It now settles only on a DEFINITE rejection,
unless job absence is deployment-authoritative (safe topology), where the
confirmed-absent job is itself the evidence.

P1 — `lastError` is an `AgentTriggerDeliveryFailure` object, not a string. A
duplicated local interface declared it `string` (against CLAUDE.md's no-duplicate-
types rule), so both the sweep and the engine reconciler passed the object into the
String-typed run/schedule `error` fields; Mongoose would reject the cast, the per-row
catch would swallow it, and the run would keep its global capacity slot. The dep type
now reuses the canonical `AgentTriggerDeliveryFailure` and both call sites pass
`.message`. Re-typing immediately surfaced a stale test that had asserted a string.

P1 (pre-existing) — the base-config global stop is honored in `getLimits` via
`isRuntimeDisabled` rather than a literal `=== false`. The stop has two shapes, and
deepMerge turns base `{ use: false }` plus a principal override of `true` into
`{ use: true }`, so the literal check reported the feature enabled and Run Now
dispatched straight through fireSchedule, bypassing the operator's emergency stop.
Now the same predicate the engine gate already uses.

P2 — a Stop whose settlement DEFERRED past the poll budget had no convergence path
where no reconciler is armed. `acknowledgeScheduledStopPersistence` now optionally
re-drives the terminal outcome once the barrier clears; `recordRunOutcome` is
match-guarded and idempotent, so an owner that already settled makes it a no-op. The
abort route passes it for a running generation; a paused job still settles explicitly.

Tests: ambiguous dead letters refused under unsafe topology but settled when absence
is authoritative, definite rejections settled either way, the failure message carried
through, and the abort route's re-drive present for running / absent for paused.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq

* fix(schedules): converge terminal runs in clustered workers, retry suspension restore

Fourth Codex round on 74f7af8. Both findings are in code from this closeout.

P2 - a clustered worker never settled a run whose generation finished but whose
outcome write failed. `recordScheduleOutcome` retries three times and then returns
false; the owner honors that by PRESERVING the terminal job as the only surviving
evidence, and the armed engine's reconciler replays exactly that. The clustered
entrypoint arms no engine, and its sweep only covered deleting schedules
(settleAbandonedRuns) and dead deliveries - an identity-matched job was skipped
outright. So an ordinary live schedule's run stayed `started`, held its GLOBAL
capacity slot, and kept a preserved job that carries no `completedAt` and is
therefore invisible to the store's finished-job sweep: both leaked until store
expiry.

The live-schedule pass now also converges from a retained terminal job, mirroring
the reconciler branch it stands in for: honor the owner's stamped outcome over the
generic status (so a balance refusal still walks its streak rather than resetting
it), clear the reserved conversationId when the generation never emitted its
created event, and delete the retained job only AFTER the outcome write is durable.

This stays positive-evidence-only and safe in every topology. Presence, not
absence, is the evidence: an identity-matched job is authoritative wherever it is
observed - a shared store shows the real generation, a process-local store can
only be showing this process's own - which is why it needs no
canInferOwnerDeathFromMissingJob fence, unlike the absence-based paths. The
in-flight abort/resume fences still defer, so an `aborted` job cannot settle a run
whose owner may still be persisting. Both cases now share one pass over one window
rather than two, and `retainedOutcome` moved to types.ts so the sweep reuses the
engine's mapping instead of duplicating it (and stays independent of the engine).

P2 - a failed restore stranded a live account's schedules. Cancelling an account
deletion restores the suspended rows while the deletion fence is still armed and
then releases that fence; nothing re-drives the restore afterwards, so one
transient write failure left the user with silently disabled, next-run-less
schedules. The restore is now retried at the single choke point both the HTTP
controller and the CLI share. Retrying is safe because each attempt re-reads only
the rows STILL carrying the token: a partially-applied unordered write converges
on exactly the stragglers, and a fully-applied one finds nothing.

The fence is still released when every attempt fails, deliberately: retaining it
would refuse the live account's schedule writes AND make beginAgentTriggerUserDeletion
report `in_progress` forever, blocking the retry that is the convergence path (a
later attempt adopts this snapshot, so its cancel restores these exact rows). Both
callers now log the user id and suspension token so the state stays recoverable by
hand if that never happens.

Tests: retained terminal job settled and its evidence released, stamped outcome
preferred over the generic status, stamped failure reason carried, reserved
conversationId cleared for a never-created conversation, identity-mismatched
terminal job ignored, abort-in-flight deferred; restore retried past a transient
write failure and converging on a partially applied restore.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq

* fix(schedules): converge unprojected pauses and release replayed bookkeeping jobs

Two of the four findings I flagged as pre-existing to my closeout commits. Both are
in code this PR introduces, so merging would have shipped them; both are the same
capacity/evidence-leak class the last four rounds have been closing. The remaining
two (slotless rows escaping the per-user cap, and the non-rotating `started`
reconciliation bucket) are genuinely latent and deliberately left alone.

An UNPROJECTED PAUSE held a global capacity slot forever. The pause projection is
what moves a run row off `started`; `recordScheduleOutcome` already retries it, but
the request controller discarded the result, so three failed attempts were dropped
silently. The armed engine's reconciler replays that state, but the clustered sweep
did not: a paused job is not terminal, so the retained-job path returned without
settling, and the dead-delivery path never inspects an identity-matched job at all.
The row stayed `started` with no cutoff that would ever clear it.

The call site now surfaces the failure, and the sweep converges it, mirroring the
reconciler's pause branch: project `requires_action` (which frees the slot) but do
NOT release the job's evidence — unlike a terminal job it is still live, awaiting an
approval. The resume hand-off fence still defers, so re-projecting cannot release a
slot a continuation just claimed.

The BOOKKEEPING REPLAY pass leaked its retained job. A run reaches that pass only
because its owner crashed before bookkeeping — which is also before it could release
the job it retained for exactly this recovery. The active-run pass clears its own;
this one never did, and a preserved job is deliberately kept WITHOUT `completedAt`
so the store's finished-job sweep cannot reap it early, so nothing else ever would.
It now clears after `finalizeBookkeeping` succeeds — identity-guarded, a no-op when
no job is retained, and skipped entirely when the replay itself failed, since the
retained job is the only surviving evidence in that case.

Tests: pause projected and the slot freed while the live job's evidence is kept,
paused job deferred during a resume hand-off, retained job released once replayed
bookkeeping is durable, and retained job kept when the replay fails.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq

* fix(schedules): fence the clustered pause replay against a concurrent resume claim

Codex round 5, on code I added in f735341. The finding is correct and the exposure
is one I introduced.

The pause replay lives in a sweep that runs in EVERY clustered replica, so several
sweepers can observe the same unprojected pause. Its guards were all derived from an
in-memory row SNAPSHOT: hasResumeHandoffInFlight reads the snapshot's
resumeClaimedAt, and recordRunOutcome's pause branch matched any row currently in
`started`/`requires_action` with no fence of its own. The race: sweeper A projects
the pause and frees the slot; the owner's approval then claims a fresh one
(markRunResumeClaimed takes the row to `started` WITH resumeClaimedAt in one write);
sweeper B, still holding the pre-projection snapshot, passes its hand-off check and
replays — `$unset: { capacitySlot, resumeClaimedAt }` under a continuation that is
already running. The run reverts to `requires_action` while its generation proceeds
outside global capacity.

The engine's reconciler makes the same call and has the same snapshot-derived guard,
but v1 arms exactly one engine, so it has no concurrent racer; the sweep is the first
thing to run this transition in parallel. Left the engine alone rather than widening
the change: its `requires_action` re-affirmation is deliberate and single-writer.

Fixed where the race is, in the write itself. `recordRunOutcome` takes an optional
`requireNoResumeClaim`, which adds `resumeClaimedAt: { $exists: false }` to the pause
filter, and the sweep sets it. Because the stamp is written in the SAME update that
moves the row to `started`, its absence is atomic proof no resume owns the row. The
flag is deliberately NOT set by the generation owner: its own re-pause legitimately
clears the stamp as the hand-off's completion signal.

The fence cannot block the recovery it exists to enable: markRunResumeClaimed only
matches `requires_action`, so a genuinely stuck `started` row can carry no resume
claim — it is in fact blocking its own approval until this replay frees it.

Tests: a replay from a stale snapshot leaves a resume-claimed row's status, slot, and
claim stamp intact, while a stuck row with no claim still recovers; and the sweep is
asserted to send the fence.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq

* fix(schedules): fence stale resume claims by age, not existence

Codex round 6, on the fence I added in ee00c60. Correct again, and the defect is
the mirror image of the one it fixed.

The fence was an EXISTENCE check (`resumeClaimedAt: { $exists: false }`) while its
caller's guard is a FRESHNESS check (hasResumeHandoffInFlight, bounded by
RESUME_HANDOFF_STALE_MS). They agree while a claim is fresh and disagree once it is
abandoned: a worker that dies after markRunResumeClaimed takes the row to `started`
and stamps resumeClaimedAt — but before the continuation resumes or
releaseRunResumeClaim rolls it back — leaves the stamp set forever. Past the bound
the sweep correctly stops deferring and tries to recover the row, but the write
rejected it purely because the field still existed. The row stayed `started` holding
its global capacity slot, and its approval was unresumable for good, since
markRunResumeClaimed only matches `requires_action`. That is exactly the stuck state
this replay exists to clear, so the fence had reintroduced it for the crashed-resume
case.

`requireNoResumeClaim: boolean` becomes `resumeClaimStaleBefore: Date`, and the
filter matches a row with no claim OR a claim older than that cutoff. The sweep
passes the SAME bound its in-flight check uses, so the two can no longer disagree.
A genuinely racing claim is by construction fresh — it is created after the sweeper's
snapshot — so the race from round 5 stays closed.

Tests: a row whose resume claim was abandoned by a dead worker now recovers (status,
slot and stamp all cleared), alongside the existing two — a fresh claim still repels
a stale-snapshot replay, and an unclaimed stuck row still recovers.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq

---------

Co-authored-by: Claude <noreply@anthropic.com>
2026-08-20 11:51:30 -04:00
Danny Avila
986b1218ac
🔓 fix: Unblock Detached Subagent Preparation (#15016)
* fix: retain detached subagent execution lifetime

* test: verify detached timer lifecycle

* fix: settle Meilisearch query middleware

* fix: preserve detached child failure diagnostics

* test: select active preparation watchdog

* test: keep watchdog assertion target-compatible
2026-08-20 11:14:27 -04:00
Danny Avila
5e3c680761
🪃 feat: Wake Parent Agents on Child Completion (#14975)
* feat: wake parent agents on child completion

* wip: harden child completion wakeup lifecycle

* fix: close the completion-wakeup static failures

Type the durable-claim store fixture, the continue-envelope test helper,
and the terminal message's task metadata so the wakeup suites compile
against the shapes they actually exercise. Replace `Array.prototype.at`,
which the package target library does not provide.

Capture the prepared child thread in a non-optional local before the
provider callback closes over it, and narrow the trigger envelope itself
on `mode === 'continue'` rather than a separately copied mode, so reading
the continue target is sound. Lift the parent-message fallback out of a
nested ternary into a named resolver.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H1cCMDrTWaRNkmtKjpWELZ

* test: cover the active-predecessor admission fence

The Redis job-creation call gained a thirteenth scalar argument, so the
spec helper reconstructed the HSET pairs one slot early and rebuilt an
invalid job hash; three creation tests failed on that alone.

Give the fence itself direct coverage in both store adapters, which it
had none of despite deciding whether an automatic continuation may
replace a live parent turn. Each proves a running and a requires_action
predecessor are refused with the state a controller needs for a finite
409, that an absent or settled predecessor is admitted, and that an
ordinary user turn without the policy still replaces its predecessor.
The Redis case also asserts a refused continuation leaves the parent's
durable job and chunks untouched.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H1cCMDrTWaRNkmtKjpWELZ

* fix: harden completion wakeup rollout and claims

* fix: close completion wakeup race windows

* test: keep the child store fixture exact

* fix: close final subagent wakeup gaps

* fix: preserve ambiguous completion claims

* fix: release pre-admission wakeup claims

* fix: stabilize subagent completion recovery

---------

Co-authored-by: Claude <noreply@anthropic.com>
2026-08-19 12:18:31 -04:00
Danny Avila
259f1e0c32
🛰️ feat: Route Live Subagent Controls Across Replicas (#14971)
* feat: route live subagent controls across replicas

* fix: initialize task routing in cluster workers

* fix: harden cross-replica task routing

* fix: expire routed task owners independently

* fix: close cross-replica routing edge cases

* fix: bound owner refresh and close routed cancellation gaps

Refresh owned task registrations in bounded parallel batches so a full
heartbeat pass stays well inside the 30-second directory lease instead of
serializing one Redis EVAL per registration.

Route conversation-deletion cancellation through a dedicated owner-side
scope operation. The owner applies the deletion predicate to its complete
local task set, so a scope holding more children than the model-facing
list cap no longer leaves live executors running after their parent is
removed.

Key a consumed claim's retained response by its operation rather than by
one caller's correlation id, so a later poll recovers a terminal result
whose responses were all lost. Live claim statuses stay uncached so a
poll always observes the task's current state.

Type the model-facing `maxLength` bounds with a narrow local string
schema; the SDK's JsonSchemaType does not declare the keyword, and the
runtime checks continue to enforce the same limits.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H1cCMDrTWaRNkmtKjpWELZ

* fix: retain claimed results apart from control replays

A consumed claim is the only routed response whose loss destroys data, so
it no longer shares one bounded cache with control replays that unrelated
command traffic can evict. Claims are retained under their own budget, and
the requester acknowledges a result it received so the owner releases the
copy immediately instead of holding it for the full replay window.

Resolve the post-delete cancellation pass from durable leases. The deleted
conversations cannot be read back, so re-reading each one only scaled the
cascade while probing the owner directory once per removed id; one lease
read now resolves every live child address instead.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H1cCMDrTWaRNkmtKjpWELZ

* fix: never consume a result the owner cannot replay

Retention for consumed claims is bounded, so a burst of undelivered results
could evict an earlier one and lose it for good. The owner now admits a claim
only while it can retain a worst-case result, and refuses the routed claim
otherwise instead of consuming it, leaving the result on the task for a later
poll. Retained claims are never displaced; control replays keep evicting.

Key a control replay by the command itself rather than by one caller's
correlation id. The transport's own retry reuses a single envelope, but a
caller that saw the owner as unavailable reissues the command under a new id,
which steered, queued, or interrupted the child a second time.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H1cCMDrTWaRNkmtKjpWELZ

* fix: own a claimed result until it is acknowledged

A consumed terminal result is task-owned state, not a cache entry. It now
carries no expiry at all: the owner holds it until a caller acknowledges
receipt, and only then is it released. Retention stays bounded by the
existing admission gate, which refuses a claim the owner could not keep
rather than consuming a result it might drop.

Identify a control by the caller's invocation instead of by its content.
The tool mints one id per invocation and routing carries it, so a routed
retransmission of that invocation replays the owner's result while two
deliberate identical commands arrive under distinct ids and both apply.
Content-derived identity could not tell those apart and would have
answered the second from a stale snapshot.

Wait for the dpkg frontend lock in the best-effort Playwright font step.
Its timeout kills npx while the apt-get it spawned keeps the lock, which
then failed the fatal Redis install and ended the MCP replica jobs before
any test ran (#14983).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H1cCMDrTWaRNkmtKjpWELZ

* fix: treat acknowledgement as part of delivering a result

Publishing an acknowledgement once and ignoring the outcome meant a result
could be reported as delivered while the owner never learned it could let
go, and since that retention neither expires nor evicts, enough lost
acknowledgements would fill it and refuse every later remote claim.

An acknowledgement is now confirmed: publishing to zero subscribers is not
success, it retries inside the ordinary request window, and a claim whose
acknowledgement cannot be confirmed reports the retryable unavailable path
instead of handing back a result the owner still holds. A later poll
recovers that result and acknowledges it, and releasing is idempotent.
Owner registration also outlives the task while a result is unacknowledged,
so the retained result cannot become unreachable.

Take the control invocation identity from the provider's tool-call id
rather than minting one per execution, so replaying the same tool call
stays idempotent while two distinct calls with identical payloads both
apply.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H1cCMDrTWaRNkmtKjpWELZ

* style: sort the widened node:crypto import

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H1cCMDrTWaRNkmtKjpWELZ

* fix: own control invocations and cancellation plans at the task seam

Applies one logical control exactly once for its owning task rather than
in the transport, so a local caller and a routed caller of the same
invocation agree, and reusing an invocation id for different content is
refused instead of silently applied. Invocation identity now comes from
the run, agent, and provider tool-call id hashed to a bounded 32
characters, so a repeated `call_0` never bleeds across tasks and no id can
overrun the routed bound.

Cancellation for conversation deletion is now resolved into a plan while
those rows are still readable, then replayed against the owner directory
after the cascade is deleted. Owner registration is awaited before any
provider work, so a child that cannot be addressed never starts.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H1cCMDrTWaRNkmtKjpWELZ

* style: separate the control invocation map from the next member

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H1cCMDrTWaRNkmtKjpWELZ

* fix: close subagent deletion, claim, and control invocation gaps

Bulk conversation deletion now runs behind a durable owner admission fence.
Draining alone could not close the race: a child admitted on another replica
after the drain read its leases would start provider work against a parent
about to disappear. The fence is written before any lease is read and each
child revalidates it after its own lease is written, so one of the two always
observes the other. It expires on its own, so a process lost mid-deletion
cannot leave an account unable to run subagents.

A terminal child result is no longer kept alive in the owning replica's
memory until someone acknowledges it. Collection is recorded durably on the
child's own message against the polling invocation, so the poll whose
response was lost recovers its own result while a different invocation is
told the result was already collected. Owner-side retention returns to an
ordinary bounded cache that expires, which is what abandoned polls needed:
they can no longer occupy claim capacity until the process restarts.

The deletion drain now cancels each task under one invocation held for the
whole drain, stops re-sending once the owner answers, and retries only
deliveries it could not confirm. A routed control replay also validates the
command fingerprint, so one invocation id carrying different content reaches
the owner to be refused instead of collecting the earlier command's success.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H1cCMDrTWaRNkmtKjpWELZ

* test: assert the drain's calls before restoring its spies

Restoring a spy also clears its recorded calls, so the drain assertions
ran against an emptied mock. Formats the durable claim method tests.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H1cCMDrTWaRNkmtKjpWELZ

* fix: close the follow-on gaps in the deletion fence and result claim

The admission fence now carries an ownership token, so an overlapping
deletion's fence is never lifted by the one that finishes first, and both
fence writes invalidate the cached auth user document. It also covers the
other bulk-delete path: `DELETE /` with no conversation filter removes every
conversation, so it runs behind the same fence rather than a bare drain.

The durable record now decides who holds a one-shot result. An owner replaying
a retained response could hand the same terminal claim to a second invocation;
that invocation is told the result was already collected, while the one that
consumed it still recovers its own. A task with no durable record to arbitrate
keeps whatever the owner answered.

Drain cancellation treats `not_found` as unconfirmed: a missing registration
while the durable lease is still live means the child may be running, so the
command is retried under its invocation once the owner republishes itself.
Control fingerprints are hashed, so retaining one per invocation costs a fixed
few bytes instead of a bounded message.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H1cCMDrTWaRNkmtKjpWELZ

* fix: hold every deletion fence and keep live idempotency records

An owner now holds one admission fence per concurrent bulk deletion instead
of one at a time, so admission reopens only when the last deletion finishes
regardless of completion order. Expired fences are pruned as new ones arrive
and the set is bounded, so an abandoned fence cannot accumulate or lock an
account out.

A failed durable claim write is no longer read as an absent record. Handing a
terminal result over without recording its claimant would let another
invocation collect the same one-shot output once the database recovered, so
the collection reports the retryable unavailable path and leaves the result
for a later poll.

Control invocation records now evict tasks the store no longer holds before
live ones, over a bounded scan. Dropping a live task's record would let a
caller retry apply its queue, steer, or interrupt a second time once the
transport replay had also expired.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H1cCMDrTWaRNkmtKjpWELZ

* fix: keep the deletion fence portable and never drop a live record

The admission fence is written with plain update operators again. DocumentDB
rejects pipeline-form updates, and this runs before any deletion, so the
pipeline form would have failed both bulk-delete endpoints outright on a
supported database target.

An excess deletion is now refused rather than silently displacing the oldest
active fence, which would have reopened admission for a deletion still
running. Expired fences are pruned before the cap is tested, so only genuinely
concurrent deletions count against it.

Control invocation records now sweep every settled task's entry when the
window fills, and a window of entirely live records refuses the new control
before touching the child instead of evicting one. Applying a command with no
room to record it would let the caller's own retry apply it twice.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H1cCMDrTWaRNkmtKjpWELZ

* fix: hold the fence, bound recovered results, and expire stale commands

The admission fence is renewed for as long as its deletion runs, so a very
large account or a stalled database cannot let it lapse while conversations
are still being removed. Only the deletion's own fence is renewed, and the
renewal stops with the operation.

Cancellation now covers every conversation the cascade removed, not only the
ones a plan named: a grandchild lives in its own parent's scope, which a plan
naming the deleted root never reaches.

A routed request carries the deadline its caller waits for, and an owner drops
one that arrives past it. A publisher disconnected mid-request queues the
envelope offline and delivers it after the caller was told the owner was
unavailable, which would otherwise steer a child the caller believes untouched.

A result recovered from its durable child message is bounded like a routed one.
The message keeps the child's untruncated output, so recovery could otherwise
return far more than the routed result limit allows.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H1cCMDrTWaRNkmtKjpWELZ

* test: size the fence window so a renewal can be observed

The renewal test set a 30ms drain timeout but the five-minute grace window
dominates it, so the interval was 100 seconds and no renewal could fire
inside the test's deletion. The grace window is an option now, matching the
store's other timings, and the test sizes the window to 90ms.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H1cCMDrTWaRNkmtKjpWELZ

* fix: wire the durable claim method and close the fence follow-ons

The production store never received `claimSubagentTaskResult`, so every
terminal result would have surfaced as unavailable once a task settled. The
host wires that object from JavaScript, where the factory's parameter type
checks nothing, so the factory now refuses a store missing any method it
calls rather than failing at the first claim.

The routing transport takes a dedicated publisher with the offline queue
disabled. The shared client held commands issued during a disconnect and
delivered them after the caller had given up, which the request deadline
narrowed but could not close inside the clock-skew allowance.

Fence renewal invalidates the cached auth document like the fence and release
paths, and a renewal reporting its entry gone re-takes the fence instead of
letting the deletion run on unfenced. The post-delete cancellation retries a
transiently unreachable owner: the conversations are already gone, so it is
the only pass that can still stop a late-admitted child.

A replaced replay entry no longer leaves its bytes counted, which would have
inflated the cache's total until unrelated responses were evicted.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H1cCMDrTWaRNkmtKjpWELZ

* test: wait on observed lease renewal instead of a fixed delay

The shared-lease renewal test held a 60ms lease and slept 100ms before
asserting an overlapping worker was refused, so a loaded runner that
starved the 10ms heartbeat past the TTL let the lease lapse and the
second worker run. Spy on acquisition and renewal, then wait until a
renewal succeeds past the acquired lease's own deadline — direct
evidence the heartbeat carried it past expiry, with no timing
assumption — and give the lease enough headroom that a stalled timer
no longer decides the outcome.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H1cCMDrTWaRNkmtKjpWELZ

* fix(agents): close the routing, fence, and cache gaps found in review

Five separate seams, each with its own failure:

`Cluster.duplicate` reads its first argument as a startup-node list and
its second as the overrides, unlike `Redis.duplicate`, so the publisher's
`enableOfflineQueue: false` was silently dropped under
`USE_REDIS_CLUSTER` and a command issued mid-disconnect could still
reach a child after its caller was told `unavailable`. Route both
through `duplicateIoRedisClient`.

The control window's capacity refusal ran before the store knew whether
it owned the task, so unrelated local load could veto a cancellation
bound for another replica. Establish that the task is local first and
leave a remote one to its owner's window.

`clearInterval` stops only future fence renewals. One already waiting on
the database could resolve after the release, read its own lifted fence
as expiry, and write a replacement that nothing remained to lift —
closing subagent admission for the account until it aged out. Track the
in-flight renewal, refuse overlapping passes, and await it before
releasing.

Every owner bounds its own task list, but the aggregation appended each
batch whole, so the model-facing list grew with the number of replicas
holding the scope. Cap the merged list while still reading every reply
for the stale-registration sweep.

The admission-fence prune commits independently of the fence that
follows it, so a refused or failed push left the cached auth document
describing entries the collection no longer held. Invalidate whichever
way the second write goes.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H1cCMDrTWaRNkmtKjpWELZ

* fix(agents): cap the merged task list the poll tool actually reads

Each owner bounds its own reply and the remote aggregation bounds their
sum, but `listTasks` merged that bounded remote list with however many
children this replica owns and returned it whole. `check_background_task`
could therefore still receive roughly twice the advertised cap. Bound the
deduplicated, sorted result and export the cap so both seams share one
number.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H1cCMDrTWaRNkmtKjpWELZ

* test: admit every task the merged-list cap test starts

The base store admits ten concurrent runs per scope by default, so
starting 150 at once left most refused for capacity and the assertion
never reached the merge it was written to check. Raise the cap for this
store only; admission is a different invariant with its own tests.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H1cCMDrTWaRNkmtKjpWELZ

* fix(agents): let a deletion notice its admission fence lapsing

Renewal failures were logged and swallowed, so a run of rejected writes
let the last confirmed `fencedUntil` pass while the deletion carried on
believing admission was still closed — long enough for another replica
to admit a child against conversations about to be removed. Track the
deadline only a confirmed write advances, and check it after the drain,
before anything is deleted: nothing has been removed at that point, so
the operation fails closed and the caller retries once the fence can be
held. A lapse detected after the rows are gone is logged instead, since
reporting failure there would invite a retry against conversations that
no longer exist.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H1cCMDrTWaRNkmtKjpWELZ

* test: raise both concurrency caps the merged-list test trips

Raising the per-scope limit left the store-wide `maxRunningTotal` at its
default hundred, so fifty of the hundred and fifty starts were still
refused. Verified against the base store directly this time: with only
the per-scope cap raised it admits a hundred, and with both raised it
admits all hundred and fifty.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H1cCMDrTWaRNkmtKjpWELZ

* fix(agents): close the fence renewal gap and keep running tasks listed

A renewal that started before its deadline but landed after it was still
credited with extending the fence from its own start time, so a window in
which admission stood open was papered over: a child could take a lease
the drain had already read past and the deletion would proceed without
cancelling it. The deadline now only advances when the write lands while
the previous one still holds; anything later records a lapse the fence
cannot be restored backwards over.

The model-facing cap sorted oldest-first and sliced, which dropped the
newest tasks — including children that had only just started running,
and which the poll tool offers no other way to discover. Bound by status
instead: running children first, then the most recent settled results.
Both caps share one helper, and the routed aggregation now bounds after
its loop so the choice is made across every owner's reply rather than by
whichever answered first.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H1cCMDrTWaRNkmtKjpWELZ

* fix(agents): finish the cap and the fence at the seams they still missed

The status-aware cap only reached the requester: an owner's own reply
still sliced positionally, so a replica holding more than the cap dropped
its running children before the requester could bound anything. Both
sides now share `boundedTaskList`.

A fence that lapsed during the deletion itself was only logged. The rows
are gone by then, so failing is still wrong, but the child another
replica admitted while the fence was down is not: the fence is retaken
and the drain repeated to cancel it.

A child's lease renewal had the same retroactive hole the admission fence
had — Mongo filters on the `now` captured before the call, so a write
landing after the lease expired still moves the row forward, while an
owner drain reading active leases in that gap saw the thread as free. The
lease now carries its own deadline and a late renewal stops the executor.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H1cCMDrTWaRNkmtKjpWELZ

* test: cover the lease lapse and the post-deletion re-drain

The owner-side cap shipped with a regression test; these two did not.
One drives a lease renewal that succeeds only after the lease it was
extending had expired and asserts the executor stops; the other lets the
fence lapse during the deletion itself and asserts a second drain runs
while the request still reports success.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H1cCMDrTWaRNkmtKjpWELZ

* fix(agents): close live-task lifecycle gaps

* test(redis): exercise cluster node discovery

* fix(test): type cluster discovery seam

* fix(ci): wait for orphaned apt processes

* fix(ci): reserve time for apt drain

* fix(ci): skip optional fonts in MCP jobs

* fix(agents): recover tasks after owner loss

* fix(agents): preserve local task discovery

* fix(agents): initialize fail-fast cluster publisher

* style(agents): sort routing test imports

---------

Co-authored-by: Claude <noreply@anthropic.com>
2026-08-19 02:16:35 -04:00
Heinz-Alexander Fuetterer
0ab3414c01
🪙 chore: GPT-5.6 Terra and Luna Token Cost Rates (#14964)
* fix: update openai gpt-5.6 family token costs

* test: cover GPT-5.6 pricing and float ratios

---------

Co-authored-by: Danny Avila <danny@librechat.ai>
2026-08-18 08:12:32 -04:00
Ravi Kumar L
006e421cd2
💡 feat: add DB-backed admin insights (#14898)
* feat: add Mongo-backed admin insights

* feat: gate insights with environment variable

* fix: tighten insights access and activity metrics

* fix: preserve insights date selections

* perf: parallelize insights search aggregation

* test: wait for MCP conflict recovery

* test: satisfy strict MCP recovery typing

* fix: disable insights pagination while loading

* fix: localize insights range shortcuts

* fix: bound insights search input
2026-08-18 07:51:51 -04:00
Danny Avila
547bd8c4bf
🧵 feat: Persist View-Only Subagent Threads (#14957) 2026-08-18 07:41:42 -04:00
Danny Avila
aa35cd42b1
📬 feat: Add Durable Agent Trigger Delivery (#14925)
* feat: wire trusted agent trigger dispatch

* feat: add durable agent trigger delivery

* fix: annotate trigger envelope byte limit

* test: isolate trigger startup in server specs

* fix: fence trigger delivery during account deletion

* test: isolate trigger service in user controller specs

* fix: close trigger deletion admission race

* fix: harden account deletion fences

* fix: close durable trigger review gaps

* fix: require offline stale-fence recovery

* fix: type trigger lane sequence ids

* fix: fence admin user deletion triggers

* fix: make trigger deletion recovery durable

* fix: harden offline user deletion

* fix: serialize trigger lane publication

* style: sort trigger delivery imports

* fix: recover orphaned trigger publications

* fix: preserve trigger recovery ordering

* fix: fence trigger publication during purge

* fix: defer remote trigger deletion fences

* fix: close durable delivery cleanup races

* fix: drain CLI generation owners before deletion
2026-08-17 09:25:08 -04:00
Danny Avila
485abef3fa
🥚 refactor: Default Agents to Preferred Stateful Workspace Scope (#14908)
* feat: add user default for stateful agent workspaces

* style: sort stateful workspace imports
2026-08-16 22:31:01 -04:00
Danny Avila
f829aca9fb
🧩 fix: Align Tenant and MCP Configuration Resolution (#14904)
* fix: Align Tenant and MCP Configuration Resolution

* fix: Preserve Operator-Owned MCP Entries

* fix: Preserve Configuration Source Ownership

* style: Normalize Middleware Import Order

* fix: Preserve Process Server Precedence

* test: Align Tenant-Aware E2E Setup
2026-08-16 22:30:46 -04:00
Marco Beretta
fdc9c77f6e
🗄️ feat: Archive All Chats From Data Controls (#14885)
* feat: archive all chats from data controls

Adds an "Archive all chats" row under Data controls > Your data, next to
Shared links, with a confirmation dialog. It calls a new
POST /api/convos/archive/all endpoint backed by archiveAllConvos, which
archives every conversation currently visible to the user in a single
updateMany and refreshes the stats of every chat project the archived
conversations belonged to.

Temporary and retention-expired conversations are skipped: they are
already hidden from the chat list, so archiving them would only surface
them in the archived view. The update runs with timestamps disabled so
each conversation keeps its own updatedAt and the archived list stays
sorted by real activity.

Archiving a conversation now also drops the new-chat message cache alias
for it. A chat's first turn writes the same message array under both the
conversation key and the new-chat key, so without this the messages of a
just-archived chat kept rendering on the new chat screen until a reload.
Deleting already handled this; archiving did not.

* fix: keep archive-all state consistent

* fix: drop stale detail caches after bulk archive

* fix: harden archive-all request handling

* fix: reconcile archive batch failures

* Fix project stats refresh races and archive route boundary

* Fix archive-all review findings

* Fix archive scan index and partial-batch stats refresh

Reconcile project stats for already-committed archive batches when a later
batch fails, and index the archive scan as { user, _id } so non-tenant
pagination can use _id order.

* Fix Recoil reset after a partial archive-all failure

Refetch the submitted conversation on error and start a new chat only
when that conversation is still active and already archived.

* Fix archive recovery from resetting a newly opened chat

Re-read the active Recoil conversation after the archive-state lookup
resolves, so a slow getConversationById cannot start a new chat if the
user already opened another conversation.

* Fix project-stat reconciliation after archive races

Keep retrying optimistic project-stat writes instead of returning a
stale document after three lost CAS attempts, and retry destination
project discovery after a transient distinct failure.

* Fix archive reset and project-count increment races

Leave already-archived chats open after archive-all, recount new
project conversations instead of incrementing, and skip a delayed
increment when a concurrent refresh already recorded that chat.

* Recover destination projects after discovery retries exhaust

Keep committed conversation IDs when post-archive distinct fails, then
rediscover those projects in finally so a moved conversation's
destination still gets reconciled after the error is rethrown.

* Fix archive-all recovery batching and remount pending state

Recover destination projects in 500-id chunks so the final lookup
cannot exceed Mongo's command size, and share archive-all pending
state through a mutation key so Settings remounts stay disabled.

* Stamp bulk-archived chats and refresh the pinned cache

Bulk archive wrote only isArchived, so the archived table dated every
swept chat by createdAt and the default archivedAt sort dropped the whole
run into the legacy null group. Stamp one timestamp for the sweep; the
filter only matches unarchived chats, so an existing stamp cannot move,
and timestamps: false still preserves each updatedAt.

The pinned section fetches on its own key with a five-minute stale time,
so an archived pin kept rendering in the sidebar until that expired.
Invalidate it alongside the other lists on both success and failure.

Also drop the async from the failing-batch updateMany mock: its
Promise<never> is not assignable to the Query return type, while a plain
synchronous throw types as never.

* Bound archive recovery state with the sweep marker

Recovery held every committed conversation id for the life of the
request so the finally block could re-run project discovery after an
in-loop distinct gave up. Slicing that array into 500-id queries capped
the BSON command size but not the heap, so a very large history could
exhaust a worker mid-archive.

The archivedAt stamp already identifies exactly what this call
committed, so recovery is now one distinct scoped to it. That filter is
a prefix of the existing user/isArchived/archivedAt index, and the two
discovery call sites collapse into one filter-taking helper.

* Reconcile archive stats when a write outcome is unknown

A batch that commits but whose result never returns, a stepdown or a
connection drop between commit and acknowledgement, left archivedCount
at zero, so the finally block skipped both marker recovery and the stats
refresh. The chats were archived, so no retry could find them again: the
sweep filter no longer matches them and their projects kept stale
counts.

Both now key off the write attempt rather than the returned count.
Nothing else needs to change, because the marker is stamped by the same
write whose result went missing.

* Retry dropped project refreshes and guard stale pointer writes

Two ways a project could keep stale stats after archive-all.

A refresh that rejected was logged and dropped for good. Its chats are
archived, so no retry of archive-all can find them again to recompute
against, and the likeliest rejection is the recoverable one:
refreshChatProjectStatsForUser gives up when the project changed under
every compare-and-set attempt. Failures are now collected and replayed
once the rest of the run has stopped competing with them.

A save already in flight could also undo the sweep. Its conversation
document still said visible, so its tail took the pointer branch and
wrote lastConversationId back to a chat the sweep had just archived,
leaving the project advertising activity on a chat the workspace hides.
The pointer write now confirms the chat is still visible first, and
recomputes the project when it is not.

* Verify project pointers after the write, not before

Checking visibility before the pointer write only moved the race earlier:
a sweep landing between the check and the update still archived the chat
and cleared the project, and the write then restored it as
lastConversationId.

The check now runs after the write and repairs instead of preventing. A
sweep that lands earlier is caught here; one that lands later refreshes
the project itself, and refreshChatProjectStatsForUser compare-and-sets,
so it cannot commit a count it read before this write. Same single
indexed read as the check it replaces.
2026-08-16 22:11:57 -04:00
Danny Avila
1b7e2a4e6a
⚡ perf: Optimize First Load of Large Conversations (#14901)
* ⚡ perf: Index the Conversation Fetch and Trim the Client Message Projection

* ⚡ perf: Memoize the Message Tree per Cache Write

* ⚡ perf: Serve Message Reads via the Trimmed Projection and an Ownership Probe

* ⚡ perf: Defer Collapsed Disclosure Bodies Until First Expansion

* ⚡ perf: Progressively Mount Long Threads from the Scroll Anchor

* 🩹 fix: Address Codex Findings on Retention, Anchoring, and Cache Bounds

* 🩹 fix: Poll the Oversized Export Precondition Through the Progressive Mount

* 🩹 fix: Keep Video Results in the Client Message Projection
2026-08-16 19:45:21 -04:00
Marco Beretta
832bac39ad
🗄️ feat: Record When a Conversation Was Archived (#14863)
* feat: record when a conversation was archived

The archived chats dialog has a "Date Archived" column that was bound to
createdAt, so it showed when the chat was created rather than when it was
filed away. Nothing recorded the latter.

Conversations now carry archivedAt, set on archive and cleared on
unarchive, and the column reads it. Chats archived before the field
existed have no stamp and fall back to createdAt, which is exactly what
that column already showed for them.

The archive view sorts on the new field. archivedAt is absent on every
previously archived chat, so the missing-value group is the common case
here rather than an edge case: the cursor's null handling, written for
titles, now covers both, and an absent stamp survives the cursor as null
instead of collapsing to the epoch and replaying the whole archive.

* fix: address review findings on the archived-at stamp

- Protect `archivedAt` from saveMessageToDatabase's unset sweep. Any
  persisted field missing from endpointOptions is unset, so sending a
  message in an archived chat cleared the stamp while leaving isArchived
  true, silently dropping it into the legacy fallback group.
- Order the legacy group by the createdAt the dialog displays rather than
  by last activity. The cursor's secondary key is now chosen per sort
  field, so the fallback the cell renders and the order the server
  returns cannot disagree.
- Put that secondary key in the archive index too, so paging the legacy
  group does not fall back to a blocking sort.

* fix: keep archivedAt on a redundant archive request

Opening an archived chat and hitting the archive shortcut, or retrying
the POST, sent isArchived: true again and replaced Date Archived with
now. saveConvo now stamps only on the unarchived-to-archived transition
and still clears the field on unarchive.

* fix: make archive timestamp updates atomic

* test: type the archive race spy against the driver signature

* fix: archive without an aggregation-pipeline update

DocumentDB documents no support for pipeline-form updates on any engine version,
and the repository's compatibility assessment records that a prior P0 rewrote the
three that existed. Stamping archivedAt through a $cond pipeline reintroduced one,
which would have sent every archive and unarchive to the route's 500 handler on a
supported 5.0 deployment.

The conditional stamp is now a compare-and-set on isArchived, which keeps the
transition atomic without a pipeline: only the write that finds the chat unarchived
stamps it, so a duplicate or retried archive leaves the original date alone and an
unarchive that lands first is re-stamped. Schema defaults and createdAt-on-insert
go back to mongoose's own setDefaultsOnInsert and $setOnInsert, and tenantId is
once again stripped by the tenant-isolation plugin rather than by hand.

* fix: do not report a racing archive as a missing chat

Both conditional writes of the compare-and-set miss when the archive flag flips
between them: the chat was already archived when the transition write ran and
unarchived again before the already-archived write. saveConvo returned null for
a conversation that plainly exists, so POST /api/convos/archive answered 404.

Confirm the conversation is really gone before accepting that result, and retry
the pair when it is not. An unknown id still costs one existence read and falls
straight through to the 404.

* fix: resolve a fully contended archive to the chat's real state

Alternating archive and unarchive requests can split every attempt of the
compare-and-set: each transition write sees the chat archived and each
already-archived write sees it unarchived. Exhausting the retries therefore
proved nothing about whether the conversation exists, and the no-upsert archive
route turned a lost race back into a 404.

Read the conversation once more when the retries run out and answer with its
actual current state instead.
2026-08-16 17:07:44 -04:00
Marco Beretta
7857a99d63
⚡ perf: Flip the Pinned Flag Without a Full Conversation Save (#14862)
Pinning routed through saveConvo, which reads every message id for the
conversation and writes the whole array back just to set one boolean, and
can trigger a project-stats recompute on top.

None of that applies to a pin: it moves no chat between projects, changes
nothing the project workspace hides, and opens no retention window. A
dedicated setConvoPinned does the single findOneAndUpdate instead.

Measured against an in-memory MongoDB with the real message methods
wired in, on a 120-message chat: two driver commands and 3706 bytes
before, one command and 245 bytes after. The write scales with the
message count, so the gap widens on longer chats.

Archiving keeps using saveConvo, which it needs for exactly the project
stats and retention work a pin does not.
2026-08-16 17:07:27 -04:00
Marco Beretta
5d88b8453e
🪟 fix: Resolve Context Windows and Pricing for Newer Model Families (#14877)
* fix: resolve context windows for Qwen3.5+ and newer model families

Qwen3.5 model ids matched the `qwen3` key and inherited its 40,960 token
window instead of their native 262,144, so pruneMessages dropped whole
conversations down to the system message once tool output grew large.

Add the 3.5 through 3.8 generations, Meta Muse Spark, Muse Glimmer and
Llama 4, plus newer DeepSeek, GLM, Kimi, Grok, Mistral, Nova, Cohere and
MiniMax entries. Llama 4 Scout is capped at 1M rather than its 10M native
ceiling, since no host serves near that and a 10M value would stop
pruning from ever firing.

findMatchingPattern now matches a vendor-prefixed id on its model segment
first. No map key contains a slash, so a prefix could only ever
contribute a spurious longer match: moonshotai/kimi-k2 matched moonshot
(8 chars) over kimi-k2 (7) and reported half the real window.

* fix: price newer models and make the pricing test helper faithful

Models added to the context map billed either at defaultRate or at an
older generation's rate. Llama 4 and Muse Spark fell to $6/1M, Kimi K3
undercharged 5x through the kimi key, and Grok 4.5/4.6 overcharged
through grok-4. Add rates for 21 models.

tx.ts needs no matcher change of its own, since it receives
findMatchingPattern by injection from @librechat/api. The vendor-prefix
fix therefore already corrects moonshotai/kimi-k2, which was matching
moonshot at $2.00/$5.00 rather than kimi-k2 at $0.60/$2.50.

The data-schemas test helper implemented a different algorithm from the
function production injects, returning the first reverse-order match
instead of the longest, so the pricing suite was not describing real
billing behavior. Mirror the real implementation. Every existing test
still passes, so nothing was relying on the lossy version.

Rates for the Qwen 3.x plus/flash tiers, DeepSeek V3.1/V3.2, Kimi
K2.6/K2.7 and Nova 2 Pro are left as they are, since published figures
disagreed by more than the current fuzzy match is wrong.

* test: assert parity between the context and pricing maps

Neither map imports the other, so a model added to one and forgotten in
the other is silently wrong at runtime: it bills at defaultRate, or it
has no window to prune against. A comment in tx.spec.ts claimed such a
test lived in packages/api, but none existed.

Exact key parity is the wrong invariant. Both maps resolve by longest
substring match, so gpt-4-32k legitimately prices through the 32k bucket
without a row of its own. Assert instead that every key resolves in the
other map, with exemption lists covering the genuine gaps and a further
test that fails once an exemption goes stale, so the lists shrink rather
than rot.

Verified the assertions actually fail: injecting a context key with no
price, a price with no context window, and a rate for an exempted model
each fails the expected test and names the offending model.

* fix: correct GLM 5.3, Kimi K3 aliases, Grok tiering and prefixed overrides

Four gaps found in review.

glm-5.3 matched the glm-5 key and reported 204,800 tokens rather than the
1,048,576 it shares with 5.2. Add both the context and pricing rows.

The dot-prefixed Kimi K3 ids were added to the context map without
pricing counterparts, so moonshot.kimi-k3 resolved to moonshot.kimi and
moonshotai.kimi-k3 to moonshot, billing completion at 2.5 and 5.0 rather
than 15.0.

Grok 4.5/4.6 carry a 500K window, which makes xAI's doubled rate above a
200K prompt reachable, but neither key existed in premiumTokenValues so
long prompts stayed on the base rate. Add the premium entries.

Matching the model segment of a vendor-prefixed id was too eager.
EndpointTokenConfig is an arbitrary record and one built from OpenRouter
is keyed by org/model, so an alias like org/model-latest could resolve to
a bare model entry and pick the wrong configured context and rates. Keep
a matched key that carries the vendor, and retry on the segment only when
the whole id resolved to a prefix-only match.

* refactor: type the model matcher as keys-only and drop the caller casts

findMatchingPattern reads keys and never inspects a value, but its
parameter was typed as a union of the two maps it happens to be called
with, so every injection site cast into it. pricing.spec.ts already
carried that cast and parity.spec.ts copied it.

Name the intent instead. Both adapters now pass their map through
unchanged, and a future change that starts reading values will fail to
compile rather than slip past a cast.
2026-08-16 16:41:02 -04:00
Marco Beretta
5fc05ac037
🕰️ fix: Stop Pinning and Archiving From Counting as Chat Activity (#14861)
* fix: stop pinning and archiving from counting as chat activity

Both routes went through saveConvo, which lets mongoose stamp updatedAt.
The sidebar orders chats by that field, so pinning hoisted an untouched
chat to the top of Today, and unarchiving a year-old chat dropped it
there too instead of back into its own date group.

saveConvo now takes preserveUpdatedAt, and both routes pass it. They also
pass noUpsert: with timestamps suppressed an upsert would insert a
conversation carrying neither createdAt nor updatedAt, so an unknown
conversation id is now a 404 rather than a silently created stub.

* test: pin a project's activity pointer against metadata-only saves

Review raised that preserving updatedAt could drag a project's
lastConversationAt back to the pinned chat's older timestamp, since the
incremental path $sets it outright.

That path is not reachable here: a pin carries no chatProjectId, so
previousChatProjectId stays null while the conversation has a real one,
projectMembershipChanged is therefore true, and saveConvo takes the full
recompute branch instead. This test holds that in place, with a newer
sibling conversation in the project so a regression to the incremental
path would fail it.

* fix: keep updatedAt through the retention backfill

Under RetentionMode.ALL a legacy chat with no stored isTemporary gets a
second write after the main update, and that one still had mongoose
timestamps enabled. The first archive of such a chat therefore bumped
updatedAt anyway and landed in Today, defeating preserveUpdatedAt on
exactly the old conversations it was meant to protect.
2026-08-16 16:20:20 -04:00
Danny Avila
fb8ae881cf
⏱️ feat: Show Run-Step Durations On Tool Cards (#14892)
* ⏱️ feat: Show Run-Step Durations On Tool Cards

Surfaces how long each tool call took, derived from the `closed_at` /
`created_at` pair already carried by `on_run_step_closed` — the same event
#14871 and #14873 use for the terminal status. No new event, no new SDK
surface.

The duration is stamped onto the content part at the same three sites as
`runStepStatus`, so it survives a reload and a resumable reconnect rather
than living only on the live React message:

- `callbacks.js`, on the aggregated part before the event is forwarded
- `RedisJobStore`, in the host-authored replay reconstruction branch
- `useStepHandler`, on the live message

Rendering lands in the shared `ProgressText`, which nine tool cards already
use, rather than in each card: one place decides whether a duration is shown
and how it reads, and the cards only forward the number. That keeps this from
adding a tenth independent state derivation to a component family whose
label/announcement/progress split is already the subject of AI-1810.

The value is deliberately absent rather than zero whenever it would be a
guess — no `created_at`, non-finite input, or a negative elapsed time from
two clocks that disagree, which is now reachable because a step can be opened
in one process and closed in another after a checkpoint resume. Sub-second
durations are suppressed as noise, and it renders only on a settled,
non-error card, where the slot is not already carrying the cancelled icon or
the error suffix.

For assistive technology the compact form (`3.5s`) is hidden and paired with
a spoken equivalent ("took 3.5 seconds"), both inside the button, so the
accessible name carries the duration without an `aria-live` region
re-announcing it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014vLhxCFMYkCaTsoFTiAjJ5

* 🎨 style: Sort Imports In Touched Files

The import-sort gate runs against the files a PR changes, so pre-existing
drift in `ProgressText.tsx` and `RedisJobStore.ts` surfaced on this branch.
Both were already unsorted on `dev`; this is the sorter's output, with no
semantic change.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014vLhxCFMYkCaTsoFTiAjJ5

* 🐛 fix: Accept Partial Timestamps In Run-Step Duration Helper

`getReportableRunStepDurationMs` declared its parameter as
`Pick<RunStepClosedEvent, 'created_at' | 'closed_at'>`, where `closed_at` is
required. That contradicted the function's own purpose: every guard inside it
exists precisely to handle stamps that may be missing.

The Redis replay branch reconstructs closures from persisted JSON and holds
nothing stronger than "might be a number", so it failed to typecheck against
the narrower signature.

Widened to an exported `RunStepTimestamps` shape with both stamps optional,
rather than asserting at the call site — an assertion would move the decision
about what is trustworthy somewhere it cannot be enforced, which is the thing
the helper exists to centralize. Callers holding a fully-typed event still
pass, since a required field satisfies an optional one.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014vLhxCFMYkCaTsoFTiAjJ5

* 🐛 fix: Suppress Duration When Failure Arrives As errorSuffix Alone

At every call site `error` carries cancellation while failure travels
through `errorSuffix` with `error` false, so gating the duration on
`!error` alone rendered "· 3.5s" beside "· failed" — and announced it.
The gate now checks both terminal-failure channels.

The original test pinned only the `error: true` path, which is why this
survived; the failed-via-suffix path is now pinned separately, both the
visible and the announced half.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014vLhxCFMYkCaTsoFTiAjJ5

* 🧩 refactor: Persist Raw Run-Step Durations, Threshold At Render Only

The three stamp sites filtered through the 1-second reportability
threshold before persisting, baking a presentation rule into stored
data: a 900ms step stored nothing, making "fast" indistinguishable from
"not derivable" and unrecoverable if the display rule ever changes.

Stamp sites now persist the raw `getRunStepDurationMs` value — absent
only when genuinely not derivable — and the renderer alone decides what
is worth showing, which `ProgressText` already did. Rendering is
unchanged. `getReportableRunStepDurationMs` is removed; it existed only
to serve the write-time filter, and a test now pins that sub-threshold
durations survive to storage.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014vLhxCFMYkCaTsoFTiAjJ5

* 🐛 fix: Suppress Duration On Backgrounded Bash And Code Cards

A backgrounded call's run step closes when dispatch returns the handle,
so the stamped duration is the dispatch time. Rendering it beside
"Running/Finished in background" misstated a detached task's runtime as
seconds — and violated the "settled card only" rule, since the card is
still tracking the detached run. Scope is exactly the two cards that
parse background handles.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014vLhxCFMYkCaTsoFTiAjJ5

* 🌍 fix: Format The Sub-10s Decimal For The Active Locale

The fractional seconds value was interpolated as a raw JS number, which
hardcodes the en-US decimal point into every language — "1.4s" where
the locale writes "1,4 s" — and translators cannot fix a number
formatted in code. The value is now formatted via Intl.NumberFormat
with i18n.language, following MessageTimestamp's pattern of threading
the language into the util; plural-key selection stays on the numeric
value. A malformed language tag falls back to the plain number.

Also documents the two accepted limits of the derivation, so they read
as decisions rather than oversights: positive clock skew is
undetectable from a single stamp pair, and the value is wall-clock
elapsed, so a step held open across a suspension (checkpoint resume,
HITL approval wait) includes that time.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014vLhxCFMYkCaTsoFTiAjJ5

* 🐛 fix: Persist A Durable `backgrounded` Marker Through Harvest; Localize Minute Digits

Codex round 3, both findings confirmed.

**Background origin survived only as transient state.** The dispatch
handle in `tool_call.output` and the live status-marker attachment are
both gone once the harvester patches the settled task's stdout over the
handle — so the round-2 suppression (`backgroundHandle == null`) came
back on after harvest or reload, showing dispatch time as the task's
runtime. Following the same rule as e4bd15d (persist facts, decide at
render): the harvest patch now stamps `backgrounded: true` onto the
tool call in the same atomic write that erases the handle — on the heal
path too, which re-applies over full-row saves that reverted the part.
The cards gate on handle-or-marker; the dispatch duration itself stays
stored.

**Minute-branch digits bypassed locale formatting.** The seconds branch
went through Intl.NumberFormat while minutes interpolated raw numbers,
so Arabic/Persian locales flipped to ASCII digits above one minute. All
interpolated values now flow through the (renamed) formatDurationValue;
an ar-EG test pins the localized digits.

data-schemas cannot be installed in this environment (same npm ci 403 as
packages/api), so message.ts/harvest.ts are syntax-checked with
resolution off and otherwise verified by review; CI runs their real
typecheck and suites.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014vLhxCFMYkCaTsoFTiAjJ5

* 🧪 test: Assert The `markBackgrounded` Stamp In Harvest Expectations

The successful-harvest test's exact `toHaveBeenCalledWith` object did
not include the newly forwarded `markBackgrounded`, so the API suite
would fail on it. All three harvest-call expectations now assert
`markBackgrounded: true` — the exact-object one of necessity, the two
`objectContaining` ones deliberately, since the durable stamp (on the
best-effort file-failure path and the reapply heal alike) is now part
of the behavior under test.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014vLhxCFMYkCaTsoFTiAjJ5

* 🎨 style: Wrap Harvest Spec Expectation Per Prettier

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014vLhxCFMYkCaTsoFTiAjJ5

---------

Co-authored-by: Claude <noreply@anthropic.com>
2026-08-16 16:17:02 -04:00
Marco Beretta
8a946290f6
📌 fix: Fetch Pinned Chats Independently of the Chats List (#14860)
* feat: give the pinned chats section its own fetch

The sidebar's pinned section filtered pinned chats out of the paginated
chats list, which only holds the 25 most recently updated conversations.
Once 25 newer chats existed, a reload hid the pin until the list was
scrolled far enough to fetch the page it lived on.

Pins are now fetched directly via GET /api/convos?pinned=true behind a
dedicated query, so every pin paints with the sidebar regardless of where
it falls in the chats list. Pin and unpin invalidate that query, and the
shared conversation cache helpers keep it in step so a rename, delete or
archive is reflected without waiting for a refetch.

Pins stay out of the date groups, which groupConversationsByDate already
handled.

* fix: address review findings on the pinned chats section

- Drain the cursor rather than capping the pinned request at 100. Since
  pins are kept out of the chats date groups, anything this query dropped
  was invisible in the sidebar entirely, not merely further down a list.
- Apply the active bookmark filter to the pinned request and key its
  cache by it, matching the chats list beside it.
- Move a pin to the top of the section when the caller asks for it, so a
  pin that just received a message leads the way it does in the chats
  list instead of waiting for a refetch.
- Invalidate the pinned list when a conversation is unarchived, since
  archiving removes it from that cache and nothing put it back.
- Index the pinned lookup: it filters on user + pinned and sorts by
  updatedAt, which no existing compound index covered.
- Protect `pinned` from saveMessageToDatabase's unset sweep. Any
  persisted field missing from endpointOptions is unset, so sending a
  message in a pinned chat silently unpinned it.

* fix: keep the pinned cache reconciled across the other convo mutations

Second review pass on the independent pinned query.

- Fall back to the pins already loaded in the chats pages when the
  dedicated request fails. Pins are stripped from the date groups, so an
  error otherwise emptied the section and hid them everywhere.
- Restore default focus and reconnect refetching, matching the
  conversations query. A pin changed in another tab is only reconciled by
  a refetch, since that tab's mutation never touched this cache.
- Invalidate the pinned list from the mutations that can produce or alter
  a pinned chat without going through pin itself: duplicate, fork,
  import, project assignment, and shared-link deletion.

* fix: invalidate pins on tag and project-deletion changes

Third review pass, same class as the last: the pinned query is keyed by
the active bookmark filter, so changing a chat's tags can move it in or
out of that filtered set, and deleting a project unsets chatProjectId on
its chats, pinned ones included.

* fix: cancel in-flight pinned fetches when deleting a conversation

Deletion cancelled the regular and archived queries but not the pinned
one, so a pinned GET issued before the delete could resolve after the row
was stripped and write the deleted conversation back, leaving a row that
navigates to a missing chat. Restoring default focus and reconnect
refetching in the previous commit made those in-flight fetches more
likely, so this widened rather than appeared.

Cancelled on mutate, and invalidated on success since cancelling a race
is best effort.

* test: make the SSE query-cache mock key-aware

The conversation cache helpers now run a second, pinned-keyed findAll
pass. This mock ignored its key argument and always returned an
allConversations entry, so those pinned writes were attributed to
allConversations and the write-count assertions saw three instead of two.

* fix: keep pins in sync through upsert and pin-only pages

Root-level SSE updates and resumable settlement call upsert rather than
update, so the independently cached pinned row never moved or refreshed.
An all-pin first page also left the chats virtual list empty, so
onRowsRendered never asked for the next cursor.

* fix: keep pins current through SSE recovery and project delete

Resumable SSE reconciliation invalidated conversation and allConversations
only, so an independently cached pin kept stale title and order.
Deleting a project-backed pin that lived only in that cache also skipped
the project query, because the mutation never read chatProjectId there.

* fix: keep pins current after bookmark edits and failed pages

Renaming or deleting a bookmark rewrote tags on conversations but left
the tag-keyed pinned cache pointing at the old filter. An all-pin page
whose next fetch failed also retried forever because the empty-list
effect had no memory of the attempt. Unpinning a pin that only lived in
the dedicated cache removed it from Pinned without inserting it into
Chats, and later cursor pages cannot recover a row whose updatedAt just
jumped ahead of the current cursor.

* fix: keep pins visible after a failed refetch

A failed pinned refetch left React Query holding the previous list, so
the nullish fallback never ran and a newly pinned chat vanished from
both sections. Unpinning an older pin also inserted it into every
cached chats variant, including bookmark and search results it would
not match. Drop the checked-in agent task prompt.

* test: type the pinned conversation fixtures correctly

The delete mutation takes a plain string conversationId, but reading it
back off a TConversation fixture widens it to string | null. Hoist the id
into its own constant so the call site passes the real string.

Type the tag fixture as TConversationTag so it carries the required _id
and user fields the mocked resolved value expects.

* style: sort the sidebar imports to the repo order

The new pinned-section imports went in out of the longest-to-shortest
order the import sorter enforces.

* fix: keep drained pins and empty chat caches from breaking the sidebar

A pinned page failing partway through the drain rejected the whole query, so
every pin already fetched was discarded and the section fell back to whatever
the chats cache happened to hold. Publish the accumulated pins before
rethrowing so the retry renders against the partial set.

Unpinning a chat that only lives in the pinned cache reinserted it into the
chats list by spreading the first page, which is absent once removal has
filtered out the last loaded row. Rebuild that page instead, matching the
upsert path.

* fix: order fallback pins by their timestamp

The merge kept dedicated rows in Map insertion order and appended the pins
recovered from the chats cache after them. A chat pinned while the dedicated
refetch is failing is the newest pin, so the server would return it first, yet
it landed last and could sit below the section's visible 30vh. Sort the merged
set newest-first so a fallback row takes the place the server would give it.

* fix: keep the shared badge and the move-to-top order on pins

The pin response has no isShared: the flag is derived per list request by
attachSharedFlags, which only runs for the list queries. Reinserting an unpinned
chat into Chats therefore dropped its shared-link badge, because unlike an
in-place update there is no existing row to carry the flag over from. Read it off
the cached pin before the update removes that row.

The chats cache refreshes updatedAt when it moves a conversation to the top, but
the pinned cache only reordered, leaving the previous turn's timestamp on the row.
Sorting the section newest-first then put it straight back. Refresh the timestamp
there too, so the move survives the sort and both caches agree.
2026-08-16 11:28:33 -04:00
Marco Beretta
0b995065bc
🗂️ feat: Rework the Projects Dashboard, Sidebar and Scoped Composer (#14866)
* style: Redesign the Projects Dashboard and Sidebar

Give /projects a sticky navbar, quieter search/sort toolbar, and
folder-style cards. Drop the create-dialog close control, restyle
the sidebar Projects row, and align the workspace with the same
layout language.

* feat: Edit a Project Name and Description

Add a shared edit dialog so a project can be renamed and given a
description from the workspace or the sidebar menu. The create flow
already stored descriptions; this is the matching update path.

* feat: Delete a Project from the Workspace

Extract the project delete confirmation into a shared dialog and
expose it on the workspace header so deleting no longer requires
the sidebar menu.

* feat: Add Edit and Delete Actions to Project Cards

Give dashboard cards a more-options menu that opens the same edit
and delete dialogs as the workspace, so those actions are not
workspace-only.

* feat: Match Project Descriptions When Searching Projects

Project search only matched the name, so a project found by its
description was invisible in the sidebar and the projects dashboard.
Match the escaped search term against name or description.

* feat: Let Consumers Place and Size the ControlCombobox Popover

The popover always matched the trigger width, sat 4px from it and used
the same enter animation, which is wrong for a pill-shaped trigger in a
composer and for a full-width field in a dialog.

Add popoverClassName, matchTriggerWidth, gutter and portal so a consumer
can opt out of each. All four keep the current behaviour by default, so
existing comboboxes are unchanged; portal in particular stays true, as
turning it off inside a scrollable dialog would clip the list.

* fix: Re-render Conversation Rows When Pinned State Changes

areConversationListItemFieldsEqual left pinned out of its comparison, so
a row memoised on it kept rendering the stale pin state until some other
tracked field changed.

* fix: Keep the Project Scope When Starting a Chat From a Project

Starting a chat from the project workspace set chatProjectId on the
draft but left the URL on the unscoped route, so a reload or a refresh
of the route dropped the project. Navigate to the project-scoped new
chat URL alongside the draft.

* style: Move the Project Chip Into the Composer

The chip floated above the composer as a separate row, which read as an
unrelated control and pushed the conversation starters down. Render it
inside the composer border as the first row instead, and pass the
project through ChatForm so the memoised form still controls it.

The remove button no longer fades in on hover, since a control that only
appears on hover is unreachable by touch. Its popover opens upward with
a matching bottom-origin animation that honours reduced motion.

* feat: Rebuild the Change Project Dialog on the Searchable Combobox

The dialog used a bare select, so picking a project meant scrolling an
unsearchable native list capped at the default page of 25. Use the
searchable ControlCombobox, request the full first page, and disable
Save until the selection actually differs from the current project.

The combobox opts out of portalling so its search field sits inside the
dialog's focus trap and can be typed in, and the dialog is
overflow-visible so the list is not clipped.

Unassigning now lives on the menu's own Remove From Project action, so
the dialog no longer needs an empty option. Returning focus to the menu
button rather than the menu item fixes focus being lost on close, since
the item unmounts with the menu.

* feat: Add an Overflow Menu to Project Workspace Chats

Chats in a project workspace could only be opened. Managing one meant
finding it again in the sidebar, so add the same actions to the row:
change project, remove from project, and delete.

The row becomes a card matching the project cards, and the endpoint icon
is rendered at landing size rather than in a tinted tile. Its memo
comparison now uses areConversationListItemFieldsEqual, since the render
props comparison ignored fields the row displays.

* feat: Rework the Projects Sidebar Section

The section header duplicated the projects count and the New Project
action already on the dashboard, and spent a row on a chevron button
separate from its label. Collapse it to a single label toggle with an
All Projects action, and drop the per-project count that was hidden on
hover anyway.

The new chat action becomes a real link to the project-scoped URL, so it
can be middle-clicked and opened in a new tab, and modified clicks fall
through to the browser. On the new chat route it commits ?projectId
synchronously, because a deferred search param update lets ChatRoute see
a project-scoped draft on an unscoped URL and wipe it.

Row actions stay visible on devices without hover, where an action that
appears on hover cannot be reached.

* style: Widen the Projects Dashboard and Workspace Layout

The dashboard and the workspace sat on bg-surface-primary at different
max widths, so moving between them shifted the content and the shade did
not match the rest of the app. Put both on bg-presentation at max-w-6xl.

Project and chat cards gain a border, since colour alone separated them
from the background and that separation is thin in light mode. The
translucent blurred headers become opaque, and the scale-on-press
transforms are dropped.

In the workspace the edit and delete actions move out of the heading row
into their own group, so a long project name no longer pushes them
around.

* fix: Reopen the Change Project Dialog From the Conversation Menu

Closing the Ariakit menu in the same handler that opens the dialog made
the menu's own dismissal land on the freshly mounted Radix dialog, which
closed it again before paint, so Change project did nothing.

Leave the menu close to the dialog, which already receives setMenuOpen
and closes it once the assignment succeeds, matching the share and
delete handlers beside it.

* fix: Keep the Project Chat Menu Mounted While its Dialogs Open

Hiding the Ariakit menu in the same handler that opens Change project or
Delete restores focus to the menu trigger, which the dialog mounting
alongside it reads as an outside interaction and closes on, so the
action could do nothing.

Both dialogs already receive setIsMenuOpen and close the menu once they
finish, so leave the close to them, as the conversation menu does.

* perf: Fetch a Project's Chats Only Once its Row is Expanded

Collapse hides its children with CSS and inert rather than unmounting
them, so every project row's chat query ran on sidebar load, up to one
request per project, even with the whole section collapsed.

Gate the query on the row's expanded state. React Query keeps what it
already fetched, so reopening a row is still instant.

* refactor: Move the Bottom Popover Animation Into the Shared Primitive

ControlCombobox owns its popover animations in AnimatePopover.css, so
the upward variant it needs belongs there rather than in the
application stylesheet, where the control's appearance would diverge
from the package that ships it.

The app already loads the package stylesheet, so the animation resolves
the same way the existing variants do.

* fix: Stop Project Names and Descriptions Being Silently Truncated

The dialogs accepted any length and reported success, while the
persistence layer trimmed names to 100 and descriptions to 1000
characters, so reopening a project revealed text had been dropped with
no warning.

Share both limits from data-provider and cap the inputs at them, so the
fields stop where the server would have cut them and the rule has one
definition instead of the three it had.

* fix: Do Not Report the Loaded Page Size as the Project Total

The dashboard counted the projects fetched so far, so an account with
more than one page read as exactly one page's worth and the supposed
total grew with each Load more.

Show the loaded count as a lower bound while another page exists.

* chore: Satisfy the Static Checks for the Projects Rework

Sort the delete dialog's imports to the repository order, and drop the
three English keys this branch orphaned: the sidebar menu now says Edit
project, the dashboard labels its own sort control, and the change
project dialog no longer offers an Unassigned option now that removal
lives on the menu.

* fix: Highlight Only the Route Project in the Sidebar

A leftover conversation project was still lighting a second row after
opening another project's workspace. Prefer the workspace route, and
only fall back to the conversation project outside that view.
2026-08-16 11:06:18 -04:00
Danny Avila
bce93f9c55
🎯 refactor: Infer Agents Endpoint for Model Specs Naming an Agent (#14889)
* 🎯 fix: Infer Agents Endpoint for Model Specs Naming an Agent

A model spec whose preset names an `agent_id` but omits `endpoint` was
unusable. `isModelSpecEndpointMatch` compares the request's endpoint to
`preset.endpoint` by strict equality, so an undefined endpoint matched
nothing and every request selecting the spec was rejected with a bare
`Model spec mismatch` — an error naming neither the spec nor the missing
field.

The selector had the matching half of the same gap: `handleSelectSpec`
read `preset.endpoint` directly, so it sent no endpoint and skipped
assigning `agent_id` to `model`. Fixing only the server would leave the
request malformed, so the resolution is shared between both.

- Add `resolveModelSpecEndpoint` to `librechat-data-provider`, inferring
  the agents endpoint when a preset names an agent and none is set. An
  explicit `endpoint` always wins, so configured specs are unaffected.
- Use it for endpoint matching and in the selector, so the menu and the
  request pipeline resolve a spec identically.

* 🔁 refactor: Materialize Inferred Spec Endpoints at Config Load

The review showed the lazy-resolver approach was unsound end to end:
config validation rejected an endpoint-less spec before the resolver
could ever run (`tPresetSchema` requires the `endpoint` key), and the
resolver was applied at 2 of ~8 read sites, leaving selection handlers,
startup presets, access filters, and provider-key reachability reading
the raw preset.

Materialize once at the boundary instead:

- `tModelSpecPresetSchema` now makes `endpoint` optional (`nullish`).
  This is barely a widening — `endpoint: null` already validated — and
  only for model-spec presets; `tPresetSchema` is untouched.
- `materializeModelSpecEndpoints` writes each spec's resolved endpoint
  back onto its preset. `createAppConfigService` applies it at both
  effective-config assembly points — YAML base load and DB-override
  merge — so admin-panel specs stored in override documents are covered.
  Identity-preserving, so cached configs see no new references when
  nothing needs filling in.
- Every consumer now reads complete specs; the client's lazy resolve in
  `handleSelectSpec` is reverted to a raw read. `getModelSpecPreset` and
  the two hand-rolled preset constructions resolve the endpoint
  explicitly, which the narrowed preset type now enforces at compile
  time for any `TPreset`-shaped destination.
- `isModelSpecEndpointMatch` keeps the resolver as request-time defense.

* 🩹 fix: Materialize Spec Endpoints Before the YAML Missing-Endpoint Guard

`processModelSpecs` warns and skips any spec whose preset lacks an
endpoint, and it runs inside `loadBaseConfig` — so the previous commit's
materialization received a YAML list from which the inferable spec had
already been dropped. Only DB-override specs (merged after the guard)
actually benefited.

- Materialize at the entry of `processModelSpecs`, so inference happens
  before the guard and YAML agent specs survive it. The guard keeps
  skipping genuinely endpoint-less specs. The `createAppConfigService`
  calls stay: the base-path one guards alternate `loadBaseConfig`
  implementations, the merged-path one covers override documents, and
  both are identity-preserving no-ops when specs are already complete.
- Constrain the widened schema: omitting `endpoint` is only legal when
  the preset names an `agent_id`. A preset with neither validated as a
  hard error before the key became optional, and silently accepting it
  would trade that startup-time error for a dead spec. An explicit
  `endpoint: null` (valid before this PR) keeps validating.

* 🩹 fix: Infer Only From Non-Empty Agent IDs, Never Over Explicit Null

Two edge cases in the inference contract:

- `agent_id: ''` (what a form-backed writer persists for an untouched
  field) passed the nullish checks, validating and materializing a spec
  that names no agent. Both the refinement and the resolver now require
  a non-empty id, so such config fails validation loudly instead of
  producing a selectable spec that cannot work.
- `endpoint: null` alongside an `agent_id` was treated as inferable,
  silently activating a spec that validated — and was skipped — before
  this PR. An explicit null is a statement, not an omission: the
  resolver now infers only when the key is absent, preserving prior
  behavior for previously valid configs.
2026-08-16 11:04:52 -04:00
Danny Avila
06bf324cf0
🛤️ feat: Per-Agent Code Execution Routing With Stateful Session Scopes (#14848)
* feat: route code execution per agent profile

* chore: sort execution profile imports

* test: preserve stateful environment literal types

* fix: isolate stateful code environments by user

* fix: preserve per-agent code routing end to end

* fix: route code priming by execution profile

* fix: isolate code profile lifecycle state

* fix: preserve mixed-profile code resources

* fix: complete stateful skill routing
2026-08-16 09:42:15 -04:00
Danny Avila
1d789c41a5
🧩 fix: Normalize MCP UI Resource Rendering (#14868)
* fix: normalize MCP UI resource rendering

* fix: filter unsupported MCP UI resources

* fix: preserve MCP UI marker examples

* fix: handle MCP UI resource edge cases

* fix: harden MCP UI marker sanitization

* fix: scope MCP UI marker sanitization

* fix: parse MCP UI marker contexts

* fix: align MCP UI sanitizer parsing

* fix: match MCP UI renderer syntax

* fix: align blockquote marker spans

* fix: decode MCP UI text node sources

* fix: sanitize nested subagent markers

* fix: bound MCP UI sanitizer traversal

* fix: keep MCP UI marker mapping linear

* style: sort security patch imports

* fix: harden nested MCP UI sanitization

* fix: mirror citation cleanup for MCP UI markers

* fix: clean decoded citation markers

* fix: clean assembled citation markers

* fix: align MCP marker sanitization with rendering

* fix: match persisted MCP marker render paths

* fix: preserve highlighted citation boundaries

* fix: align MCP markers across content renderers

* fix: preserve citation renderer boundaries

* fix: match legacy thinking trim semantics
2026-08-15 12:49:59 -04:00