Commit graph

15 commits

Author SHA1 Message Date
Danny Avila
08f9d7f7e0
🧵 feat: Stream a Background Subagent Child Through Its Own Generation Job (#18169)
* 🫀 feat: Let a Job Owner Keep a Silent Run Alive

A generation job's idle clock advances only with emitted chunks, so a run that
works without output for 20 minutes is reaped while it is still alive.
touchJob refreshes the idle clock and the storage lifetimes an append would
have extended, fenced on the generation epoch and limited to the replica that
owns the runtime.

* 🧶 refactor: Share One Fold From Child Frames to Content Parts

readSubagentContent folded public child frames into content parts inline.
The fold is now a synchronous function a second caller can use, so a frame
can be folded and published without yielding in between.

* 🏷️ feat: Mark the Generation Job of a Background Subagent Task

Job metadata gains subagentTaskId and subagentParentConversationId, set only
by the subagent task store. A marked job is a child-thread job, so it stays
out of the active chat list, and isBackgroundChildJob tells it apart from an
event-bound child, whose job the chat controller owns.

* 🧵 feat: Run a Background Child as a Resumable Generation Job

createChildGenerationHost opens a generation job keyed by the child thread id
and feeds it the child's public step frames. Text deltas that queue behind a
slow write merge instead of dropping, the terminal state is claimed before the
caller persists and the final event is published only afterwards, and no
stream failure can fail the child: the run and its result carry on without a
live view.

* 🪢 feat: Stream a Background Child Through Its Own Generation Job

The subagent task store takes a childGenerations dependency. Once a child's
thread and input row are durable it opens a job keyed by the thread id, feeds
it each public frame, refreshes its idle clock on every lease renewal, and
settles it around its own persistence, so a reader sees the final event only
after the result row exists. A cancel, a failure and a job aborted from
outside all end both the child and its stream.

endpoints.agents.subagentActivity.generationJobs turns it off. A stream that
cannot be opened or written never fails the child.

* 🔌 feat: Give the Server's Task Store Its Child Stream

The host wiring builds the child generation host from the generation job
manager and hands it to the subagent task store, so a background child's
stream is reachable in the running server.

* 🚧 fix: Refuse Chat Abort and Steer on a Background Child's Job

The chat abort route saves a user row and a partial response and upserts the
conversation, and the steer route queues text only the chat controller drains.
Both authorized on the job's owner alone, so either could act on a background
child's job and write a chat-shaped history into a view-only thread, or accept
a steer nothing would ever apply. Both now answer 409 for a job the subagent
task store owns; the child is stopped and steered through its control
endpoint.

* 🧹 fix: Evict a Finished Job Before a Live One at Capacity

The in-memory job store made room by deleting its oldest job whatever its
status, so a full store could end a run somebody was still waiting on. It now
takes the oldest finished job first. Also covers a job manager that shuts down
while a background child runs: the child still persists its cancellation.

* 🛰️ test: Read a Background Child's Stream From Another Replica

Against real Redis: a second replica resumes with everything the child has
streamed, receives only newer deltas live, then the final event; a child that
streams past the manager's 5,000-event early buffer before anyone attaches
still resumes whole on both replicas; and the owner's touch keeps a silent
child's job alive.

* 🧽 chore: Sort Imports in the Child Stream Files

* 🧪 test: Expect the Child Stream Lever in the Activity Defaults

* 📜 feat: Store What a Background Child Streamed as Standard Content (#18173)

A child's result row carried its text and a bounded activity projection, not
the content parts every agent message stores, so a reader had to rebuild the
child's output from the projection. The row now also carries what the child
streamed, on completion, failure and cancellation, in the standard shape.

endpoints.agents.subagentActivity.generationContentBytes bounds it (4 MiB by
default, 0 stores none). Content over the bound is left off, so the row always
fits and keeps its text and projection as before.

* 🛟 fix: Keep a Stalled or Failing Child Stream From Touching Its Child

Four ways the stream could still reach the run it only observes. A frame that
could not be copied threw into the child's own progress report. A job store
that never answered held the child's start behind job creation and its result
behind the terminal claim. And a stream that had stopped being written was
never ended, so its job read as a live child for twenty minutes after the
child finished.

Frames are now copied defensively, opening and settling are bounded, a job
that arrives after its child stopped waiting is ended, and a detached stream
is still touched and still settled with its full final event.

* 🧯 fix: Let a Child Stream Observe Its Run Without Ever Stopping It

Review round 1.

A failed Redis append retires the job's runtime by aborting its controller,
and the child stream forwarded every abort of that controller to the task
store as a cancellation. A stream-store outage therefore stopped the child's
model and tool work. The stream no longer signals the child at all: deletion,
shutdown and the control endpoint already stop a child through the task
store, and an aborted, reaped or retired job now only ends the live view.

The in-memory store's capacity eviction preferred any terminal job, including
one that still owed persistence, a host action, a schedule reconciliation or
a provider drain. Those are no longer treated as finished.

The chat abort route's refusal for a background child's job moves out of the
CJS route into createBackgroundChildAbortGuard, registered ahead of it.

* ♻️ fix: Replace a Dead Background Child's Job on Its Thread

A background child's job outlives an owner that crashed, or a settlement that
could not reach the store, for up to twenty minutes. A continuation of the
thread in that window was refused its own stream, because the dead job still
read as live. The caller of open holds the thread's lease, so a background
child's job still marked live there has lost its owner: it is replaced and its
readers are told to refetch. A live job that is not a background child's, such
as an event-bound child paused for approval, is still never replaced.

* 🧱 fix: Fail the Abort Guard Closed and Budget Content Against the Whole Row

Review round 2.

The abort guard passed the request on when its job lookup failed, so a
retried lookup in the route could run the chat-shaped abort on a child the
guard never saw. A failed lookup now ends the request at the error boundary.

A child's streamed content was checked against its own limit only. A result
row also carries the text and the transcript, so content that passed the
limit could push the document past MongoDB's 16 MiB and turn a successful
child into a failed one. The content is now the optional field: it is left
off whenever the row's large fields together would exceed the budget.

Also accounts for the guard's job read in the two abort route specs that
sequence it, and gives the limiter spec's api mock the guard.

* 🪶 refactor: Settle a Child Stream Once and Fold a Frame It Cannot Copy

The task store settles a child's stream on its outcome path and again in its
cleanup. The second call made another terminal claim that could only lose;
the stream now remembers that it claimed and only persists. A frame that
cannot be copied is still folded from the original, so the stored content
stays whole when the stream detaches over it.

* 🧪 test: Cover the Child Stream's Failure Branches and the Content Fold

The change-risk report showed the stream's fail-open branches untested: a
terminal claim that lands after its wait, a claim or a final publish that
throws, the settle timeout, the backlog limit, and a rejected touch. Each now
has a case. The shared content fold gets direct cases for a closed step, a
commentary phase and an applied steer.
2026-11-15 08:35:08 -05:00
lia-by-librechat[bot]
ff962fcaa2
📔 fix: Recover Subagent Content With Shared Chat Rendering (#17290)
* test: cover retained subagent chat history parity

* feat: Define Task-Scoped Subagent Content Recovery Contract

* style: Format Retained History Regression Tests

* feat: Add Task-Scoped Subagent Public Content Recovery Foundation

* feat: Render Subagent Turns With Standard Chat Content

* fix: Bound Recovery Capture and Keep the New Writer Opt-In

* feat: Recover Subagent Chat Content From Committed Task History

* fix: Preserve Recovered Metadata and Shared Reader State

* fix: Preserve Journal Retry Bytes, Unicode and Recovered Phases

* fix: Catch Up Terminal Child Content Beyond Pinned Pages

* fix: Bound Child Content Settlement and Restore Preparation Timing

* fix: Preserve Retained Child History After Late Append Completion

* fix: Reconcile Late Child Commits and Preserve Terminal Content

* fix: Retain MCP Scope While Detached Children Execute

* test: Assert Shared Task Ownership Across Request Adapters

* fix: Preserve Subagent Recovery Across Flush and Deletion Races

* test: Enforce Indexes in Subagent Deletion Regressions

* fix: Respect Revoked Recovery When Cached Content Exists

* fix: Honor Recovery Revocations and Hide Read-Only Approvals

* test: Type Captured Tool Metadata in Approval Regressions

* fix: Keep Recovery Fallbacks Read-Only

* fix: Bound Subagent Recovery Work and Preserve Terminal Answers

* fix: Reconcile Late Content Writes and Preserve Selected Outcomes

* fix: Preserve Durable Outcomes Independently of Capture Completeness

* fix: Reconcile Serialized Subagent Text Blocks Without Duplicate Outcomes

* fix: Preserve Journal Cleanup Scope Before Conversation Erasure

---------

Co-authored-by: Lia <lia@librechat.ai>
2026-10-22 15:32:40 +02:00
lia-by-librechat[bot]
bb0543fd15
📼 fix: Replay Running Subagent Activity (#16541)
* 📼 fix: Replay Running Subagent Activity

* 📼 fix: Preserve Activity Across Rollouts and Reconnects

* 📼 fix: Backfill Missing Foreground Activity

* 📼 fix: Fence Replay and Drain Admitted Activity

* 📼 fix: Close Subagent Replay Review Gaps

* 📼 fix: Resume Activity After Replay Recovery Expires

* 📼 fix: Use Singular Subagent Omission Copy

* 📼 fix: Preserve Public Replay and Displayed Activity

* 📼 fix: Write Expanded Live Activity Before Queuing

* 📼 fix: Track Disjoint Subagent Replay Coverage

* 📼 fix: Preserve Tool Details and Omission Totals

* 📼 fix: Recover Tool Timing During Replay Backfill

* 📼 fix: Reconcile Covered Replay Timing Metadata

* 📼 fix: Restore Message Phases During Replay Backfill

---------

Co-authored-by: Lia <lia@librechat.ai>
2026-10-02 13:24:47 -04:00
Danny Avila
7238be3e5e
⚡ fix: Deliver Waiting Completion Wake-ups the Moment They Are Ready (#16339)
* 🐢 fix: Back Off Waiting Completion Wake-ups and Deliver Them When Ready

* fix: Signal every readiness path and make the completion wait cap configurable

* fix: Mark held completion deliveries instead of pipeline expedite, and announce store-won approval expiry

* fix: Scope settle expedites to the resumed conversation and close the remaining signal gaps

* fix: Announce a won approval expiry once and release subagent wake-up registrations

* 🧪 ci: Count System Instructions in Mock Model Usage (#16350)

The e2e fake model reported input tokens for the chat messages alone, because the SDK hands a test override model the pruned messages without the systemRunnable pipe. Once #16301 gave every memory-enabled chat a system prompt, the calibrated context snapshot put used tokens below the instructions and the gauge dropped its Messages row. Count input over the complete prompt, as a real provider bills it.

---------

Co-authored-by: Lia <lia@librechat.ai>
2026-09-24 22:28:28 -04:00
Danny Avila
2d794a0439
💓 fix: Detect Silently Dead Redis Sockets With Deadline-Bounded Heartbeats (#15539)
* fix: Detect silently dead Redis sockets with deadline-bounded heartbeats

A Redis peer can vanish without a FIN or RST (a dropped NAT entry, a proxy
failover, a migrated VM). The socket stays `ready` from ioredis's point of
view, no error fires, and every command on it waits for the kernel's
retransmission timeout — about fifteen minutes at Linux defaults.

Observed in production: while the main client's socket was dead,
POST /api/agents hung until Cloudflare returned a 524; while the subscriber's
socket was dead, every attach failed with "Timed out synchronizing Redis
subscription" for sixteen minutes across five conversations.

`REDIS_PING_INTERVAL` could not help — it awaits ping() with no deadline, so
the probe just joins everything else that is waiting.

- Add startRedisHeartbeat: races a PING against a deadline and destroys the
  socket when the probe goes unanswered, so ioredis reconnects, replays queued
  commands, and re-subscribes a subscriber's channels.
- Destroy the stream rather than disconnect(true), whose FIN a vanished peer
  never acknowledges; cluster clients fall back to the regular reconnect.
- Add createIoRedisSubscriber so every dedicated subscriber gets an error
  listener and its own heartbeat, and route the generation-stream and both
  subagent-routing subscribers through it.
- Set ioredis keepAlive from REDIS_KEEP_ALIVE (default 10s).
- Add REDIS_PING_TIMEOUT, REDIS_SUBSCRIBER_PING_INTERVAL and REDIS_KEEP_ALIVE,
  documented in .env.example.
- Give the fake RESP server a silent mode and a SUBSCRIBE reply so the
  dead-peer case can be tested against real clients.

* fix: Probe each cluster node and refuse a non-positive heartbeat deadline

A cluster-level PING is routed to one node, so a healthy reply proved nothing
about the other sockets. The heartbeat now resolves its targets each tick —
every node for a cluster, the client itself otherwise — and races an
independent deadline per target, tearing down only the socket that stops
answering. A non-positive deadline disables the heartbeat with a warning
instead of forcing a reconnect on every tick, and the single-probe test no
longer depends on wall-clock slack.
2026-09-03 09:45:16 -04:00
Danny Avila
a5fd00cd7a
🎼 feat: Unify Agent Turn Execution Behind One Immutable Plan (#15296)
* feat: unify agent turn execution planning

* fix: preserve automatic subagent activity handles

* fix: gate subagent wakeups for ephemeral parents

* fix: scope subagent wakeup guidance by agent
2026-08-27 22:26:46 -04:00
Danny Avila
d641c398d5
🧳 fix: Port Subagent Control Receipt Writes to DocumentDB-Safe Operators (#15171)
* fix: harden subagent control receipt persistence

* fix: harden durable subagent control replay

* fix: await terminal control receipts on shutdown

* fix: close subagent control replay races

* test: type stale-owner transport fixture

* fix: quiesce durable subagent controls

* test: await subagent shutdown durability boundary

* fix: serialize durable subagent controls

* fix: fail shutdown on cleanup errors

* fix: report cancellable result availability accurately

* fix: fence subagent control receipt ownership

* fix: close distributed control receipt races

* test: type control receipt race fixture

* chore: require authoritative control receipts

* fix: close subagent control lifecycle races

* style: separate control reservation member

* test: harden subagent settlement wait

* fix: preserve authoritative control replay state
2026-08-24 20:05:39 -04:00
Danny Avila
a997275902
🧾 feat: Persist Authoritative Subagent Control Receipts (#15168)
* feat: persist subagent control receipts

* fix: require control receipt persistence

* fix: preserve authoritative control history
2026-08-24 11:39:56 -04:00
Danny Avila
8969ee4b18
🎚️ feat: Configure Agent Event Runtime in YAML (#15128) 2026-08-23 02:37:33 -04:00
Danny Avila
1de88e7e91
📨 feat: Continue Bound Child Agents from Events (#15112)
* feat: add authenticated agent event ingress

* style: sort agent ingress imports

* fix: harden agent event ingress

* fix: bind event provenance to API keys

* fix: inspect event input with legacy PII filters

* fix: scope event status reads to source keys

* fix: bind event status reads to remote sources

* feat: add bound event-driven child turns

* fix: harden event-bound child continuations

* fix: satisfy event binding type contracts

* fix: close event actor lifecycle races

* fix: harden event actor dispatch continuity

* fix: fence event actor resume lifecycle

* fix: bind event actor state to lifecycle

* fix: preserve cascade write outcomes

* test: type cascade failure injection

* style: sort cascade test imports

* fix: harden event child lifecycle boundaries

* fix: make event cleanup retryable

* fix: annotate event retention clock

* fix: reconcile partial cascade metadata

* fix: recheck event binding expiry on resume

* fix: fence event actors by retention deadline

* fix: close event actor lifecycle races

* fix: harden event child lease acquisition

* fix: lazy-load event child lease adapter
2026-08-23 01:15:57 -04:00
Danny Avila
d3e70159ca
📡 feat: Stream Detached Subagent Activity (#15111)
* feat: stream detached subagent activity

* fix: annotate activity stream limits

* fix: isolate subagent activity imports

* fix: harden detached subagent activity lifecycle

* test: cover synchronous activity transport failure

* test: include required subagent activity identity

* fix: identify and reconnect subagent activity events

* fix: bound subagent activity lifecycles

* fix: close subagent activity handoff races

* fix: bind and synchronize activity subscriptions

* fix: detect fresh activity attachment

* fix: complete activity synchronization handoff

* fix: bind activity sync and failure circuits

* fix: expose subscription-bound synchronization

* fix: fence activity reconnect publications

* test: make detached timeout settlement deterministic

* fix: fence Redis activity attachments

* fix: close failed activity streams

* perf: reuse fenced activity frontier

* style: sort subagent thread imports

* fix: preserve queued subagent activity

* test: type activity publication counter

* fix: disconnect subagent activity subscriber

* fix: close background activity lifecycle gaps

* fix: preserve streamed activity spacing

* fix: preserve bounded live subagent activity

* fix: merge durable subagent activity safely

* fix: model detached activity coverage

* fix: type detached activity inputs

* fix: order overlapping subagent activity

* chore: sort activity test imports

* fix: buffer subagent activity handoff gaps

* fix: flush activity after parent close

* fix: advance closed activity suffixes

* fix: preserve detached activity ordering

* fix: close detached activity delivery races

* fix: bound shared Redis subscriber readiness

* fix: expire shared Redis subscription readiness

* fix: clean up late Redis subscriptions

* fix: preserve late Redis subscription fallback
2026-08-22 09:45:27 -04:00
Danny Avila
a5cb041f47
🕊️ feat: Yield to Subagent Completion Wakeups (#15066)
* 🕊️ feat: yield to subagent completion wakeups

* ⚡ fix: bound wakeup status guidance
2026-08-21 01:45:42 -04:00
Danny Avila
5e3c680761
🪃 feat: Wake Parent Agents on Child Completion (#14975)
* feat: wake parent agents on child completion

* wip: harden child completion wakeup lifecycle

* fix: close the completion-wakeup static failures

Type the durable-claim store fixture, the continue-envelope test helper,
and the terminal message's task metadata so the wakeup suites compile
against the shapes they actually exercise. Replace `Array.prototype.at`,
which the package target library does not provide.

Capture the prepared child thread in a non-optional local before the
provider callback closes over it, and narrow the trigger envelope itself
on `mode === 'continue'` rather than a separately copied mode, so reading
the continue target is sound. Lift the parent-message fallback out of a
nested ternary into a named resolver.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H1cCMDrTWaRNkmtKjpWELZ

* test: cover the active-predecessor admission fence

The Redis job-creation call gained a thirteenth scalar argument, so the
spec helper reconstructed the HSET pairs one slot early and rebuilt an
invalid job hash; three creation tests failed on that alone.

Give the fence itself direct coverage in both store adapters, which it
had none of despite deciding whether an automatic continuation may
replace a live parent turn. Each proves a running and a requires_action
predecessor are refused with the state a controller needs for a finite
409, that an absent or settled predecessor is admitted, and that an
ordinary user turn without the policy still replaces its predecessor.
The Redis case also asserts a refused continuation leaves the parent's
durable job and chunks untouched.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H1cCMDrTWaRNkmtKjpWELZ

* fix: harden completion wakeup rollout and claims

* fix: close completion wakeup race windows

* test: keep the child store fixture exact

* fix: close final subagent wakeup gaps

* fix: preserve ambiguous completion claims

* fix: release pre-admission wakeup claims

* fix: stabilize subagent completion recovery

---------

Co-authored-by: Claude <noreply@anthropic.com>
2026-08-19 12:18:31 -04:00
Danny Avila
259f1e0c32
🛰️ feat: Route Live Subagent Controls Across Replicas (#14971)
* feat: route live subagent controls across replicas

* fix: initialize task routing in cluster workers

* fix: harden cross-replica task routing

* fix: expire routed task owners independently

* fix: close cross-replica routing edge cases

* fix: bound owner refresh and close routed cancellation gaps

Refresh owned task registrations in bounded parallel batches so a full
heartbeat pass stays well inside the 30-second directory lease instead of
serializing one Redis EVAL per registration.

Route conversation-deletion cancellation through a dedicated owner-side
scope operation. The owner applies the deletion predicate to its complete
local task set, so a scope holding more children than the model-facing
list cap no longer leaves live executors running after their parent is
removed.

Key a consumed claim's retained response by its operation rather than by
one caller's correlation id, so a later poll recovers a terminal result
whose responses were all lost. Live claim statuses stay uncached so a
poll always observes the task's current state.

Type the model-facing `maxLength` bounds with a narrow local string
schema; the SDK's JsonSchemaType does not declare the keyword, and the
runtime checks continue to enforce the same limits.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H1cCMDrTWaRNkmtKjpWELZ

* fix: retain claimed results apart from control replays

A consumed claim is the only routed response whose loss destroys data, so
it no longer shares one bounded cache with control replays that unrelated
command traffic can evict. Claims are retained under their own budget, and
the requester acknowledges a result it received so the owner releases the
copy immediately instead of holding it for the full replay window.

Resolve the post-delete cancellation pass from durable leases. The deleted
conversations cannot be read back, so re-reading each one only scaled the
cascade while probing the owner directory once per removed id; one lease
read now resolves every live child address instead.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H1cCMDrTWaRNkmtKjpWELZ

* fix: never consume a result the owner cannot replay

Retention for consumed claims is bounded, so a burst of undelivered results
could evict an earlier one and lose it for good. The owner now admits a claim
only while it can retain a worst-case result, and refuses the routed claim
otherwise instead of consuming it, leaving the result on the task for a later
poll. Retained claims are never displaced; control replays keep evicting.

Key a control replay by the command itself rather than by one caller's
correlation id. The transport's own retry reuses a single envelope, but a
caller that saw the owner as unavailable reissues the command under a new id,
which steered, queued, or interrupted the child a second time.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H1cCMDrTWaRNkmtKjpWELZ

* fix: own a claimed result until it is acknowledged

A consumed terminal result is task-owned state, not a cache entry. It now
carries no expiry at all: the owner holds it until a caller acknowledges
receipt, and only then is it released. Retention stays bounded by the
existing admission gate, which refuses a claim the owner could not keep
rather than consuming a result it might drop.

Identify a control by the caller's invocation instead of by its content.
The tool mints one id per invocation and routing carries it, so a routed
retransmission of that invocation replays the owner's result while two
deliberate identical commands arrive under distinct ids and both apply.
Content-derived identity could not tell those apart and would have
answered the second from a stale snapshot.

Wait for the dpkg frontend lock in the best-effort Playwright font step.
Its timeout kills npx while the apt-get it spawned keeps the lock, which
then failed the fatal Redis install and ended the MCP replica jobs before
any test ran (#14983).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H1cCMDrTWaRNkmtKjpWELZ

* fix: treat acknowledgement as part of delivering a result

Publishing an acknowledgement once and ignoring the outcome meant a result
could be reported as delivered while the owner never learned it could let
go, and since that retention neither expires nor evicts, enough lost
acknowledgements would fill it and refuse every later remote claim.

An acknowledgement is now confirmed: publishing to zero subscribers is not
success, it retries inside the ordinary request window, and a claim whose
acknowledgement cannot be confirmed reports the retryable unavailable path
instead of handing back a result the owner still holds. A later poll
recovers that result and acknowledges it, and releasing is idempotent.
Owner registration also outlives the task while a result is unacknowledged,
so the retained result cannot become unreachable.

Take the control invocation identity from the provider's tool-call id
rather than minting one per execution, so replaying the same tool call
stays idempotent while two distinct calls with identical payloads both
apply.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H1cCMDrTWaRNkmtKjpWELZ

* style: sort the widened node:crypto import

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H1cCMDrTWaRNkmtKjpWELZ

* fix: own control invocations and cancellation plans at the task seam

Applies one logical control exactly once for its owning task rather than
in the transport, so a local caller and a routed caller of the same
invocation agree, and reusing an invocation id for different content is
refused instead of silently applied. Invocation identity now comes from
the run, agent, and provider tool-call id hashed to a bounded 32
characters, so a repeated `call_0` never bleeds across tasks and no id can
overrun the routed bound.

Cancellation for conversation deletion is now resolved into a plan while
those rows are still readable, then replayed against the owner directory
after the cascade is deleted. Owner registration is awaited before any
provider work, so a child that cannot be addressed never starts.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H1cCMDrTWaRNkmtKjpWELZ

* style: separate the control invocation map from the next member

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H1cCMDrTWaRNkmtKjpWELZ

* fix: close subagent deletion, claim, and control invocation gaps

Bulk conversation deletion now runs behind a durable owner admission fence.
Draining alone could not close the race: a child admitted on another replica
after the drain read its leases would start provider work against a parent
about to disappear. The fence is written before any lease is read and each
child revalidates it after its own lease is written, so one of the two always
observes the other. It expires on its own, so a process lost mid-deletion
cannot leave an account unable to run subagents.

A terminal child result is no longer kept alive in the owning replica's
memory until someone acknowledges it. Collection is recorded durably on the
child's own message against the polling invocation, so the poll whose
response was lost recovers its own result while a different invocation is
told the result was already collected. Owner-side retention returns to an
ordinary bounded cache that expires, which is what abandoned polls needed:
they can no longer occupy claim capacity until the process restarts.

The deletion drain now cancels each task under one invocation held for the
whole drain, stops re-sending once the owner answers, and retries only
deliveries it could not confirm. A routed control replay also validates the
command fingerprint, so one invocation id carrying different content reaches
the owner to be refused instead of collecting the earlier command's success.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H1cCMDrTWaRNkmtKjpWELZ

* test: assert the drain's calls before restoring its spies

Restoring a spy also clears its recorded calls, so the drain assertions
ran against an emptied mock. Formats the durable claim method tests.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H1cCMDrTWaRNkmtKjpWELZ

* fix: close the follow-on gaps in the deletion fence and result claim

The admission fence now carries an ownership token, so an overlapping
deletion's fence is never lifted by the one that finishes first, and both
fence writes invalidate the cached auth user document. It also covers the
other bulk-delete path: `DELETE /` with no conversation filter removes every
conversation, so it runs behind the same fence rather than a bare drain.

The durable record now decides who holds a one-shot result. An owner replaying
a retained response could hand the same terminal claim to a second invocation;
that invocation is told the result was already collected, while the one that
consumed it still recovers its own. A task with no durable record to arbitrate
keeps whatever the owner answered.

Drain cancellation treats `not_found` as unconfirmed: a missing registration
while the durable lease is still live means the child may be running, so the
command is retried under its invocation once the owner republishes itself.
Control fingerprints are hashed, so retaining one per invocation costs a fixed
few bytes instead of a bounded message.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H1cCMDrTWaRNkmtKjpWELZ

* fix: hold every deletion fence and keep live idempotency records

An owner now holds one admission fence per concurrent bulk deletion instead
of one at a time, so admission reopens only when the last deletion finishes
regardless of completion order. Expired fences are pruned as new ones arrive
and the set is bounded, so an abandoned fence cannot accumulate or lock an
account out.

A failed durable claim write is no longer read as an absent record. Handing a
terminal result over without recording its claimant would let another
invocation collect the same one-shot output once the database recovered, so
the collection reports the retryable unavailable path and leaves the result
for a later poll.

Control invocation records now evict tasks the store no longer holds before
live ones, over a bounded scan. Dropping a live task's record would let a
caller retry apply its queue, steer, or interrupt a second time once the
transport replay had also expired.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H1cCMDrTWaRNkmtKjpWELZ

* fix: keep the deletion fence portable and never drop a live record

The admission fence is written with plain update operators again. DocumentDB
rejects pipeline-form updates, and this runs before any deletion, so the
pipeline form would have failed both bulk-delete endpoints outright on a
supported database target.

An excess deletion is now refused rather than silently displacing the oldest
active fence, which would have reopened admission for a deletion still
running. Expired fences are pruned before the cap is tested, so only genuinely
concurrent deletions count against it.

Control invocation records now sweep every settled task's entry when the
window fills, and a window of entirely live records refuses the new control
before touching the child instead of evicting one. Applying a command with no
room to record it would let the caller's own retry apply it twice.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H1cCMDrTWaRNkmtKjpWELZ

* fix: hold the fence, bound recovered results, and expire stale commands

The admission fence is renewed for as long as its deletion runs, so a very
large account or a stalled database cannot let it lapse while conversations
are still being removed. Only the deletion's own fence is renewed, and the
renewal stops with the operation.

Cancellation now covers every conversation the cascade removed, not only the
ones a plan named: a grandchild lives in its own parent's scope, which a plan
naming the deleted root never reaches.

A routed request carries the deadline its caller waits for, and an owner drops
one that arrives past it. A publisher disconnected mid-request queues the
envelope offline and delivers it after the caller was told the owner was
unavailable, which would otherwise steer a child the caller believes untouched.

A result recovered from its durable child message is bounded like a routed one.
The message keeps the child's untruncated output, so recovery could otherwise
return far more than the routed result limit allows.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H1cCMDrTWaRNkmtKjpWELZ

* test: size the fence window so a renewal can be observed

The renewal test set a 30ms drain timeout but the five-minute grace window
dominates it, so the interval was 100 seconds and no renewal could fire
inside the test's deletion. The grace window is an option now, matching the
store's other timings, and the test sizes the window to 90ms.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H1cCMDrTWaRNkmtKjpWELZ

* fix: wire the durable claim method and close the fence follow-ons

The production store never received `claimSubagentTaskResult`, so every
terminal result would have surfaced as unavailable once a task settled. The
host wires that object from JavaScript, where the factory's parameter type
checks nothing, so the factory now refuses a store missing any method it
calls rather than failing at the first claim.

The routing transport takes a dedicated publisher with the offline queue
disabled. The shared client held commands issued during a disconnect and
delivered them after the caller had given up, which the request deadline
narrowed but could not close inside the clock-skew allowance.

Fence renewal invalidates the cached auth document like the fence and release
paths, and a renewal reporting its entry gone re-takes the fence instead of
letting the deletion run on unfenced. The post-delete cancellation retries a
transiently unreachable owner: the conversations are already gone, so it is
the only pass that can still stop a late-admitted child.

A replaced replay entry no longer leaves its bytes counted, which would have
inflated the cache's total until unrelated responses were evicted.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H1cCMDrTWaRNkmtKjpWELZ

* test: wait on observed lease renewal instead of a fixed delay

The shared-lease renewal test held a 60ms lease and slept 100ms before
asserting an overlapping worker was refused, so a loaded runner that
starved the 10ms heartbeat past the TTL let the lease lapse and the
second worker run. Spy on acquisition and renewal, then wait until a
renewal succeeds past the acquired lease's own deadline — direct
evidence the heartbeat carried it past expiry, with no timing
assumption — and give the lease enough headroom that a stalled timer
no longer decides the outcome.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H1cCMDrTWaRNkmtKjpWELZ

* fix(agents): close the routing, fence, and cache gaps found in review

Five separate seams, each with its own failure:

`Cluster.duplicate` reads its first argument as a startup-node list and
its second as the overrides, unlike `Redis.duplicate`, so the publisher's
`enableOfflineQueue: false` was silently dropped under
`USE_REDIS_CLUSTER` and a command issued mid-disconnect could still
reach a child after its caller was told `unavailable`. Route both
through `duplicateIoRedisClient`.

The control window's capacity refusal ran before the store knew whether
it owned the task, so unrelated local load could veto a cancellation
bound for another replica. Establish that the task is local first and
leave a remote one to its owner's window.

`clearInterval` stops only future fence renewals. One already waiting on
the database could resolve after the release, read its own lifted fence
as expiry, and write a replacement that nothing remained to lift —
closing subagent admission for the account until it aged out. Track the
in-flight renewal, refuse overlapping passes, and await it before
releasing.

Every owner bounds its own task list, but the aggregation appended each
batch whole, so the model-facing list grew with the number of replicas
holding the scope. Cap the merged list while still reading every reply
for the stale-registration sweep.

The admission-fence prune commits independently of the fence that
follows it, so a refused or failed push left the cached auth document
describing entries the collection no longer held. Invalidate whichever
way the second write goes.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H1cCMDrTWaRNkmtKjpWELZ

* fix(agents): cap the merged task list the poll tool actually reads

Each owner bounds its own reply and the remote aggregation bounds their
sum, but `listTasks` merged that bounded remote list with however many
children this replica owns and returned it whole. `check_background_task`
could therefore still receive roughly twice the advertised cap. Bound the
deduplicated, sorted result and export the cap so both seams share one
number.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H1cCMDrTWaRNkmtKjpWELZ

* test: admit every task the merged-list cap test starts

The base store admits ten concurrent runs per scope by default, so
starting 150 at once left most refused for capacity and the assertion
never reached the merge it was written to check. Raise the cap for this
store only; admission is a different invariant with its own tests.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H1cCMDrTWaRNkmtKjpWELZ

* fix(agents): let a deletion notice its admission fence lapsing

Renewal failures were logged and swallowed, so a run of rejected writes
let the last confirmed `fencedUntil` pass while the deletion carried on
believing admission was still closed — long enough for another replica
to admit a child against conversations about to be removed. Track the
deadline only a confirmed write advances, and check it after the drain,
before anything is deleted: nothing has been removed at that point, so
the operation fails closed and the caller retries once the fence can be
held. A lapse detected after the rows are gone is logged instead, since
reporting failure there would invite a retry against conversations that
no longer exist.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H1cCMDrTWaRNkmtKjpWELZ

* test: raise both concurrency caps the merged-list test trips

Raising the per-scope limit left the store-wide `maxRunningTotal` at its
default hundred, so fifty of the hundred and fifty starts were still
refused. Verified against the base store directly this time: with only
the per-scope cap raised it admits a hundred, and with both raised it
admits all hundred and fifty.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H1cCMDrTWaRNkmtKjpWELZ

* fix(agents): close the fence renewal gap and keep running tasks listed

A renewal that started before its deadline but landed after it was still
credited with extending the fence from its own start time, so a window in
which admission stood open was papered over: a child could take a lease
the drain had already read past and the deletion would proceed without
cancelling it. The deadline now only advances when the write lands while
the previous one still holds; anything later records a lapse the fence
cannot be restored backwards over.

The model-facing cap sorted oldest-first and sliced, which dropped the
newest tasks — including children that had only just started running,
and which the poll tool offers no other way to discover. Bound by status
instead: running children first, then the most recent settled results.
Both caps share one helper, and the routed aggregation now bounds after
its loop so the choice is made across every owner's reply rather than by
whichever answered first.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H1cCMDrTWaRNkmtKjpWELZ

* fix(agents): finish the cap and the fence at the seams they still missed

The status-aware cap only reached the requester: an owner's own reply
still sliced positionally, so a replica holding more than the cap dropped
its running children before the requester could bound anything. Both
sides now share `boundedTaskList`.

A fence that lapsed during the deletion itself was only logged. The rows
are gone by then, so failing is still wrong, but the child another
replica admitted while the fence was down is not: the fence is retaken
and the drain repeated to cancel it.

A child's lease renewal had the same retroactive hole the admission fence
had — Mongo filters on the `now` captured before the call, so a write
landing after the lease expired still moves the row forward, while an
owner drain reading active leases in that gap saw the thread as free. The
lease now carries its own deadline and a late renewal stops the executor.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H1cCMDrTWaRNkmtKjpWELZ

* test: cover the lease lapse and the post-deletion re-drain

The owner-side cap shipped with a regression test; these two did not.
One drives a lease renewal that succeeds only after the lease it was
extending had expired and asserts the executor stops; the other lets the
fence lapse during the deletion itself and asserts a second drain runs
while the request still reports success.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H1cCMDrTWaRNkmtKjpWELZ

* fix(agents): close live-task lifecycle gaps

* test(redis): exercise cluster node discovery

* fix(test): type cluster discovery seam

* fix(ci): wait for orphaned apt processes

* fix(ci): reserve time for apt drain

* fix(ci): skip optional fonts in MCP jobs

* fix(agents): recover tasks after owner loss

* fix(agents): preserve local task discovery

* fix(agents): initialize fail-fast cluster publisher

* style(agents): sort routing test imports

---------

Co-authored-by: Claude <noreply@anthropic.com>
2026-08-19 02:16:35 -04:00
Danny Avila
547bd8c4bf
🧵 feat: Persist View-Only Subagent Threads (#14957) 2026-08-18 07:41:42 -04:00