Commit graph

20 commits

Author SHA1 Message Date
Marco Beretta
bdce771435
🧹 fix: Guard Mongo Cleanup In Schema Lifecycle Tests (#17994)
Some checks are pending
Codegraph E2E Vote Gate / Gate (push) Waiting to run
Codegraph E2E Votes / vote (shard 1/10) (push) Waiting to run
Codegraph E2E Votes / vote (shard 10/10) (push) Waiting to run
Codegraph E2E Votes / vote (shard 2/10) (push) Waiting to run
Codegraph E2E Votes / vote (shard 3/10) (push) Waiting to run
Codegraph E2E Votes / vote (shard 4/10) (push) Waiting to run
Codegraph E2E Votes / vote (shard 5/10) (push) Waiting to run
Codegraph E2E Votes / vote (shard 6/10) (push) Waiting to run
Codegraph E2E Votes / vote (shard 7/10) (push) Waiting to run
Codegraph E2E Votes / vote (shard 8/10) (push) Waiting to run
Codegraph E2E Votes / vote (shard 9/10) (push) Waiting to run
Docker Dev Branch Images Build / publish (push) Waiting to run
Frontend Unit Tests / Client build recovery regression (push) Waiting to run
Frontend Unit Tests / Codegraph select (push) Waiting to run
Frontend Unit Tests / Build packages (push) Waiting to run
Frontend Unit Tests / TypeScript type checks (client) (push) Blocked by required conditions
Frontend Unit Tests / Tests: @librechat/chat (push) Blocked by required conditions
Frontend Unit Tests / Tests: @librechat/client (push) Blocked by required conditions
Frontend Unit Tests / Tests: Ubuntu (shard 1/2) (push) Blocked by required conditions
Frontend Unit Tests / Tests: Ubuntu (shard 2/2) (push) Blocked by required conditions
Frontend Unit Tests / Vite build verification (push) Blocked by required conditions
Frontend Unit Tests / frontend (push) Blocked by required conditions
Sync Locize Translations & Create Translation PR / Sync Translation Keys with Locize (push) Waiting to run
Sync Locize Translations & Create Translation PR / Create Translation PR on Version Published (push) Blocked by required conditions
Playwright E2E Tests / Codegraph select (push) Waiting to run
Playwright E2E Tests / e2e () (push) Blocked by required conditions
Playwright E2E Tests / MCP list_changed (replica count 1) (push) Blocked by required conditions
Playwright E2E Tests / MCP list_changed (replica count 2) (push) Blocked by required conditions
Playwright E2E Tests / MCP Apps (browser and server) (push) Waiting to run
Playwright E2E Tests / e2e (push) Blocked by required conditions
* fix: Guard Mongo Cleanup In Schema Lifecycle Tests
2026-11-13 19:29:25 +01:00
Danny Avila
0abbf3313d
🔐 fix: Apply Base Config Overrides Before Sign-In and Refuse Script Policy Links (#17351)
* 🔐 fix: Apply Base Config Overrides Before Sign-In and Refuse Script Policy Links

Sign-in decisions made before the user is known now read librechat.yaml plus the `__base__`
override of the request's tenant, or of the deployment outside a tenant, and fail closed
(`resolvePreAuthAppConfig`). Registration, the first phase of password reset and the first
email-domain check of the LDAP, OpenID, SAML and social strategies use it. The second check
applies the account's role overrides with or without a tenant, and a new account is checked
against the role it will be created with before anything is inserted. OpenID checks again when
the admin claim or role sync changes the role. Password reset issues no link when the
account's config cannot be read.

The signed-out `GET /api/config` serves Base `registration.socialLogins` and the pre-login
interface fields (`resolvePreLoginAppConfig`), falling back to librechat.yaml when overrides
cannot be read. `turnstile` becomes a base-only section, so Turnstile comes only from
librechat.yaml.

Config reads that merge documents stay in one tenant partition (`tenantPartition`, Mongoose
binding `tenantPartitionFilter`): the active tenant's documents, or the tenantless ones outside
a tenant. The config store's Mongoose binding reads, writes, lists and restores role documents,
`__base__` included, in that partition, and its `listActive` draws every document from it;
group and user documents stay addressed by id. Only a `list` with `acrossTenants` reads every
partition, which `listConfigTombstones` uses so the startup MCP tombstone report still names
each tenant's documents. The tenant-isolation conformance suite gains single-partition cases.

Privacy policy and terms of service `externalUrl` must be http(s), `mailto:`, `tel:`, relative
or blank (`isSafeLinkUrl`). Admin writes of another scheme are refused, a librechat.yaml link is
dropped with a warning at load, a stored one is dropped at merge, and every payload and the
client remove one that reaches them. The admin config meta reports
`features.preAuthBaseOverrides: true`.

* 🧪 test: Layer a Required-Field Override Through interface.currency Instead of Turnstile

Turnstile is set in librechat.yaml only now, so profile overrides of it are skipped and the
layering scenario could no longer see a role's Turnstile options completed by a user's site key.
The scenario now layers `interface.currency`, whose `code` and `rate` are both required: the role
stores only the rate, the user only the code, and /api/config serves both.

* 🛂 fix: Admit a First OpenID Login Under Its Claimed Role Before Creating the Account

The admin-role claim and role sync are now decided before a first-login OpenID account is created, and createUserOnce admits the email under that role's config, read in parallel with the USER role's, before inserting anything. A login the claimed role refuses creates no account and no start balance. Admitted logins are unchanged: the account is still created as USER (now explicitly), moved into the claimed role, re-checked and audited as before, with the claims decided once. An account a concurrent first login created gets the claims decided again for its own role and tenant.
2026-10-22 22:49:54 +02:00
Danny Avila
4e0d9145e0
⚛️ feat: Atomic Admin Config Changes Endpoint (set, unset, tombstone, entry upsert/remove/rename) (#17328)
* 🗄️ refactor: Portable Admin Config Store with Revisions, Key Encoding and Coherent Writes

Admin config writes went straight to MongoDB with dotted-path `$set`, `$unset`, `$pull` and `$addToSet`, and nothing checked `configVersion`. That caused six problems.

* ⚛️ feat: Atomic Admin Config Changes Endpoint (set, unset, tombstone, entry upsert/remove/rename)

Admin clients can change a principal's config overrides only one kind of change at a time:
- PATCH `/fields` for sets, at most 100 entries;
- POST `/fields/tombstone` for one tombstone;
- DELETE `/fields` for one reset.

* 🔁 fix: Reload librechat.yaml on Admin Config Writes

Base-only sections such as `filters` and `messageFilter` live only in librechat.yaml, and an admin
config write was how a running server picked up an edit to them. The coherent-cache change limited
admin-write invalidation to the override and MCP caches, so the YAML base was never re-read after
startup: the content-filter e2e suite, which activates filters by editing the YAML and writing a
reload sentinel, saw `filters: undefined` from GET /base and every activation timed out.

Admin config writes clear the YAML base config again, alongside the override and MCP caches, and
the clear is broadcast with the others, so every instance reloads its YAML. The reload keeps the
last good config on failure and republishes the tools derived from it, so the global tool cache is
left for it to overwrite rather than emptied first.

* 🧭 fix: Cap Dotted Field Path Depth and Hold Revision Preconditions on No-Op Writes

A dotted fieldPath was split without the 32-segment cap that segments already had, so a deep path reached the recursive path writer and failed with a 500. Both spellings now share MAX_CONFIG_PATH_DEPTH and answer 400 CONFIG_PATH_INVALID / too_deep on PATCH, tombstone and DELETE /fields.

Writes reduced to nothing (protected interface permissions, base-only or dedicated sections) answered 200 without checking expectedRevision. They now compare it with the stored document and answer 409 CONFIG_VERSION_CONFLICT with currentRevision on a mismatch, as the write itself would.

* 📡 fix: Retry the Config Invalidation Subscription When Redis Comes Back

A SUBSCRIBE that failed at startup (Redis unreachable, the client's retries spent) was only logged, and ioredis resubscribes only to channels it has joined, so the instance missed every other instance's config invalidation until it restarted. The subscription is now sent again each time the subscriber becomes ready until one succeeds; after that the client's own resubscribe takes over, and closing stops the retries.

* 🧷 fix: Tighten Config Change Batches Around Array Entries, Cancelled Changes and MCP Tombstones

- A tombstone with an index segment below a field configSchema does not define (an unknown key, or what an intersection or any holds) is refused as inside_array: the merge removes inherited values from objects only, so it was stored and reported applied while the element stayed visible.
- unset joins set and rename in the merged-by-key position check: unset endpoints.custom.0.headers answers 400 indexed_merge_key_write, since the position can name another endpoint by the time the batch applies. upsertEntry and removeEntry address entries by key.
- A batch whose changes cancel out (set interface.customWelcome, then unset interface) writes nothing: no empty document, revision bump, audit entry or invalidation. Its changes are reported as unchanged.
- A whole-section mcpServers tombstone reports each librechat.yaml stdio server it leaves in place as ignored (process_mcp_server).
- removeEntry no longer attributes the whole collection to itself, and existing issues of merged-by-key entries are matched by key rather than index, so removing an entry beside a legacy-invalid one (or shifting it) is no longer refused for that entry's issue.

* 🪪 fix: Keep a Requested Priority on New Principals and Report Ignored MCP Renames

POST …/changes discarded a broad manager's priority for a principal without a document when no change altered it: the batch answered written: false instead of creating the priority-only document PUT creates. An explicit priority on an absent principal now counts as a change.

A rename under an MCP server librechat.yaml runs as a process persisted an override the runtime merge drops, but was not reported in ignored. It now is, like set and tombstone. An unset there stays unreported: it asks for the YAML value, which is what runs.

* 🔁 fix: Drop YAML Reloads and Merges Overtaken by a Config Write

Admin config writes clear the YAML base config on every instance, so a write is how a running
server picks up an edit to librechat.yaml. Two races could undo that:

- A base reload that read the old YAML before a write's clear cached it after the clear, and the
  base has no TTL, so the old base was served until the next write.
- A request that read the old base before the clear, and was still resolving principals, cached
  its merge under the new override generation for the override TTL.

The config service now keeps a base-config generation that `clearAppConfigCache` bumps. A reload
publishes the base, the tools and the last good config only if its generation is still current,
a read after a clear never joins a reload started before it, and the newer reload starts once the
stale one settles so the loader's process-wide side effects end with the newest YAML. Merges
capture the generation, base included, before they read the base config.

Tests:
- service.spec: a clear during a reload (stale success, stale failure, a read waiting on the newer
  reload) and a merge built on the pre-clear base.
- invalidation.redis.spec: a write on one instance makes another reload librechat.yaml.
- admin-config-reload.spec (api): every admin config write route, through the real router,
  handlers, database and config service, reloads librechat.yaml; a request with nothing to write
  does not.
- invalidateConfigCaches.spec no longer asserts an order the parallel clears do not have.

* 🧪 test: Reload librechat.yaml After a Committed Change Batch and Not After an Unwritten One

* 🧮 fix: Validate Change Batches on Their End State and Report Base-Only Fields a Tombstone Keeps

A batch's validation scope was built from every intermediate write, located in the state each
change produced. A change a later one undid (set `balance.startBalance`, then restore `balance`)
still widened the scope, so an issue the stored document already had there rejected an
unrelated change, and an entry a later change moved (upsert one entry, then remove one before
it) pointed its written index at a neighbour with a legacy issue. The written paths are now
located in the batch's end state and kept only where that state differs from the stored
document, with entries of merged-by-key arrays compared by key.

A tombstone above a base-only field (`endpoints.agents`, `endpoints.agents.pullRequests`) is
stored and hides the rest of the subtree, while the runtime merge puts back the provider, host,
OAuth app and shared cache slots librechat.yaml sets. The response now lists those fields in
`ignored` with reason `base_only_path`, as it already does for the stdio MCP servers a
whole-section tombstone leaves in place.

* 🧬 fix: Encode Config Keys in Objects Built in Another Realm

The key encoder treated an object as plain only when its prototype was this realm's
Object.prototype, and returned any other object unchanged, keys and all. Under Jest,
structuredClone (which the secret pipeline uses) returns objects from the host realm, so the
items of an array the pipeline had cloned, such as Azure groups or custom endpoints, reached
the engine with their dotted keys raw. Node in production has one realm, but the encoding must
not depend on prototype identity: a skipped object stores its dotted keys raw without an error.

A plain object is now one whose prototype is null or a root Object.prototype from any realm.
The encoding spec covers an object built in a vm context.

* 🧪 test: Check Dotted Record Keys Reach the Store's Whole-Document Write

dev's admin config route test asserted the keys on patchConfigFields, which the handlers no
longer call; it now asserts them on the draft writeConfigDocument receives.

* 🚩 feat: Publish the expectedRevision, prunesEmptyParents and awaitedInvalidation Config Flags

GET /api/admin/config/meta now reports the three write-coherence behaviours this branch
implements, so an admin client can drop the workarounds each one replaces:

- expectedRevision: responses carry `revision`; PUT, PATCH /fields, POST /fields/tombstone and
  PATCH /active take `expectedRevision` in the body and both DELETE routes in the query. A
  mismatch answers 409 CONFIG_VERSION_CONFLICT with currentRevision, a malformed value 400
  INVALID_EXPECTED_REVISION; unpinned writes re-read and re-apply up to 3 times.
- prunesEmptyParents: a field delete or tombstone removes the objects it empties, stopping at
  arrays; shells-only documents count as empty for assign-only lifecycle calls.
- awaitedInvalidation: writes answer after the config caches are cleared (broadcast with Redis),
  GET /base bypasses the merged-config cache, and a merge read before a clear is never cached.

meta.spec probes each one against the behaviour: a stale and a malformed precondition on a
PATCH, the store operations' unset and tombstone on nested objects and an array element, and a
held invalidation, the GET /base read options and a merge raced by a clear.

* 🔀 fix: Keep Config Keys Raw Where the Engine Stores Them, So Older Servers Read Them During Upgrades

Servers before the key encoding (v0.8.9-rc1 among them) read every override key literally.
The Mongoose binding encoded every write, so during a rolling upgrade the first write from an
upgraded server turned a stored `gpt-4.1` or Bedrock inference profile into `gpt-4%2E1` for
every server not yet upgraded, and an older server's restore of a deleted group or role
config dropped the `keyEncoding` marker its schema lacks, after which upgraded servers read
the encoded keys literally.

The binding now sends keys raw first and encodes them only when the engine refuses a raw
field name (DocumentDB, Cosmos, FerretDB or MongoDB 4.4 for `$` names): the refused write
changed nothing, so it runs again encoded under the same revision precondition, and that
process encodes later writes holding such keys directly. MongoDB 4.4+ stores dotted names
(5.0+ also `$` names inside a value), so there an older server reads exactly what it wrote. A
document whose raw keys spell escape sequences while none holds a raw `.`, `$` or NUL is
still written encoded, since a raw read would decode it. A raw write clears the marker.

Reads decode a marked document (raw keys an older server added beside encoded ones, or wrote
under a marker it left, carry no escape sequence and read as stored) and an unmarked one
unless some key holds a raw `.`, `$` or NUL, which no encoded document does; that reads an
encoded document whose marker an older server dropped. Encoding stays in the binding for
every engine that needs it (decision #15); the residual cases are in documentdb-compat.md.

* ⏱️ fix: Remove Config Cache Writes a Clear Overtook in Flight

With an asynchronous APP_CONFIG store (Redis or a custom Keyv store), a clear can bump the
generation and finish its delete after a reload or merge checked the generation but before its
`set` landed. The older value then repopulated `_BASE_` until the next write, or a merged
config for the full override TTL.

The base reload, the last-good restoration and the merge cache now check the generation again
once their write lands and delete what they wrote if a clear overtook it. A deleted entry only
costs the next read a reload or a merge; a newer base load cannot be removed this way, since it
starts only after the stale one settles.

* 🔐 fix: Require the Stored Web Search Key Again When a Change Batch Moves Its URL

#17323 refuses a PUT, PATCH, tombstone or field delete that changes or removes a web search
URL beside a stored API key unless the key is entered again, so a stored key cannot be sent
to a destination it was not entered for. POST /changes skipped that check: a batch could set,
reset or tombstone `webSearch.searxngInstanceUrl` (or replace `webSearch` without the key,
which keeps the stored key) and point the stored key at another host.

A batch is now judged on the document it would store, like PATCH: a URL that ends up beside
a stored key it did not travel with answers 400 CONFIG_SECRET_REENTRY_REQUIRED and writes
nothing, dry runs included. Only keys the request sent count as entered again; a secret a set
kept from the stored document does not.

* 🚩 feat: Publish the changesBatch Flag and the maxChangeOps Limit in Config Meta

GET /api/admin/config/meta now reports `features.changesBatch: true` and
`limits.maxChangeOps` (1000, from MAX_CONFIG_CHANGE_OPS), so the admin panel can gate its
one-request save on the endpoint and size its batches from the server. The meta.spec probe
checks the behaviour itself: an oversized batch answers TOO_MANY_OPS, a batch with one invalid
op writes nothing, and a valid set plus tombstone lands as one write.
2026-10-22 13:38:06 +02:00
Danny Avila
0b4dc238b3
🗄️ refactor: Portable Admin Config Store with Revisions, Key Encoding and Coherent Writes (#17326)
* 🗄️ refactor: Portable Admin Config Store with Revisions, Key Encoding and Coherent Writes

Admin config writes went straight to MongoDB with dotted-path `$set`, `$unset`, `$pull` and `$addToSet`, and nothing checked `configVersion`. That caused six problems.

* 🔁 fix: Reload librechat.yaml on Admin Config Writes

Base-only sections such as `filters` and `messageFilter` live only in librechat.yaml, and an admin
config write was how a running server picked up an edit to them. The coherent-cache change limited
admin-write invalidation to the override and MCP caches, so the YAML base was never re-read after
startup: the content-filter e2e suite, which activates filters by editing the YAML and writing a
reload sentinel, saw `filters: undefined` from GET /base and every activation timed out.

Admin config writes clear the YAML base config again, alongside the override and MCP caches, and
the clear is broadcast with the others, so every instance reloads its YAML. The reload keeps the
last good config on failure and republishes the tools derived from it, so the global tool cache is
left for it to overwrite rather than emptied first.

* 🧭 fix: Cap Dotted Field Path Depth and Hold Revision Preconditions on No-Op Writes

A dotted fieldPath was split without the 32-segment cap that segments already had, so a deep path reached the recursive path writer and failed with a 500. Both spellings now share MAX_CONFIG_PATH_DEPTH and answer 400 CONFIG_PATH_INVALID / too_deep on PATCH, tombstone and DELETE /fields.

Writes reduced to nothing (protected interface permissions, base-only or dedicated sections) answered 200 without checking expectedRevision. They now compare it with the stored document and answer 409 CONFIG_VERSION_CONFLICT with currentRevision on a mismatch, as the write itself would.

* 📡 fix: Retry the Config Invalidation Subscription When Redis Comes Back

A SUBSCRIBE that failed at startup (Redis unreachable, the client's retries spent) was only logged, and ioredis resubscribes only to channels it has joined, so the instance missed every other instance's config invalidation until it restarted. The subscription is now sent again each time the subscriber becomes ready until one succeeds; after that the client's own resubscribe takes over, and closing stops the retries.

* 🔁 fix: Drop YAML Reloads and Merges Overtaken by a Config Write

Admin config writes clear the YAML base config on every instance, so a write is how a running
server picks up an edit to librechat.yaml. Two races could undo that:

- A base reload that read the old YAML before a write's clear cached it after the clear, and the
  base has no TTL, so the old base was served until the next write.
- A request that read the old base before the clear, and was still resolving principals, cached
  its merge under the new override generation for the override TTL.

The config service now keeps a base-config generation that `clearAppConfigCache` bumps. A reload
publishes the base, the tools and the last good config only if its generation is still current,
a read after a clear never joins a reload started before it, and the newer reload starts once the
stale one settles so the loader's process-wide side effects end with the newest YAML. Merges
capture the generation, base included, before they read the base config.

Tests:
- service.spec: a clear during a reload (stale success, stale failure, a read waiting on the newer
  reload) and a merge built on the pre-clear base.
- invalidation.redis.spec: a write on one instance makes another reload librechat.yaml.
- admin-config-reload.spec (api): every admin config write route, through the real router,
  handlers, database and config service, reloads librechat.yaml; a request with nothing to write
  does not.
- invalidateConfigCaches.spec no longer asserts an order the parallel clears do not have.

* 🧬 fix: Encode Config Keys in Objects Built in Another Realm

The key encoder treated an object as plain only when its prototype was this realm's
Object.prototype, and returned any other object unchanged, keys and all. Under Jest,
structuredClone (which the secret pipeline uses) returns objects from the host realm, so the
items of an array the pipeline had cloned, such as Azure groups or custom endpoints, reached
the engine with their dotted keys raw. Node in production has one realm, but the encoding must
not depend on prototype identity: a skipped object stores its dotted keys raw without an error.

A plain object is now one whose prototype is null or a root Object.prototype from any realm.
The encoding spec covers an object built in a vm context.

* 🧪 test: Check Dotted Record Keys Reach the Store's Whole-Document Write

dev's admin config route test asserted the keys on patchConfigFields, which the handlers no
longer call; it now asserts them on the draft writeConfigDocument receives.

* 🚩 feat: Publish the expectedRevision, prunesEmptyParents and awaitedInvalidation Config Flags

GET /api/admin/config/meta now reports the three write-coherence behaviours this branch
implements, so an admin client can drop the workarounds each one replaces:

- expectedRevision: responses carry `revision`; PUT, PATCH /fields, POST /fields/tombstone and
  PATCH /active take `expectedRevision` in the body and both DELETE routes in the query. A
  mismatch answers 409 CONFIG_VERSION_CONFLICT with currentRevision, a malformed value 400
  INVALID_EXPECTED_REVISION; unpinned writes re-read and re-apply up to 3 times.
- prunesEmptyParents: a field delete or tombstone removes the objects it empties, stopping at
  arrays; shells-only documents count as empty for assign-only lifecycle calls.
- awaitedInvalidation: writes answer after the config caches are cleared (broadcast with Redis),
  GET /base bypasses the merged-config cache, and a merge read before a clear is never cached.

meta.spec probes each one against the behaviour: a stale and a malformed precondition on a
PATCH, the store operations' unset and tombstone on nested objects and an array element, and a
held invalidation, the GET /base read options and a merge raced by a clear.

* 🔀 fix: Keep Config Keys Raw Where the Engine Stores Them, So Older Servers Read Them During Upgrades

Servers before the key encoding (v0.8.9-rc1 among them) read every override key literally.
The Mongoose binding encoded every write, so during a rolling upgrade the first write from an
upgraded server turned a stored `gpt-4.1` or Bedrock inference profile into `gpt-4%2E1` for
every server not yet upgraded, and an older server's restore of a deleted group or role
config dropped the `keyEncoding` marker its schema lacks, after which upgraded servers read
the encoded keys literally.

The binding now sends keys raw first and encodes them only when the engine refuses a raw
field name (DocumentDB, Cosmos, FerretDB or MongoDB 4.4 for `$` names): the refused write
changed nothing, so it runs again encoded under the same revision precondition, and that
process encodes later writes holding such keys directly. MongoDB 4.4+ stores dotted names
(5.0+ also `$` names inside a value), so there an older server reads exactly what it wrote. A
document whose raw keys spell escape sequences while none holds a raw `.`, `$` or NUL is
still written encoded, since a raw read would decode it. A raw write clears the marker.

Reads decode a marked document (raw keys an older server added beside encoded ones, or wrote
under a marker it left, carry no escape sequence and read as stored) and an unmarked one
unless some key holds a raw `.`, `$` or NUL, which no encoded document does; that reads an
encoded document whose marker an older server dropped. Encoding stays in the binding for
every engine that needs it (decision #15); the residual cases are in documentdb-compat.md.

* ⏱️ fix: Remove Config Cache Writes a Clear Overtook in Flight

With an asynchronous APP_CONFIG store (Redis or a custom Keyv store), a clear can bump the
generation and finish its delete after a reload or merge checked the generation but before its
`set` landed. The older value then repopulated `_BASE_` until the next write, or a merged
config for the full override TTL.

The base reload, the last-good restoration and the merge cache now check the generation again
once their write lands and delete what they wrote if a clear overtook it. A deleted entry only
costs the next read a reload or a merge; a newer base load cannot be removed this way, since it
starts only after the stale one settles.
2026-10-22 13:04:35 +02:00
Danny Avila
2591a15c10
🔑 feat: Keep Dotted Record Keys in Admin Config Writes and Publish the Config Contract (#17322)
* 🔑 feat: Keep Dotted Record Keys in Admin Config Writes and Publish the Config Contract

Keep dotted record keys in admin config write bodies behind a path-scoped
sanitizer, reject $-prefixed and prototype keys after auth, report dotted_key
issues with segments, and publish GET /api/admin/config/meta (features, rules,
limits, versions) so clients can gate on backend capabilities.

* 🧪 test: Expect Issue Segments in the Config Override e2e Scenario

Override validation issues now carry `segments`; assert the full issue shape.
2026-10-22 09:59:08 +02:00
Marco Beretta
e2b2839375
🗃️ refactor: Remove Dead Data Schemas Methods and Data Provider Exports (#17113) 2026-10-15 12:23:29 +02:00
Marco Beretta
2be35b2f91 💬 refactor: ChatForm UI redesign with unified palette (#14473)
* 💬 refactor: ChatForm UI redesign with unified palette

Replace the scattered composer controls (badge row, tools dropdown, attach
menu, floating steer chips) with one system: a searchable palette with
favourites, a composer bar holding active tools plus effort and speech
controls, a tray staging files and manual context, and a queue rail that
buffers messages while a reply is running.

Collapses the 94 development and review-fix commits of PR #14473 into one
commit so the branch could be rebased onto dev once; the upstream merge of
this pull request is a squash merge either way. The pre-rebase history is
kept locally at tag prerebase-backup (afde3e4b9a).

* fix: finish the dev rebase of the composer redesign

The rebase onto dev left four defects the merge itself produced, plus one
review finding the branch had never answered:

- ChatForm's `min-w-0` JSX comment lost its closing brace, so the file did
  not parse at all.
- `useSteering` carried both dev's local draft consumer and the host-injected
  one under the same name; the host's `consumeDraft` owns the storage key
  autosave actually parked on, so the local copy is gone.
- `useUploadOptions` kept dev's `awaitingAgentProvider` gate after the merge
  replaced the block that derived `isSavedAgent`; the predicate is back, from
  the same `isEphemeralAgentId` the shared allowance helper uses.
- `MessagesView` still reserved a band for the in-flight steer overlay and
  read `steerOverlayHeightFamily`, which nothing writes now that pending
  steers render inside the streaming reply. The reservation is gone and
  `ScrollButton`'s lift is optional; composer panels keep standing the button
  down through `composerOverlayCountFamily`.
- The aggregate MCP chip keyed on `mcp:pinned`, which a server literally
  named `pinned` also produces (`mcp:${serverName}`), so both entries shared
  one React key and one packing slot. It uses a key outside that namespace.

Also drops the locale keys whose only consumers the redesign deleted, ports
dev's extracted-text delivery path into `useAttachExisting` (every
attach-existing surface routes through it now), repoints the mock e2e
built-in helpers at the palette, and repairs test fixtures the merge left
stale (a query mock that blanked agent metadata, a Jotai store mismatch, an
ambiguous tray query, an ephemeral-agent fixture standing in for a saved one).

* test: cover the composer redesign's acceptance scenarios end to end

Twelve mock-harness scenarios, one `@scenario:` tagged test each, over the
behaviours this change is accountable for: the single searchable palette
(open, search, favourite across a reload, keyboard-only operation), the
composer bar and tray (active tool chip, a staged file riding with the
message, a per-turn thinking level that applies to the next turn only), the
queue rail (Enter buffers during a run and drains after it, keyboard reorder
changes the send order, the draft survives run completion), and the composer's
portaled surfaces (Stop reaches the run from inside a portal, a pending tool
approval in the bar resumes the run).

Two contract scenarios are deliberately absent, with reasons: split-pane Stop
isolation is not runtime-reachable because `ChatRoute` mounts one pane, and
code-approval mode mid-run needs an attached stateful code environment the
mock lab does not configure, so dev's `ChatForm.approvalMode` unit spec stays
its coverage.

* fix: make the composer's acceptance run and the whole client suite green

The first verification run of the twelve scenarios passed six and failed six,
and the full client jest suite (which CI runs sharded, unlike the diff-scoped
local precheck) failed fourteen tests. One of each group was a real defect; the
rest were queries and fixtures still aimed at surfaces the redesign replaced.

Defects:

- `useQueueDrain` held its send lock after a revealed queued turn settled, so
  the next server-owned row could never be selected. The reveal path now owns
  its own lock and releases it when the revealed turn clears.
- `ChatForm` recomputed the during-run slot without `consumeComposer` in its
  dependency list, and still carried a landing flag the host prop replaced.

Tests and lab:

- The five `ChatForm.attachments` focus cases now drive the palette dialog and
  the tray's quote chips instead of the deleted attach menu and quote popup,
  with the same strict focus assertions; the harness gains the `main` landmark
  the palette portals into, and the touch fixture a complete `matchMedia`.
- `Palette.spec` no longer closes a hoisted `jest.mock` factory over a later
  `const`; `useSteering.spec` follows dev's toast wording.
- The six failing scenarios were repointed at what this lab actually renders:
  uploads go through the palette's local-computer row, the reasoning control
  needs `interface.parameters`, the queued-row contract is the row and its
  drain rather than a control the submit consumes, the palette's keyboard path
  uses a built-in the lab grants, the slow-reply wait uses the fixture's chunk
  marker, and the approval scenario selects its agent through the builder flow
  the passing approval specs use.

* fix: close the eight findings Codex raised on the rebased head

Codex reviewed the rebased heads and reported eight issues; all eight were
real. Two of them concern code this session added.

Behaviour:

- A custom or OpenRouter endpoint that declares neither `paramDefinitions`
  nor a `reasoningFormat` no longer gets a reasoning control inferred from the
  generic settings list. Capability now has to be declared (a reasoning-key
  parameter definition, a non-disabled `reasoningFormat`, or a known reasoning
  provider as the default params endpoint); a `disabled` format always
  resolves nothing, and every consumer — the composer control, the pre-send
  validation and the server — asks the same resolver, so a gateway backed by a
  non-reasoning model can no longer offer a level the turn would fail on.
- Restoring a queued attachment keeps its `llmDeliveryPath`, so an extracted
  document is routed as its text rather than as raw provider content.
- The placeholder-to-durable reasoning migration now requires this pane's own
  live submission, so opening another conversation from a new chat no longer
  carries the pending level into it. The created/sync case it was written for
  still carries.
- The queue rail treats an acknowledged server-owned row as an ordering
  boundary: local rows still reorder among themselves, a server-owned row
  cannot move, and a move that would cross one is refused out loud — the
  queued-turn API has enqueue, list and cancel only, so a client-side reorder
  across that boundary could not be made authoritative.
- The palette resolves an agent's skill scope with `resolveAgentSkillsScope`
  instead of treating an empty list as an open catalog, and the attach target
  resolves `useResponsesApi` with `resolveUseResponsesApi`, matching what the
  run will actually do.

Policy:

- The reasoning-override behaviour moved out of the CJS middleware into
  `packages/api/src/modelSpecs/reasoningOverride.ts`, which returns a
  discriminated result; the middleware is wiring again, and the behavioural
  cases moved with it.
- `showComposerTips` and `showFilesDialog` are Jotai atoms now, the tips
  preference keeping its storage key and reaching the composer as a prop.

* fix: make the redesigned composer reachable from the mock e2e helpers

CI's Playwright shard failed four `file-provisioning` tests on this branch
because the shared upload helpers still clicked `#attach-file-menu-button` and
`#attach-file-button`, entry points the redesign removed — attaching now goes
through the palette, which also owns the file input. Both helpers drive the
palette now, `uploadViaLegacyOption` mapping its two destination names onto the
palette rows that route to the code environment and the vector store, and the
specs that drove the old surfaces directly (`code-upload`, `thread-renderers`,
`unified-upload`) follow.

Also closes the last client/server disagreement the acceptance run exposed: a
custom endpoint that declares a reasoning key in `paramDefinitions` had that
declaration merged into the provider defaults and then discarded by model-name
inference, so the composer offered a level for `mock-model-a` that the server
rejected with "Invalid reasoning override". An explicitly declared key now
selects the merged setting before inference, while an undeclared generic custom
or OpenRouter target still resolves nothing.

The twelfth acceptance scenario is rescoped: a paused approval keeps the
composer usable and resumes to completion, but the composer-hosted approval
panel and chip are not asserted, because the pending action is keyed by the
stream's conversation id while the composer stays bound to the new-chat
sentinel — in code byte-identical to dev, and mirroring the write onto the
sentinel did not surface the panel. Tracked at berry-13/LibreChat#39.

* test: finish the acceptance run and keep the client typecheck green

- The thinking scenario read `reasoningOverride` from a `userMessage` wrapper
  the chat request does not have; the trace shows the request carrying
  `{ key: 'effort', value: 'max' }` at the root, so the assertion reads it
  there. Source behaviour was already correct.
- The approval scenario timed out on mobile because the Agent Builder drawer
  stayed open after selecting the agent and swallowed the pointer events meant
  for the composer; it now closes the sidebar and waits for the drawer to go
  inert before sending.
- Two specs typed a named custom endpoint's label as an `EModelEndpoint`, which
  CI's client typecheck rejects (the label is exactly what the case is about).

All twelve acceptance scenarios now pass in desktop light, desktop dark and
mobile emulation.

* fix: close the nine findings Codex raised on the verified head

Two were defects in code this session added while closing earlier findings:

- A reasoning key declared in `paramDefinitions` but absent from the generic
  settings list was accepted as capability and then never inserted, so the
  resolver exposed and validated the generic `reasoning_effort` instead. A
  declared key is now appended to the effective settings and resolved before
  any fallback, for `effort`, `thinkingLevel` and `thinkingBudget` alike.
- The queue's reorder guard validated from the destination row's own index, so
  `target === index` rejected every pointer drag and only keyboard reordering
  worked. It validates the dragged row's identity now; the server-owned
  boundary, its announcement and the disabled grips are unchanged.

The rest:

- Stop no longer falls through to the document-wide control once a focused
  pane has been resolved: focus in an idle pane while another generates used
  to abort the other pane's run.
- `PendingSteers` is mounted once in `MessagesView`, above the renderer
  switch, so a sent or failed steer keeps its in-flight, retry and
  send-as-new surface with `FLAT_THREAD` off, where the recursive renderer
  had none and the composer no longer carries a fallback.
- The file-list policy moved out of the CJS route into
  `packages/api/src/files/list.ts`, and its cap is `interface`-level config
  (`fileListLimit`, default 100) instead of a literal in the route.
- The steer-arm confirmation window is `interface.steerArmConfirmationTimeoutMs`
  with its current 10s as the default, travelling the same path as the other
  client-consumed timings.
- The pause predicates judge the active branch's tail rather than the last
  assistant in array order, so an inactive sibling branch can no longer
  enable or disable the inline escalation control.

* fix: move persisted reasoning validation into api package

* fix: restore composer entry points and defaults

* style: normalize reasoning helper formatting

* test: open mobile sidebar before account menu

* test: use settled sidebar layout for mobile account menu

* test: handle empty mobile sidebar in account scenario

* test: target account menu in mobile drawer

* fix: keep composer surfaces and mobile drawer reachable

* fix: let thread approvals receive pointer events

* fix: stabilize composer steering and shortcuts

* fix: commit queue drags and centralize file modal

* fix: return a canceled steer's text to the composer

Canceling a pending steer dropped the words the user had written: the
bubble disappeared and nothing carried the text back, so a reclaim that
the server confirmed still lost the message.

Hand the reclaimed text to `pendingComposerTextByConvoId`, which the
composer already drains on its first render, and append rather than
overwrite when the composer holds a draft the user typed while the cancel
request was in flight.

* fix: close the six findings Codex raised on the pre-rebase head

Pending steers keyed off the Recoil conversation id, which lags the route
during warm-cache navigation, so the surface could sit at the destination
thread's tail while its Cancel and Escalate acted on the source run. It
now keys off the rendered tree, as the mount window already does.

Arming a steer removes the control that was activated, dropping keyboard
focus on the document body. `useSteerEscalate` already exposes `onArmed`
for exactly that; the bubble now passes one that moves focus to the row's
Cancel action.

Opening the file manager from the account menu left its dialog with no
opener: the modal rejects an active element inside a menu, and nothing
supplied a fallback, so closing it returned focus to the body. The opener
is published alongside the open flag and cleared on close, which keeps the
shortcut path's composer-focus behaviour intact.

A steer the server rejected was swept into the queue with no marking, and
the run-end drain then sent those words as a fresh turn without the user
touching Retry or Send as new. Converted failures now carry
`needsExplicitSend`, which the drain skips the same way it skips a row the
rail is mid-edit on.

The queued-turn reconciliation window was a hard-coded 60s, unreachable by
an operator behind a slow proxy or a delayed read replica. It ships as
`interface.queuedTurnReconciliationTimeoutMs` with that value as default,
beside the steer arm timeout it belongs with.

Reasoning-override shape validation moved to
`parseReasoningOverrideRequest` in packages/api, leaving the middleware to
map a rejection onto its own error response.

* fix: keep the composer's reasoning level in step with the parameters panel

`stableConversation` memoized the conversation without any reasoning field
in its key, so editing the level in the Parameters panel produced a new
conversation object that every listed dependency ignored. The composer
went on showing the old level while the next ordinary send already used
the new one.

* test: stop the sidebar helper waiting for a drawer desktop never renders

`ensureSidebarOnScreen` measured `#mobile-drawer` whenever the sidebar had
no Pinned region. Desktop renders no drawer, and `boundingBox()` on a
locator that matches nothing waits for it, so every scenario that reached
that branch at a desktop viewport timed out instead of continuing. Check
for the drawer's absence rather than measuring it.

* test: open the mobile drawer from its inert state, not its geometry

A closed drawer stays mounted and keeps its layout box, so measuring the
Pinned region reported it on screen while every control inside was still
untappable. `inert` is the state the component actually sets when it
closes, so the helper reads that and clicks the header opener before
anything else measures.

* fix: return a canceled steer whole, through one recovery boundary

A steer that stops being a steer was handed back three different ways,
and each one lost something. The immediate cancel wrote `steer.text`
into the composer handoff atom, dropping the attachments, quoted
excerpts, manual skill picks and reasoning override that `settleReclaimed`
then deleted with the pending object. The cancel that won its race
against its own POST cleared the pending marker and restored nothing at
all, so those words disappeared. A definitively rejected steer could
only be resent unchanged, because the row offering to edit it was gone.

Give the recovery one boundary. `useSteerRehome` hands the whole steer
to the composer through the guarded restore the queue rail already uses,
and converts it into an ordinary queued follow-up when the composer
refuses: a draft in the box, another chat on screen, a paused question,
or a composer that unmounted while the reclaim was in flight. The chip
goes either way, since a pending row claiming the server still holds the
words is a lie once they live somewhere else.

The composer is a sibling of the message list rather than its ancestor,
so the thread had no way to reach that restore. The pane's host now
holds the reference and both sides read it there, which also means a
reclaim resolving after the composer is gone refuses instead of writing
into a dead form.

Because the restore refuses rather than overwrites, a draft typed while
a cancel is in flight is no longer glued to the returned steer; the
steer becomes its own queued message and the draft is left as typed.

* test: drive the surfaces these scenarios actually reach

Four scenarios were failing against surfaces the user never touches.

The approval scenario clicked Approve on the thread's copy of the
pending card. The composer's review panel floats over the tail of the
thread and owns the batch submit, so that copy sits behind it and the
click retried until the test timed out. Drive the panel instead; the
thread card stays asserted as the record of the request.

The palette scenario counted rows the moment the dialog painted, then
expected clearing the filter to restore that number. The catalog arrives
in waves: the MCP server list is released by the app-wide warmup timer
and the skills catalog only starts fetching once the dialog's open
effect has run, so the count kept climbing under the test. Wait for the
last wave and for the count to hold still, then snapshot.

The sidebar helper opened the mobile drawer twice through two signals
that disagree. It waited for `inert` to clear, which happens on a state
commit several frames ahead of the slide, then fell through to a
bounding box still reading off-screen and clicked the opener again, into
a machine already opening. One open, through `inert`, then wait for the
drawer to settle in the viewport.

Splitting the cancel scenario follows the recovery change: an empty
composer takes the steer back, and a staged draft keeps the box while
the steer becomes a queued message of its own.

* fix: hold a server-rejected steer for an explicit send

Editing a definitively failed steer falls back to the queue when the
composer will not take it. Queued like any other follow-up, the run-end
drain would send it the moment the reply finished, which turns the
server's refusal into an automatic new turn carrying the payload it
refused.

Mark that fallback the way the rest of the rejected-steer path is
marked, so the row waits for the user to send it.

* test: restore two assertions this branch broke while rewriting specs

Both specs were updated for the redesigned surfaces and both lost a step
they needed, and both fail deterministically in CI as a result.

thread-fold pressed Enter mid-run and waited for a queued row, under a
comment claiming Enter queues by default. It does not: the default
during-run action is steer, so Enter produced a pending steer and the
queued row the next line waits for could never appear. Ctrl/Cmd+Enter is
what takes the other action, which is the one this test needs.

mcp-oauth-readiness asserted on the server card's Cancel button straight
after clicking Connect. Connect opens the OAuth dialog over the panel,
and it is modal, so while it is open the card underneath is marked
aria-hidden and Cancel is not in the accessibility tree at all, which is
why the failure reads "element(s) not found" rather than "not visible".
The rewrite dropped the dismissal the previous flow performed; restore it
as a helper both call sites share.

* test: reach the account menu without the pinned section

The two account-menu tests borrowed `ensureSidebarOnScreen`, which also
settles the Pinned section's placement for the pinned specs. They never
touch that region, and on a run where an earlier test left a pinned
conversation the section exists but has not been laid out yet, so both
tests spent fifteen seconds polling it and failed on the wait rather than
on anything they assert.

Give them a helper that does only what they need: open the mobile drawer
from `inert`, the one signal that distinguishes a closed drawer from an
open one while it stays mounted, then wait for the slide to reach the
viewport. Desktop renders no drawer and needs nothing.

* test: open the drawer by what it shows, not one early read

The helper decided once, from a single read of `inert` taken right after
navigation. On a fresh mobile load the drawer can still be mounting, so
that read reports a state the app has not settled into, the helper
returns believing the drawer is open, and the account menu inside it is
never shown.

Retry against the thing the caller actually needs, the account menu being
visible, re-checking before each attempt. While the drawer is open its
control is labelled "Close sidebar", so the retry cannot toggle it shut.

* fix: keep the composer observing MCP authorization where its tools apply

The old badge row mounted its provider with observeToolAuthorization set
whenever the composer showed ephemeral tools, so on a plain endpoint the
chat kept MCP connection status and discovered tools in the query cache.
The redesigned composer mounts the same provider without the prop, so it
defaults to off and nothing on the chat side loads that state any more.

The Agent Builder's tool library reads that cache. With it empty, every
server reads as Unknown with no tools, and clicking one opens its detail
dialog offering to connect instead of adding it to the agent.

Pass the same gate the badge row used: showTools is the old
showEphemeralBadges expression.

* test: let the palette close before reopening it, and name what is asserted

enableBuiltinTool pressed Escape and returned while the palette was still
animating out. A caller that reopened it straight away resumed the same
popover, scroll offset included, so the Attach rows at the top sat outside
the virtualized window and upload helpers timed out looking for them. Wait
for the dialog to be gone before returning.

The staged-file assertions matched the bare file name, which the palette
now also lists under recent uploads while it closes; scope them to the
tray they are about.

The OAuth dialog dismissal waited for a title built from the server's
display name, but the dialog is titled with the server name.

* feat: make the queued-send lock expiry configurable

The per-pane lock that stops the run-end drain and an explicit Send now
from submitting the same queued message twice gives up a broken claim
after a fixed 60 seconds. Until then every drain and every Send now for
that pane is refused, and a deployment had no way to tune that window.

Expose it as interface.queuedSendLockTimeoutMs beside the queued-turn
reconciliation timeout, defaulting to the same 60 seconds so nothing
changes unless an operator sets it. The interface loader copies each
field explicitly, so the value is added there too; without that line a
configured value would never reach the client.

* fix: decline interrupt-and-steer while reasoning is staged

With a reasoning choice staged, the Interrupt & steer row is disabled,
but its keyboard chord still reached interruptSteer, which fell back to
interruptAndSend: it aborted the run and threw away the partial answer
the action promises to keep.

A staged reasoning choice belongs to a new generation that a steer into
the live one cannot carry, so decline the chord exactly as the row does,
the same way the function already declines while paused on an approval.
The words stay in the composer, where Enter queues them as a turn of
their own.

* fix: wake the queue drain when a late steer retry is queued

Retry marks a failed steer sending, and a sending steer is kept out of
the run-end sweep so its own POST settles it. When the run finishes
while that POST is in flight, the drain spends its one-shot signal on a
queue the steer is not in yet; the late ack or NO_ACTIVE_RUN then moves
it into the queue, where it sat until the user sent it by hand or
another run ended.

Every other late conversion re-posts the last completed run end through
the composer's rewakeDrain. The retry lives in the thread, so the
composer now publishes that function through the same host it already
uses to hand a reclaimed steer back to the draft, and the retry calls it
after converting. It stays a no-op unless the run completed cleanly and
no newer signal is armed, or when no composer is mounted.

* fix: honour thinking-budget ranges wider than 200,000

The request-scoped reasoning override capped thinkingBudget at 200,000
in the shared schema. An operator's paramDefinitions may advertise a
wider range, and the composer renders it, but every value above the cap
was dropped by the control's parse and rejected by the server's before
the range-aware check could run.

That check already exists: resolveReasoningOverride validates the value
against the endpoint's resolved range. Drop the fixed ceiling so the
resolved range is the bound, keeping the integer and lower-bound rules.
The new case covers both directions on a custom endpoint whose range
goes to 500,000.

* refactor: persist the preliminary reasoning override through the shared helper

The agents controller validated the request's reasoning override with
the schema itself before writing the preliminary user message. That is
behavior in the CJS layer, and it duplicated resolvePersistedReasoningOverride,
which @librechat/api already exports and BaseClient already uses for the
same field.

Call the helper instead. A preliminary message is always a fresh turn,
so it is never an edit or a compaction. The two controller specs that
replace @librechat/api with a hand-built mock now take the real helper.

* fix: clear the file manager's state at the session boundary

The file manager's open flag and its opener live in Jotai's default
store, which outlives the authenticated route. A file manager left open
at logout reopened for the next account, still holding a ref to the
previous session's unmounted opener. Reset both in useClearStates, next
to the other composer state this branch already clears there.

* fix: decline interrupt-and-steer on the first turn while reasoning is staged

* test: read the phone layout from the viewport and keep the drawer open

* fix: name both reasons a queued message cannot be edited yet

* test: reuse a published file through the composer palette

* style: move the composer onto the design-system rules

* style: move the file row's new layout styles into classes

* fix: keep the composer's reused thumbnails, focus ring and disabled dimming intact

* fix: bring a steer back when its deferred cancel does not remove it

* fix: treat a run as over only when every pane holding it is idle

* refactor: pass enter-to-send into the composer from its host

* feat: make the palette's recent-file count configurable

* fix: let a file already in the composer be reattached at the limit

* test: hide the chip before replaying a deferred cancel

* refactor: restore the resume request through one packages/api facade

* fix: keep a conversation's staged reasoning across a visit to another model

* fix: keep a connected MCP server's variables configurable from the palette

* fix: resolve every pane-scoped shortcut through the pane holding focus

* refactor: keep every reasoning-override decision out of the /api wiring

* perf: skip paging the skill catalog for an agent that can show no skill

* test: type the reasoning snapshot the override helper records

* test: pin the applied budget and the untouched parameters in the override specs

* style: give the stop button the send button's shared submit recipe

* fix: name only the newline key when no chord sends

* fix: name the stop key only once the stop control can act

* fix: clear every staged reasoning selection at sign-out

* fix: stop advertising send-now while staged reasoning forces a queue

* fix: resolve the reasoning target of an ephemeral agent in the composer and the request

* test: type the staged reasoning the sign-out spec seeds

* fix: check the composer will take a queued row before discarding its parked copy

* style: carry the composer's classes onto Tailwind v4

* style: sort the composer's classes in Tailwind v4 order

* fix: return a refused queued row to the durable queue instead of memory

* i18n: localize the whole attachment-limit message

* refactor: validate the reasoning override inside the packages/api facade

* style: give the composer's submit slot its own IconButton variant

* test: pin the submit slot to IconButton's submit variant

* test: stub the seen-state hook the rebased MessagesView now calls

* fix: turn file search on for an embedded file attached through the shared hook

* chore: record the composer's design-rule counts against the rebased base

* fix: keep the composer's container query inert under Tailwind v4

* test: point the v4 focus scenarios at the redesigned composer's controls

* fix: keep a re-queued row on its own lineage after the run ends

* test: assert a re-queued row carries its lineage

* refactor: pass the dictation preferences into the composer from its host

* test: read the usage atom from the store the ask hook renders under

* fix: read a pending steer's pause from its own pane's branch

* a11y: keep focus on the quote chips until the last one is removed

* fix: keep staged reasoning for the next message across a manual compaction

* test: expect focus on the remaining quote chip after removing one of two

* style: sort the imports the canary rebase merged

* refactor: remove the side-panel files table the account menu replaced

* fix: settle a retried steer's late ack against a racing cancel

A cancel clicked while the retry POST was in flight left the optimistic
marker unread: the ack re-minted a live chip and injected words the user
had asked back, and a failed retry after that cancel lost them entirely,
because the failure path maps chips the optimistic cancel had removed.
The ack path now defers to the marker the way the composer's does, the
failure path re-adds the chip before settling it, and the queued row a
late ack produces binds its recovery identity only under protocol v2,
the same default the composer's degradation path trusts.

* fix: strip a replayed reasoning override that no longer validates on resume

A paused turn's reasoning override is replayed onto its resume request
and then validated as if it were fresh client input, so an operator
changing the endpoint's reasoning config between pause and approve made
the checkpoint permanently unresumable with a 400. The replay now marks
the request, and a replayed override that fails validation is stripped
so the resume proceeds on defaults; a fresh override that fails is still
rejected.

* refactor: derive the enforced spec fields inside the modelSpecs facade

The set of keys an enforced model spec locks is the input contract of
the reasoning-override refusal rule, and deriving it in the route
middleware left the policy untestable from packages/api and free to
diverge from the facade that owns the rest of the preset application.
applyModelSpecPreset now returns the enforced fields beside the private
ones it applies, and the middleware consumes them.

* fix: restore the pre-override parameters when a resume degrades

Stripping a replayed reasoning override that no longer validates removed
only the metadata field, while the resume had already replayed the paused
turn's model parameters with the override applied into the endpoint
option, so the provider still received what validation had refused. The
strip now inverts the application from the replayed base: the recorded
pre-override value returns and the thinking flag the override forced on
comes off.

* fix: resolve the effective responses route for upload targets

The attach target returned the conversation or agent's saved
useResponsesApi flag, so an Azure chat auto-routed to the Responses API
by responsesApiRouting preflighted its uploads against the chat
completions route and lost provider-document uploads. The resolver now
applies resolveEffectiveUseResponsesApi with the endpoint, model,
web-search state and routing policy, the same resolution the during-run
upload target makes.

* fix: name the interrupt chord in the hint only while it still interrupts

The during-run hint advertised Alt+Enter Interrupt & send even when the
chord had been claimed: a submitMessage rebound to Alt+Enter, a chord
yielded to a global shortcut, or shortcuts disabled altogether each make
the keydown resolver return something else. The hint now reads the same
resolver verdict the during-run send button does and omits the chord it
no longer owns.

* test: assert the unseen count the scenario adds rather than a zero baseline

The tab title counts every unseen conversation, and a shared
verification run leaves legitimately-unseen conversations from earlier
scenarios behind, so the three title-count scenarios failed on other
scenarios' leftovers instead of the behavior they seed. Each now reads
its baseline and asserts the count it adds and clears.

* test: hold the unseen count to its direction on the mobile project

The away poll that lights a dot also discovers conversations the client
had not counted yet, so on the mobile project the title count can jump
past any baseline taken mid-test. Opening still has to clear this
conversation's contribution and an away reply still has to raise the
count, which is what the assertions now hold.

* test: enable the MCP Apps server through the redesigned palette

The MCP Apps integration spec still drove the old composer's MCP
Servers menu, which the redesign replaced with toggleable rows in the
attach-and-tools palette; the lane failed on the missing menu on every
run. The helper now opens the palette, waits for the server wave, and
flips the row's pressed state.

* fix: preserve numeric reasoning defaults on blur

* test: reopen persisted code cards before checking highlights

* test: follow the current file manager navigation

* fix: offer only reasoning options a request override can carry

* fix: keep the favorited copy when skills share a name

* fix: restore composer focus by the interaction that closed the palette

* test: expect a keyboard close to refocus the composer on every project

* chore: re-record the design-rule backlog on the rebased composer

* chore: drop the files table and label the rebase brought back

* test: tolerate other projects' unseen conversations when marking one unread

* test: open the file table header scenario from the account menu

* fix: let the stop shortcut reach a hidden stop control from a dialog

* fix: keep staged reasoning out of tool-limit recovery and MCP App turns

* fix: degrade a queued turn's stale reasoning override instead of dead-lettering it

* fix: refresh the attach target when web search changes the route

* fix: re-arm the run-end drain after moving a steer to the queue

* test: hover the MCP row before reaching its revealed Connect action

* test: pick the parts scenario's MCP server from the tools palette

* fix: cancel a recovered row's parked steer only after the composer takes it

A recovered row's only durable copy is the steer parked on the server, and
once its run has ended nothing can re-create it, so a composer that refused
the row after the cancel left the words in memory until the next reload.
The composer now takes the words first; a cancel that then fails holds the
row out of the run-end drain instead of sending it a second time.

* fix: name only the idle composer affordances that act

* fix: keep steer escalation off until the pane owns the rendered conversation

The branch tail that decides whether a run is paused is keyed by the pane's
conversation, which can still name the previous chat while a cached one
renders. Until the pane catches up, a pause on any cached branch now keeps
Escalate disabled instead of reading a missing or foreign tail as unpaused.

* fix: route the host's Enter and auto-send preferences into the composer's nested hooks

ChatForm takes enterToSend and autoSendText from its host, but useTextarea,
DuringRunSendButton and both speech engines still read the store atoms, so a
host whose values differ advertised one behavior while the handlers ran
another. Pass the host values through instead.

* fix: settle a browser dictation take only after its final transcript lands

stopListening resolves on the recognizer's end event, after the final result
reached useSpeechRecognition's reducer but before React committed it, so the
take settled while the composer still held the interim text. Settle from an
effect declared after the transcript effect, off a state update queued behind
the result.

* fix: drop a restored reasoning override the current setting no longer offers

The staged override was only checked for a matching parameter key, so a value
restored after a capability change on the same target stayed staged, blocked
live steering, and was dropped at send time. Validate it against the setting's
options and range.

* test: scan the attach and tools palette for accessibility

The a11y suite still opened the removed tools dropdown. Open the palette
instead, and add a scenario that runs Axe over the open palette.

* fix: make the palette's virtualized list the grid that owns its rows

The rows sat under a wrapper grid behind react-virtualized's scroller, which
always renders aria-label and aria-readonly and so could not be made
presentational. Axe reported both the disallowed attribute and the grid's
invalid child. The scroller is now the labelled grid, with a rowgroup holding
the rows.

* fix: state the palette grid's row total and each row's position

Only the rows in view are mounted, so without aria-rowcount and aria-rowindex
assistive technology reads the rendered window as the whole grid.

* fix: key an empty destination's pending steers to the route

An empty tree has no message naming its conversation, so the pending-steer
surface fell back to the chat context, which still names the source chat
during warm navigation, and its controls acted on the source run. Fall back
to the chat's messages key, which follows the route.

* fix: keep assistant uploads available when the file config fails

Assistants upload directly and never needed the destination policy, which is
why the replaced AttachFileChat rendered their uploader outside the policy
gate. Restore that exception so a failed or paused file config request does
not remove every attachment entry.

(cherry picked from commit f568a4ba30)

Original-PR: https://github.com/LibreChat-AI/LibreChat/pull/14473
2026-10-01 03:46:18 +00:00
Danny Avila
84851d8655
🍃 fix: Serialize Index Builds and the Agent Edge Cleanup for DocumentDB (#15670)
* 🍃 fix: Serialize Index Builds and the Agent Edge Cleanup for DocumentDB

Amazon DocumentDB admits one index build per collection at a time and rejects a second with code 40333. MongoDBSaver.setup() starts the compound and TTL builds of each checkpoint collection concurrently and returns the rejections, which buildMongoSaver logged and ignored, so the TTL index was the likely loser on every boot and checkpoints never expired. setupCheckpointIndexes re-runs the idempotent setup() while any rejection is a 40333. auditLog and the two index migrations bypassed createIndexesWithRetry; buildIndexWithRetry is the raw-collection form and they use it.

removeAgentIdsFromEdges pruned deleted agent ids with an aggregation-pipeline update, which DocumentDB rejects; the guard missed it because the pipeline reached updateMany as a function return value. It is now a read, a prune in code, and a compare-and-set bulk write, keeping the per-graph atomicity the pipeline had.

The static guard now carries the full 5.0 unsupported-operator matrix, rejects $set/$unset at stage position and the $count accumulator, follows calls to functions that return an array, and holds every index build to the retrying helpers.

* 🍃 fix: Keep Companion Setup Errors Past the Deadline and Trace Appended Stages

Codex round 1. setupCheckpointIndexes now carries the non-conflict errors of the last pass and returns them beside the conflict when the peer-build deadline passes, so a permanent failure on one collection is still logged when the other stays busy. The alias-stage detector also inspects the arguments of push, unshift and concat and both branches of a conditional element, since a pipeline assembled by push was invisible to it. Both tests fail without their fix.

* 🍃 fix: Page the Edge Cleanup, Count the Build Deadline From the First Conflict, and Widen the Guard

Codex round 2 and Copilot. buildWhenCollectionFree now starts its deadline at the first conflict rather than the first attempt, so a saver that admits one long build before reporting a companion 40333 still gets its rerun instead of caching with an index missing. removeAgentIdsFromEdges reads matching graphs a page at a time (sort by _id, limit) and uses the filter as its cursor, bounding application memory the way the server-side update did; the loop is bounded by passes without progress rather than a fixed attempt count. The guard follows array-returning class and object methods, resolves stages held in variables, and exempts the Meilisearch createIndex by its primaryKey option instead of by file. retryWithBackoff rejects NaN options instead of skipping every attempt. Every new test fails without its change.

* 🍃 fix: Walk the Edge Cleanup Behind an _id Cursor and Propagate Setup Rejections

Codex round 3. removeAgentIdsFromEdges pages behind an _id cursor so every pass resumes where the last one stopped instead of rescanning the cleaned prefix, with one sweep from the top once the cursor is exhausted to pick up compare-and-set misses; a test lands a concurrent edit on a first-page graph and checks the miss is cleaned. setupCheckpointIndexes rethrows a rejection of setup() itself so buildMongoSaver keeps its in-process fallback, converting only a build conflict that outlasts the deadline. The guard resolves a returned identifier against array-bound variables, so a pipeline built in a local and returned is still a pipeline update.

* 🍃 fix: Bound Cleanup Sweeps and the Audit Index Wait, Reuse the Removed-Id Set

Codex round 4. removeAgentIdsFromEdges gives up after a bounded number of sweeps from the top when references keep being added, since progress otherwise resets the stall bound forever (reachable in deleteUserAgents, which deletes the agents after cleanup); the removed-id set is built once per cleanup rather than per graph. The audit append index build on the request path now has a deadline, so a peer holding the collection makes the append fail open instead of holding the request; createAuditLogMethods takes optional index-build overrides so a test can shorten it. The guard treats a variable annotated with an array type as array-valued whatever its initializer, and states that dataflow is followed within a file only.
2026-09-06 15:02:43 -04:00
Danny Avila
c13990a398
🍃 test: Restore Sweep Coverage of Typed-Criteria Methods (#15609)
Some checks failed
Docker Dev Images Build / publish (push) Waiting to run
Sync Helm Chart Tags / Ignore non-main push (push) Waiting to run
Sync Helm Chart Tags / Sync chart tags (push) Waiting to run
Publish `@librechat/client` to NPM / pack (push) Has been cancelled
Publish `@librechat/client` to NPM / publish-npm (push) Has been cancelled
* 🍃 test: Restore Sweep Coverage of Typed-Criteria Methods

The typed-criteria refactor (#15580) validates query criteria key by key and
fails closed on anything unrecognized — correctly, since a silently accepted
bad criterion would widen a filter into an unscoped `find` or `deleteMany`.
The sweep synthesizes arguments from parameter names, so a parameter named
`query` received a string, which `buildFilter` read as a criteria object and
rejected at index '0'.

Six methods therefore went from adjudicated to un-driven with the suite still
green: `getActions`, `getAssistants`, `deleteAction(s)`, and
`deleteAssistant(s)`. Criteria-shaped parameters are now synthesized as an
empty object, which builds an empty filter and still reaches the engine.

Recovered 13 methods in total — the six regressions plus seven that had been
silently un-driven for the same reason (prompts, groups, conversation tags),
including through the runs recorded as authoritative. Baseline coverage rises
from 372 to 385 of 518 methods. The compatibility assessment now flags this
count as the thing to watch across releases: a coverage regression hides
behind a passing suite, and only a matrix diff surfaces it.

* 🍃 test: Preserve searchMessages' String Query in the Sweep

Criteria-shaped synthesis is name-based, so it also caught
`searchMessages(query: string, ...)`, whose parameter is a genuine search
string. On a Meili-enabled deployment that object is forwarded verbatim into
`meiliSearch`, so the recorded outcome would reflect a harness-generated
request rather than the method's behavior.

`searchMessages` gets an `ARG_OVERRIDES` entry supplying a real query string;
its failure is now the honest 'MeiliSearch plugin not registered' rather than
an invalid request. It is the only method in `src/methods` whose
`query`/`criteria`/`filter` parameter is string-typed, which the synthesis
comment records along with the symptom a future one would show.

An error-driven variant (repair only on 'Unknown query criterion') was tried
first and rejected: it protects string parameters automatically but loses
seven methods whose criteria parameter makes them early-return on a string
rather than throw a recognizable error, taking coverage from 385 back to 378.
Coverage holds at 385 of 518 driven, zero engine rejections.
2026-09-04 12:58:41 -04:00
Danny Avila
81338cef95
🍃 test: Sweep Every DB Schema Method Against a Live Engine (#15392)
* 🍃 test: Sweep Every data-schemas Method Against a Live Engine

Adds `misc/documentdb/sweep.documentdb.spec.ts`: drives all 510 exported
data-schemas methods against a real engine and records, per method, how many
driver queries it issued and whether the engine rejected any of them.

Every DocumentDB incompatibility found so far — pipeline updates, `$$REMOVE`,
`$facet`, the transaction-probe false negative, the Mongoose-compiled mixed
projection — was invisible until someone thought to look for its class. The
sweep inverts that: the engine adjudicates whatever each method emits, known
class or not. Query shapes are parsed before they are matched, so an empty
database adjudicates most entry paths.

Mechanics: arguments are synthesized from each function's parameter names,
then repaired adaptively from the validation errors they provoke (ObjectId and
number casts, required schema paths, object-parameter shapes) until the method
reaches the database; the driver prototype is instrumented to count calls and
capture server errors — including on cursor consumption, where parse errors
actually surface — and duplicate-key violations are classified as benign. A
method that never issues a query is reported un-adjudicated instead of
silently green: coverage is visible, not assumed.

Two modes with identical mechanics so matrices diff cleanly: in-memory MongoDB
(`SWEEP_BASELINE=true`, free, CI-able) and live DocumentDB (`DOCUMENTDB_URI`).
`SWEEP_REPORT_PATH` writes JSON for machine diffing;
`DOCUMENTDB_STRICT=true` turns rejections into failures.

First full run against live DocumentDB 5.0.0: 363 of 510 methods issued at
least one query, zero engine rejections, zero divergences from the MongoDB
baseline. The jest config gains the main config's transform-ignore patterns so
the full method bundle (which pulls ESM markdown tooling) loads.

* 🍃 fix: Adjudicate Index DDL, Real ACL Cascades, and a Run-Scoped Sweep Database

Review round 1, all four findings real:

- The driver instrumentation now wraps `createIndex`, `createIndexes`,
  `dropIndex`, `dropIndexes`, and `listIndexes` (cursor-wrapped), so an engine
  rejecting index DDL — which `recordAuditEntry` issues before any listed
  operation and then swallows — is adjudicated instead of surfacing as a
  false `no-queries` under STRICT.
- The `removeAllPermissions` no-op override is gone: `createMethods` supplies
  a real implementation delegating to `deleteAclEntries`, so driven deletion
  cascades exercise their ACL operations (not-driven drops 147 → 142).
- Each race timer is cleared when its attempt settles, instead of hundreds of
  referenced timers living until the shared deadline and the last one holding
  the process open after the report.
- The live run connects to a run-scoped database
  (`librechat_sweep_<runId>`), so its `dropDatabase` teardown can never erase
  fixtures out from under the sibling live suites sharing the URI, nor any
  retained state in the URI's own database.

* 🍃 fix: Attribute Straggler Queries by Async Context and Adjudicate Transaction Commits

Review round 2, all three findings real:

- Per-method attribution now rides AsyncLocalStorage instead of a module
  global: a method that outlives its timeout keeps recording queries and
  engine errors under ITS OWN row rather than corrupting whichever row the
  sweep advanced to, and a final reclassification pass upgrades any row whose
  engine error arrived after its outcome was assigned.
- The session prototype's `commitTransaction`/`abortTransaction`/
  `withTransaction` are instrumented: transaction-lifecycle rejections surface
  from the session, not a collection operation, and several methods convert
  them into domain errors — previously a real commit rejection classified as
  a mere `threw` and STRICT passed.
- The compatibility doc's published numbers now come from the
  corrected-harness baseline (368 driven / 142 un-driven) and state explicitly
  that the recorded live-run matrix predates the instrumentation widening, so
  the next live run is authoritative for the newly adjudicated surfaces.

* 🍃 fix: Settle Stragglers Before STRICT and Commit a Real Transaction on a Replica-Set Baseline

Review round 3, all four findings real:

- STRICT is now enforced against the FINALIZED rows: `afterAll` waits
  (bounded) for every timed-out invocation to settle, reclassifies rows whose
  engine errors arrived late, and asserts zero rejections — previously a
  straggler's rejection could arrive after its per-test assertion had already
  passed against the `timeout` outcome.
- The baseline runs on a single-node replica set with the live branch's exact
  connection options (`autoIndex`/`autoCreate` off, run-scoped `dbName`):
  a standalone baseline would reject driven transactions for topology reasons,
  and asymmetric model auto-creation would attribute DDL in one mode and not
  the other — both manufacturing false matrix divergences.
- `resolveMCPAuthorityProof` gets a seeded case (fixture ported from the
  compat suite, executed in tenant context via the new `within` wrapper on
  overrides): generic synthesis died on target validation before a session
  ever started, so the session instrumentation adjudicated nothing. It now
  drives 22 operations through the production path, transaction commit
  included, on both engines.

* 🍃 docs: Sync the Published Baseline Counts With the Seeded Harness

* 🍃 fix: Materialize Transaction-Read Collections and Settle Stragglers Before Teardown

Review round 4, three findings real (the fourth — stale doc totals — was
already fixed on the pushed head):

- Straggler settlement moved BEFORE engine teardown: dropping the database or
  closing the client would abort outstanding work with closed-client errors
  that are not engine verdicts, leaving a timed-out row unadjudicated while
  STRICT passes.
- The authority fixtures (sweep AND compat) materialize PluginAuth and Token
  before the transaction: `loadCurrentProof` reads both inside it, and
  DocumentDB rejects any in-transaction statement touching a non-existent
  collection — the rejection surfaces as `proof_unavailable` before the commit
  is reached. This is the mechanism behind the compat authority test failing
  on every live cluster run while passing against MongoDB; with the
  collections materialized, the next live run adjudicates the transaction
  itself instead of the missing namespace.
- Engine errors are recognized by `instanceof MongoServerError` instead of a
  `name` equality, so server-originated subclasses (MongoWriteConcernError,
  MongoBulkWriteError) can no longer slip through the session wrapper as
  domain throws.

* 🍃 fix: Materialize Group and Config for the Authority Transaction and Exclude Constructor Exports

Review round 5, both findings real. The proof transaction also reads Group and
Config; the sweep fixture materializes both (the compat fixture already
creates a Config record and now materializes Group). SessionError — an error
class exported alongside the session methods — is excluded from the method
map, correcting the totals to 509 methods / 369 driven / 140 un-driven; the
published baseline is synced in the same commit.

* 🍃 fix: Build a Fresh Method Bundle per Case and Emit a Normalized Matrix

Review round 6, both findings real:

- Every row now invokes a freshly constructed method bundle. Factory-local
  caches leaked across alphabetically ordered rows — the session methods cache
  `sessionIndexesPromise`, so `ensureSessionIndexes` returned that settled
  promise and was misreported as `no-queries`. It now issues its 3 index
  operations, making the matrix order-independent (370 driven / 139 un-driven).

- The report separates run metadata from a canonical `rows` payload with
  run-specific values normalized out (run id, generated ObjectIds and UUIDs),
  so the two engines' matrices diff cleanly instead of differing on every row
  that embeds a synthesized value in its error text.

* 🍃 fix: Materialize Authority Snapshot Namespaces Before Its Transaction

Amazon DocumentDB rejects any statement inside a transaction that touches a
collection which does not exist, and `asMCPError` converts that server
rejection into `proof_unavailable` — so on a deployment where no PluginAuth
or Token row has ever been written, every MCP authority proof fails with an
error naming nothing about the real cause. This is the mechanism behind the
live-harness authority probe failing on every real cluster run since July
while passing against MongoDB.

`loadAuthoritativeSnapshot` now materializes the nine namespaces its snapshot
reads before opening the transaction, once per process. MongoDB tolerates the
in-transaction read either way, so nothing changes there; a create failure is
logged at debug and left to surface from the transaction itself.

The regression test drops the credential namespaces to reproduce the state and
asserts the resolve recreates them and succeeds; it fails without the fix.

* 🍃 docs: Record the Authoritative Hardened Live Run

The hardened harness ran against a live DocumentDB 5.0.0 cluster: 370 of 509
methods drove at least one query, zero engine rejections, and the matrix is
byte-identical to the MongoDB baseline — every outcome and query count
matches. This run adjudicated index DDL, transaction commits and aborts, ACL
cascades, and the authority snapshot transaction, none of which the first
pre-hardening run could see.

The same run turned the compatibility suite's MCP authority probe green for
the first time on any real cluster, confirming the namespace-materialization
diagnosis for the `proof_unavailable` failures seen since July.

* 🍃 fix: Retry Snapshot Namespace Creation After a Transient Failure

The catch suppressed every `createCollection` error, so a transient failure
settled the once-per-process memo as successful: creation was skipped for the
rest of the process, the transaction kept failing on the missing collection,
and every later proof returned `proof_unavailable` even after the condition
cleared. The existing reset-on-reject path could never fire because the inner
promise never rejected.

Only `NamespaceExists` (including a concurrent creator winning the race) is
suppressed now; every other failure propagates, clears the memo, and lets the
next request retry. The regression test rejects the first creation with a
transient error, asserts the proof fails, then asserts the following request
recreates the namespace and succeeds — it fails against the swallow-all form.

* 🍃 fix: Throttle Snapshot Namespace Preflight Retries With a Bounded Cooldown

Clearing the memo on every rejection made a persistent failure — an
authorization error, say — turn each authority request into another serial
collection preflight: a DDL retry storm precisely while the database is
unhealthy, since the single-flight promise only coalesces requests while an
attempt is pending.

A failure is now recorded with its timestamp and replayed without touching the
database until a bounded cooldown elapses (30s, overridable through the
existing hooks for tests). Transient failures stay retryable, just not on
every request. Both sides of the boundary are covered: one test asserts a
persistent failure issues no further creations inside the cooldown, and the
existing retry test runs with the cooldown disabled to assert recovery still
works. The throttle test fails with the cooldown gate removed.

* 🍃 fix: Instrument Db-Level Namespace DDL in the Sweep

The authority path's namespace preflight calls `Model.createCollection()`,
which delegates to `Db.createCollection` — not to any `Collection` operation
the sweep wrapped. A DocumentDB rejection of that DDL would therefore be
laundered by `asMCPError` into `proof_unavailable`, leaving the row
classified as a mere `threw` while `DOCUMENTDB_STRICT=true` passed: the exact
false-green class this suite exists to close, reintroduced by the preflight
itself.

`Db.createCollection`/`dropCollection`/`renameCollection` are now
instrumented, and `NamespaceExists` (48) joins the benign server codes since
an idempotent preflight against an existing collection is the expected
outcome, not an incompatibility. The authority row's adjudicated query count
rises from 22 to 31, which is those nine preflight operations becoming
visible.
2026-08-31 09:54:32 -04:00
Danny Avila
14d4f2789d
🍃 fix: Send a Pure Inclusion Projection From the Legacy Receipt Sweep (#15390)
Some checks are pending
Docker Dev Images Build / build (Dockerfile, librechat-dev, node) (push) Waiting to run
Docker Dev Images Build / build (Dockerfile.multi, librechat-dev-api, api-build) (push) Waiting to run
Sync Helm Chart Tags / Ignore non-main push (push) Waiting to run
Sync Helm Chart Tags / Sync chart tags (push) Waiting to run
* 🍃 fix: Send a Pure Inclusion Projection From the Legacy Receipt Sweep

The legacy actor-receipt sweep selected its candidates with
`.select('_id +agentEventActorReconciliations')`. `_id` alone does not make a
Mongoose projection inclusive and a `+` token only un-hides its field, so this
compiles to `{ _id: 1 }` plus a `: 0` exclusion for every other
`select: false` sibling — a mixed projection MongoDB tolerates via the `_id`
exception but Amazon DocumentDB rejects (`Projections cannot have a mix of
inclusion and exclusion`). The sweep failed on every 30-second maintenance
pass, receipts never expired, and the rejection took the whole Promise.all
down with it, so the sequenced lane reclamation never ran either.

- The candidate read uses the object form `{ _id: 1,
  agentEventActorReconciliations: 1 }` — a pure inclusion that still returns
  the hidden field and fetches nothing extra. A regression test spies on the
  driver call and asserts the compiled projection contains no exclusions.
- Every maintenance step now fails alone: a rejection is logged with its step
  name and counted as zero progress instead of aborting the pass, and lane
  reclamation runs regardless. Covered by a test rejecting one step and
  asserting reclamation still runs.
- The static guard gains a detector for the exact broken shape —
  `select('_id +field')` — with fixtures for both directions. The shape is
  narrow on purpose: probing real Mongoose shows a bare non-`_id` token makes
  the projection inclusive and turns `+` tokens into plain `field: 1`
  inclusions, so the repo's nine other `+`-selects are all legal and stay
  unflagged. The live audit suite gains a raw mixed-projection probe and
  drives the sweep itself as production shape number eight.

* 🍃 fix: Gate Lane Reclamation on Batch Recovery and Type the Projection Assertion

Review round 1, both findings real:

- Batch-receipt recovery is a prerequisite of lane reclamation, not an
  independent step: reclamation consumes the lane-cleanup markers
  (`fulfillLaneCleanupRequest` unsets the delivery-side marker, and
  `reclaimLaneIfInactive` clears `cleanupRequestedAt` when the lane is still
  retained), so running it against a half-recovered batch erases a request no
  later successful recovery can re-arm, retaining the lane permanently. The
  blanket isolation reintroduced exactly that; reclamation now waits for a
  batch-recovery pass that did not fail, while the genuinely independent steps
  keep failing alone. Covered by a test rejecting batch recovery and asserting
  reclamation is withheld.

- The captured-projection assertion drops its `Record<string, unknown>` cast
  and asserts deep equality with the exact expected pure-inclusion projection,
  which is both typed and strictly stronger.
2026-08-30 22:10:44 -04:00
Danny Avila
9684b2478f
🍃 fix: Restore Amazon DocumentDB Compatibility Across Trigger and Subagent Writes (#15375)
* 🍃 fix: Restore Amazon DocumentDB Compatibility Across Trigger and Subagent Writes

The README advertises Amazon DocumentDB 5.0+ support, but six sites added
between 2026-07-29 and 2026-08-30 reintroduced constructs the engine rejects,
and the transaction probe silently reported a false negative. All seven are
adjudicated against a live DocumentDB 5.0.0 cluster, not against AWS's
omission-based documentation.

Rejected by every engine version, now rewritten with supported operators:

- `claimNextAgentTriggerDelivery` used a pipeline-form update. It reads the
  globally oldest candidate under the same sort, then claims that exact `_id`
  and lifecycle state with a compare-and-swap. Ownership is still granted only
  by the fenced write. An empty queue costs the one read it costs today; a
  successful claim costs one additional round trip.
- `renewAgentTriggerDeliveryProducerLease` used a `$cond` to keep the later
  deadline. `$max` is the exact classic-operator form and evaluates no
  expression.
- `claimBackgroundToolResults` and `releaseBackgroundToolResultClaims` rewrote
  the entire `content` array through `$map`. The filtered positional operator
  addresses the matching parts directly, so each write touches only those parts
  and still needs no read-modify-write.
- The bounded subagent thread-view projections used `$$REMOVE` to omit keys.
  They now emit `null` and are pruned back to absent inside the pass that
  already materializes each record, preserving the absent-vs-present contract.
- The selected-task read used `$facet`. It is now two concurrent reads, which
  also removes the 16 MiB single-document risk the surrounding code already
  avoids for the same reason.

`supportsTransactions` probed a canary collection that does not exist, and
DocumentDB rejects a transaction touching a non-existent collection. The probe
returned `false` on an engine that fully supports transactions, so every caller
silently took the non-transactional path. It now materializes the canary first,
once per process.

Adds `documentdb.spec.ts`, a static guard over the whole class, because no
runtime test can catch it: every suite runs `mongodb-memory-server`, which is
real MongoDB and accepts all of these constructs. Adds
`misc/documentdb/audit.documentdb.spec.ts` to re-adjudicate against a real
cluster, and records the live verdicts and connection requirements in the
compatibility assessment.

* 🍃 fix: Address Review — CAS Exhaustion, AST Guard, Single-Snapshot Selected Read

Addresses all three review findings, and the strengthened guard immediately
found a seventh incompatible site the original character-level scan missed.

- Exhausting the claim's compare-and-swap budget now throws instead of
  returning null. Every lost round read a claimable row, so the queue is not
  empty; null is the engine's confirmed-empty signal and advances its idle
  backoff, which would leave queued work waiting. A throw lands on the engine's
  retryable claim-failure path, which holds the polling cadence. Covered by a
  new test asserting the bounded budget and that the loser owns nothing.

- The static guard now walks the TypeScript AST: it catches pipelines bound to
  a variable first, cast with `as`, chosen by a conditional, or carried in a
  `bulkWrite` operation's `update` property (including shorthand). Every shape
  is proven against fixtures in the suite itself, so the detectors can never
  silently go inert. The stronger scan flagged `updateToolCallResult`, whose
  pipeline lived in a `stages` variable: the backgrounded tool-call settle path
  was still broken on DocumentDB. Rewritten with the filtered positional
  operator — subfield sets preserve an existing `resultClaim` without per-part
  branching, an explicit `$unset` keeps the old whole-object replacement's
  wakeup-disarm semantics, and the attachments merge becomes `$pull` plus
  `$push` (two atomic writes that converge under the retry the claim fencing
  already provides). New tests lock the disarm and supplied-claim semantics,
  and the live audit suite now drives this method too.

- The selected-task read is a single aggregation again: each selected row
  yields both its bounded message fields and its source payload from the same
  document version, restoring the `$facet` snapshot guarantee the concurrent
  split lost, and returning the endpoint to three reads.

* 🍃 fix: Close Settle Atomicity Windows and Spread Claim Contention

Self-review pass (eight independent review angles over the branch diff).

Correctness:

- The attachments merge is a single fenced write again: read the array, merge
  with the exact semantics the old `$filter`/`$concatArrays` had, and write it
  back with the settle patch in one update guarded on the array being unchanged
  — the same compare-and-swap this file already uses for control receipts. The
  `$pull`-then-`$push` split could strip attachments on a failure between the
  writes (the throw path retires the delivery, so nothing healed it) and
  duplicated them under concurrent re-application.
- A supplied `resultClaim` is stamped BEFORE the settle write instead of after
  it. The claim-last order left a terminal, claimable, unclaimed part visible
  to the ~1s wakeup poll, which could acquire and deliver a result a manual
  poll had already delivered; claim-first leaves only inert intermediate states
  (a claim on a part that is not yet terminal admits no consumer). The stamp's
  guard is `resultClaim: null`, restoring the old `$ifNull` treatment of a
  stored null.
- `releaseBackgroundToolResultClaims` filters on `content: { $type: 'array' }`:
  the filtered positional operator errors on a row without the array where the
  old pipeline no-op'd, and that throw surfaced as retryable release failures
  that left claims stamped forever.
- The claim loop reads a BATCH of candidates per round, so the engine's
  parallel claimers spread across the batch instead of convoying on one
  head-of-queue row; exhaustion now throws the typed
  `AgentTriggerClaimContentionError`. The dead-capability adoption fence also
  pins `availableAt`, which the write copies.
- The transaction probe probes first and creates the canary only after a
  failure — plain MongoDB never pays the DDL or gains the collection — logs a
  failed create instead of swallowing it, and concurrent first probes share one
  in-flight promise.

Structure and guard:

- One `partScope(prefix)` builder yields the `$elemMatch` document filter and
  the arrayFilters element filter from a single predicate, deleting the
  aggregation `$expr` twin, and the agent-ownership rule shared by settling and
  claiming lives in one `agentOwnershipFilter` helper.
- The static guard scans all three backend workspaces (`packages/data-schemas`,
  `packages/api`, `api`), parses each file once, and shares the tree between
  both detectors.
- The audit suite asserts each probe's returned verdict, the compat suite's run
  recipe carries the connection parameters the audit header documents, and the
  `$facet` pipeline assertion covers every captured aggregation in one loop.

* 🍃 fix: Keep Attachment Contention Retryable and the Selected Snapshot Authoritative

Second codex round, both findings real:

- Exhausting the attachment-merge fence now returns an unmatched result with a
  warn log instead of throwing. The settle retry loop treats unmatched as
  retry-then-heal-later and degrades to false; a throw escaped it onto
  ambiguous-failure handling that can retire the completion, losing a healthy
  completed tool result to mere contention. Locked by a test asserting the
  bounded rounds resolve to `{ matched: false }` rather than a rejection.

- When the selected task's rows also appear in the normal page window, the
  dedup pass kept the page copy — read by an independent concurrent aggregate —
  and discarded the single-snapshot version, letting a page row claim a
  transcript that `selectedSources` does not carry. Duplicates are now replaced
  in place with the snapshot version (page order kept), so the selected
  message/source pair stays per-document coherent end to end.

* 🍃 fix: Treat a Stored-Null Claim Stamp as Unclaimed in the Claim Fence

Third codex round. The subfield-preserving settle write keeps a persisted
`resultClaim: null` that the old whole-object rewrite dropped, and the claim
path's outer `$elemMatch` fence still required the field to be absent via
`$exists: false`. The in-memory claimable scan, the claim arrayFilters, and the
settle stamp all already treat a null stamp as unclaimed, so a part carrying
one became terminal but permanently unclaimable: every manual poll and wakeup
read `ready` in memory, missed the fence, and returned `not_ready` forever.

The fence now matches missing-or-null like every other layer. The regression
test persists an actual null stamp — asserted present before claiming, so the
test cannot pass vacuously — and claims it.
2026-08-30 18:33:25 -04:00
Danny Avila
c5276fc63d
⏱️ feat: Run Scheduled Chats Through Durable Agent Triggers (#14939)
* feat: Scheduled Chats — agent-centric scheduled runs creating real conversations

Squash of the full review-hardened branch (PR 14540, supersedes 14373) onto
latest dev, preserving the exact verified tree. History prior to this commit
lived on the pre-squash branch; every invariant below survived 25 Codex review
rounds plus two external audit rounds (R26) with regression tests that fail
without their fixes.

Feature:
- Schedules CRUD + side-panel UI (cadence dialog, run cards, Run Now), roles/
  permissions (SCHEDULES:USE), interface.schedules availability, per-user limits
  and capacity slots, timezone-aware cadence with DST-conservative floors and
  misfire grace.
- Engine: single-process claim/fire loop with leases, loopback POST dispatch
  (signed schedule-fire JWT claims, per-occurrence idempotency key inside the
  route's clientRequestId charset), overlap/balance/capacity/duplicate skip
  policies with auto-disable streaks (too_many_failures, insufficient_balance),
  reconciliation from retained terminal-job evidence, erasure sweep.
- Scheduled runs create real conversations through the resumable agents chat
  path: HITL pauses surface on the card (requires_action), resumes re-apply the
  fire boundary's admission policy (revision fence, enabled, global kill switch,
  SCHEDULES:USE, availability) before continuing a billed generation.

Correctness invariants (the audit surface):
- Settlement discipline: every persistence-producing write happens-before a
  run's terminal outcome write; Stop/complete/pause race through single-winner
  terminal CAS claims (dev's TerminalJobClaim substrate) with retained,
  completedAt-less evidence for scheduled fires plus an owner-intended outcome
  stamp (scheduleOutcome) the reconciler prefers over re-derived success —
  round-tripped through the Redis hash mapper.
- Swallowed generation failures (client error content parts) classify to
  error/skipped_balance instead of success on both initial and resumed paths;
  stale stamps are refreshed evidence-first when persistence plus the Mongo
  outcome write both fail.
- Abort honesty: delivery judged by generation ownership and the CAS's actual
  from-status; republication escalates to the transport's acknowledged variant;
  the Stop route settles only genuinely paused runs.
- Account deletion: one-way barrier (deletionRequestedAt) with auth-cache
  tombstone-before-stamp, boundary rechecks across all auth strategies, quiesce
  of scheduled + interactive work with durable per-stream abort fences
  (positive-evidence acknowledgement only), owner-side finalization markers for
  post-terminal billed writes, deferred-deletion sweep (explicitly ensured
  partial index) that completes cascades autonomously. Remote OpenAI-compatible/
  Responses requests are documented as outside the quiesce and tracked in issue
  14594.
- Store compatibility: finalization markers optional on the legacy IJobStore
  contract with coherent degradation (registration fails -> synchronous title
  fallback; count reads 0).

* fix: fence terminal response persistence from deletion; retain stale-pause evidence

Two blockers from the third external review round.

Terminal persistence visible to account deletion:

- The finalization marker was registered only when post-terminal TITLE work was
  possible, but every persistence-owning terminal CAS opens the same window: the
  claim drops the job out of the active set BEFORE the response save (and the
  background user-message/convo saves), so a deletion quiesce landing there saw
  neither an active job nor a marker and could cascade while the admitted request
  could still recreate messages. Both controllers now register the marker before
  every persistence-owning claim — the fresh-turn path and the HITL resume — and
  release it only after their pending saves (and any post-terminal title) have
  landed, on success and failure paths alike. The TTL bounds a crash.
- settleAbortFence no longer clears a complete/error fence while
  `terminalPersistencePending` is set: terminal at the CAS is not settled while
  the owner is still persisting. The job facade now surfaces the flag.
- The marker trio is REQUIRED by the runtime store contract (assertJobStoreV2
  refuses a store without it at configure time, keeping the failure loud and
  deterministic) while remaining optional on the legacy public IJobStore type for
  source compatibility. The silent degrade path from the previous round is gone —
  it was not deletion-safe.

Stale-pause recovery retains scheduled evidence:

- Three crash/timeout recovery paths — ApprovalLifecycle.failStalePausePersistence
  and the InMemory/Redis stale-pause cleanups — unconditionally stamped
  `completedAt`, putting a scheduled fire's failed-pause error terminal on the
  short completed TTL. A Mongo outage longer than that TTL erased the evidence and
  the reconciler recovered the run as `interrupted` instead of `error`. All three
  now follow the controller-observed path from the previous round: scheduled jobs
  omit `completedAt` (retained-evidence TTL) and stamp the error outcome.

The PR description now explicitly narrows the deletion guarantee for the remote
OpenAI-compatible/Responses paths (tracked in issue 14594).

* fix: deletion-fence marker protocol — generation-scoped, atomic, fail-closed, all terminal paths

One consolidated pass over the finalization-marker mechanism, per the fourth
external review round. The invariant it establishes: NO persistence-owning
terminal CAS runs without a durable, generation-qualified marker covering the
window it opens, and every consumer treats a pending terminal as unsettled.

- Generation-scoped markers. Entries were keyed (userId, streamId), so a
  Stop-superseded generation finishing late could clear the marker its
  replacement registered on the same conversation. Marker fields are now
  qualified by the generation's createdAt; clears must present the same
  identity, and an unqualified legacy clear cannot drop a qualified entry.
- Atomic Redis registration. HSET-then-EXPIRE loses the fresh marker when the
  user's existing hash expires between the two commands (or the process dies
  there); registration is now a single Lua script carrying both.
- Fail closed everywhere. Registration failure (after one retry) now REFUSES
  the terminal CAS instead of proceeding uncovered: the completion claim throws
  into the error path, the error path skips completeJob and leaves the job
  ACTIVE — deletion-visible by itself, recovered by the stale-running reaper —
  and abortJob returns a new retryable `fence_unavailable` failure the Stop
  route answers with 503 and the deletion quiesce treats as an unacknowledged
  stop (fence kept). The previous round's log-and-proceed is gone.
- Every terminal path enrolled. abortJob now owns its window (register before
  the abort CAS, clear in its finally — the Stop route's checkpoint prune and
  partial save run inside beforePublish, between CAS and publication); the
  interactive and background generation-error paths register before their
  completeJob; the resume controller's error finalization registers before its
  completeJob. A lost or thrown claim releases the marker after pending saves
  flush instead of holding the user's deletion behind the TTL.
- Every pending terminal unsettled. settleAbortFence defers on
  terminalPersistencePending for ALL statuses — including `aborted`, whose
  route-side persistence the previous guard missed.

Also: a direct Redis regression for scheduled stale-pause retention (the P2
test gap), and the PR description no longer claims to carry every commit.

* fix: lease-token lifecycle fences — same-generation isolation, admission fence, undelivered-Stop retention, legacy abort enrollment

Fifth external review round; four P1s, handled as the requested consolidated
lifecycle-fence pass.

- Lease tokens. Marker fields were (streamId, createdAt), shared by every
  contender on the same generation — completion and Stop, or two racing Stops —
  so a losing contender's cleanup erased the winner's still-live marker. Every
  registrant now carries a unique lease token in the field and may only ever
  clear its own lease; unqualified legacy clears cannot touch qualified entries.

- Admission fence. Authentication can pass before the deletion barrier goes up,
  and the durable createJob is several async steps later — a deletion quiesce in
  that window saw neither an active job nor a marker and could cascade before
  the admitted request created its job. The controller now registers an
  admission lease and THEN rereads the deletion barrier: the ordering guarantees
  either this request observes the barrier (403, lease released, slot/claim
  cleanup) or the quiesce observes the lease and defers. Held until createJob is
  durable; released on every refusal and initialization-error path. Fail closed
  when the lease itself cannot be registered (503 retryable).

- Undelivered-Stop retention. abortJob released its lease in a finally even when
  delivery AND publication had provably failed — the job reads terminal
  (invisible to active-set scans), a user Stop writes no durable abort fence,
  and the remote owner keeps generating and will persist its abort-catch writes
  whenever the signal finally lands. The lease is now retained in exactly that
  case, and each resignal attempt heartbeats a fresh lease so the fence outlives
  the TTL for as long as delivery is still being driven. The abort-winning
  turn's own loser-side pending saves are additionally fenced in the controller
  catch (best-effort — those writes are already in flight).

- Legacy abort enrollment. abortMiddleware (assistants abort route fallback for
  non-assistants endpoints) awaited abortJob and then spent usage and saved the
  stopped response AFTER the abort's lease was released. Both writes now run
  inside `beforePublish`, between the abort CAS and publication, covered by the
  same lease as every other abort.

Barrier tests, each verified to fail without its fix: same-generation lease
isolation (store), racing two-Stop loser cleanup (manager, stale-read forced
CAS race), undelivered-Stop lease retention, resignal heartbeat, admission
refusal with lease-before-reread ordering plus release-on-durable-create, and
legacy-abort persistence inside beforePublish.

* fix: heartbeat-backed owner-lifecycle leases close the settlement handoff races

Sixth external review round: the remaining P1 interleavings were one structural
problem — lease handoffs that were not atomic — resolved as the requested
consolidated lifecycle-lease pass.

- Quiesce reads leases BEFORE the active-job scan. The admission-lease -> durable
  -job handoff is only atomic against a reader in the OPPOSITE order of the
  writer: writers hold the lease strictly until the job is active-set visible,
  so leases-first shows every interleaving either the lease or the job.
  Jobs-first allowed a request to create its job after the scan and release its
  lease before the count — hiding both, cascading, and letting the new
  generation persist into a deleted account.

- The abort acknowledgement is fenced by the owner-lifecycle lease. Redis ACKed
  the moment the owner's AbortController tripped; the stopping side released its
  lease on that ACK while the owner's asynchronous abort-catch persistence was
  still ahead. The transport now awaits a manager-installed pre-ACK hook that
  registers a DETERMINISTIC owner lease (exactly one owner exists per
  generation, and determinism is what lets the signal-time registrant and the
  owner's catch-side release agree across processes) before the acknowledgement
  is persisted or published; an owned same-replica abort bridges to the same
  lease before tripping its local controller. Both generation-owner catches
  (fresh turn, resume) release it once their writes land.

- A failed replacement handoff no longer orphans the predecessor. The atomic
  replacement removes it from active storage, and an unconfirmed handoff
  terminalizes the replacement too — leaving nothing a quiesce could discover
  while the predecessor's provider may still be generating. Its owner lease is
  now retained at the point the receipt fails delivery; the owner replica renews
  it through the pre-ACK fence when the signal finally lands.

- Leases HEARTBEAT while held. The five-minute store TTL only bounds a crashed
  holder; live persistence — a stalled save, a long deferred title — must never
  outlive its own fence. holdUserFinalization registers and renews every minute
  until released; the controllers' completion/error/admission leases all hold.
  The undelivered-Stop retention moved to a deterministic `stop` lease that
  every resignal attempt renews and the first successful one clears (no more
  opaque leases accumulating to TTL), and a THROWN abort transition releases the
  contender lease instead of leaking it.

- The user-document abort-fence mutations now invalidate the auth user-doc
  cache, matching every other user-doc write.

Barrier tests, each verified to fail without its fix: quiesce lease-scan
ordering (plus the observed-lease defer), pre-ACK fence ordering at the
transport, owned-abort owner-lease bridging, replacement-handoff predecessor
retention, held-lease heartbeat past the TTL, and failed-then-successful
resignal reaping the retained stop lease.

* fix: one manager-owned owner-lease span across every abort delivery path

Seventh external review round; four lifecycle-fence gaps, closed by making the
owner-lifecycle lease a single manager-owned, heartbeat-held span.

- Fail-closed acknowledgements. The pre-ACK hook registered a one-shot lease and
  the transport ACKed even when it failed; a same-replica owned abort likewise
  swallowed registration failure. The hook now acquires a HELD owner lease
  (heartbeat until the owner's catch releases it via releaseOwnerLease) and a
  rejection SUPPRESSES the acknowledgement — the stopping side stays retryable
  behind its retention lease, and every resignal re-drives the handler. The hook
  also stopped gating on `job.createdAt === generationId`: during a replacement
  handoff the store holds the replacement while the abort targets the
  predecessor, and that gate silently skipped exactly the generation being
  acknowledged (the store job is owner identity, never a generation gate). A
  local owned abort acquires the same held lease before tripping its provider;
  post-CAS the trip cannot be withheld, so acquisition failure downgrades
  delivery and the retention handoff keeps the user fenced.

- Committed-but-lost-reply disambiguation. A thrown abort transition released
  the contender lease as if nothing had happened, but a Lua CAS can commit and
  lose its reply — an aborted job invisible to active-set scans whose provider
  was never signalled, with no fence left. The throw path now re-reads the exact
  generation: only a job still live under the caller's identity proves no
  commit; committed or ambiguous outcomes hand the fence to the deterministic
  stop lease (kept on the contender lease if even that fails) before rethrowing.

- Replacement handoff covered end to end. A LOCAL replacement abort acquires the
  predecessor's held owner lease before the trip (failure reports the receipt
  undelivered, engaging retention). Failed-handoff retention is no longer a
  swallowed one-shot: it heartbeats with the durable acknowledgement proof as
  its renewal predicate — acquisition failures keep retrying for as long as the
  fence is needed, and the retainer stands down (without clearing the shared
  field) once the remote owner ACKs and thereby holds its own lease.

- A LOCAL resignal delivery hands off to the owner lease BEFORE clearing the
  retained stop lease, and keeps the stop lease when that handoff fails.

Barrier tests: hook rejection suppressing the ACK, commit-then-lost-reply
retention with its provably-uncommitted counterpart, local-resignal owner
handoff ordering, pre-ACK owner lease held past the store TTL until release
(and provably stopped after), and failed-handoff retention retrying on its
heartbeat — verified fail-before/pass-after by stashing the fixes.

* fix: finish scheduled chat lifecycle hardening

* fix: close scheduled chat review follow-ups

* fix: generation-fence abort recovery evidence

* test: wait for settled approval tool output

* test: preserve scheduled init reconciliation option

* refactor: rebuild scheduled chats on durable agent triggers

* test: reset MCP cache mock between cases

* test: isolate scheduler startup in server specs

* fix: harden scheduled run lifecycle

* fix: normalize schedule capacity conflicts

* test: type schedule collision fixture

* fix: re-fence scheduled resume and expiry

* fix: fence scheduled resume handoffs

* fix: release superseded manual schedule leases

* fix: release failed run-now claims

* fix: release superseded engine claims

* fix: repair schedule dialog interaction and rework its form

The agent picker was unusable: ControlCombobox portals its popover to the
body by default, which lands it outside the dialog's Radix focus trap. Clicks
passed through it, it could not be tabbed into, and the trap fighting Ariakit
for focus locked the page up on selection. The prop is documented for exactly
this case — pass `portal={false}` and give the dialog `overflow-visible`, as
ProjectButton already does. The time and day dropdowns defaulted the same way.

Alongside that:

- Extract the agent builder's instructions editor (special-variable menu plus
  expand-to-fullscreen) into a controlled `VariableEditor` and use it for the
  schedule prompt. Insertions now route through `onChange`, so react-hook-form
  sees them — the schedule PATCH is built from `dirtyFields`, and a `setValue`
  that skipped dirty tracking would have dropped an inserted variable silently.
- Replace the hand-rolled frequency buttons with the shared `Radio`. They marked
  the selection with `bg-surface-hover` on an outline button whose hover is the
  same token, so the selected option was indistinguishable from a hovered one;
  `Radio` is also a real radiogroup rather than four `aria-pressed` toggles.
- Wrap the fields in a real `<form>` and associate the footer button by id, so
  Enter submits. Group the frequency, day and time controls in fieldsets.
- Add placeholders for name and prompt, match the textarea fill to the other
  fields, and label the hourly case as minutes past the hour.
- Widen the dialog to `md:max-w-3xl` and pair name with agent so the form fits
  without scrolling on desktop.
- Move scheduled chats below skills and above prompts in the side nav.

The new dialog spec fails when the portal fix is reverted.

* test: cover scheduled and subagent deletion drains

* refactor: own the form-control appearance in the client primitives

Addresses the codex finding on ScheduleDialog: a feature-local `FIELD_CLASS`
restated the `Input` primitive's border, radius, height and background so it
could be pasted onto the schedule dropdowns, leaving those controls with no
connection to the primitive they were imitating.

Move that appearance into `packages/client/src/components/Field.ts` as the
single source `Input` and `Textarea` now compose, and give `Dropdown` and
`ControlCombobox` a `variant="field"` that applies it. The schedule dialog
passes the variant and carries no class strings of its own.

This also repairs a break the dev merge would otherwise have introduced: the
newer `Dropdown` splits `className` (wrapper) from `triggerClassName`, so the
old pasted classes would have landed on the wrapper and left the triggers
unstyled.

The semantic-token guard now watches the shared module and asserts each
primitive still composes it, which covers more than the two files it read
before.

* fix: keep an explicit schedules disable from becoming an opt-in

`use` is two things at once for a dual-purpose runtime interface field: a
permission bit, which DB overrides strip, and the runtime disable signal that
`getLimits` reads. Stripping it from `{ use: false, maxPerUser: 2 }` leaves an
object, and `getLimits` treats any object without `use: false` as enabled — so
an admin override written to stop scheduled billing for a role or user started
it instead.

Collapse an explicit disable to the boolean form before the strip, on both
paths that accept it: the `interface.schedules` field patch, which admitted the
object wholesale because bare runtime paths deliberately bypass the permission
gate, and the overrides merge, which reached the composite-field branch and
kept `maxPerUser`. Objects that only narrow limits are untouched, so a
principal can still be given a smaller cap.

Both regressions fail without the normalizer.

* fix(schedules): Wave A — null-balance CAS, atomic paused-card clear, clustered erasure sweep

Slice 1 (thread r3804518381): route the existing-null balance initialization
through a { user, tokenCredits: null } compare-and-set instead of a blind $set,
so a concurrent initializer/charge landing between the preflight read and the
write is never handed back its spent starting balance. On a CAS miss the
preflight re-reads the winner. The absent-record $setOnInsert path and the
credited-record refill-config sync are unchanged. Adds the initializeNullBalance
adapter (no upsert) and regression coverage for winner/miss/sync cases.

Slice 2 (thread r3804518388): updateScheduleById now drops a `requires_action`
lastRun projection atomically with the configRevision bump. Any pause present at
edit time was projected under the pre-edit revision and can never be replaced by
its own revision-fenced terminal outcome — a disabling edit would strand the
card on "Needs approval" forever. Implemented as classic-operator CAS branches
(DocumentDB rules out a conditional pipeline $unset), fenced on the card STILL
being the pause so a terminal outcome or newer occurrence that races in is
preserved. Terminal history survives untouched.

Slice 4 (thread r3803826204): expose initializeScheduleErasureSweep from the
schedule runtime facade and start it in every clustered (experimental) worker
after Mongo is up. It re-drives eraseScheduleIfDrained for soft-deleted rows so
a hidden prompt cannot outlive its drain when the delete/erase-on-settle
attempts miss. It arms nothing else and never infers owner death from a
process-local missing job (isTopologySafeToArm gates that).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq

* fix(schedules): Wave B.1 — reversible account-deletion schedule suspension

Thread r3804518383. Account-deletion quiesce marked every schedule `deleting`,
disabled it, and cleared nextRunAt destructively. When a later cascade step (or
the drain itself) failed, the controller cancelled the user-deletion fence —
restoring the user — but the schedules stayed `deleting` and were erased by the
sweep, silently losing all of a live user's scheduled prompts.

Replace the destructive marking with a REVERSIBLE, token-fenced suspension:

- suspendUserSchedulesForDeletion(userId, token) snapshots each schedule's prior
  enabled/nextRunAt under a per-attempt token, then fences firing (disable, clear
  nextRunAt, rotate claimToken). It never sets `deleting`, so a suspended row is
  not erasure-eligible. Snapshotting reads then bulkWrites (a classic update
  cannot copy field values under DocumentDB), fenced per row so an already-
  suspended/soft-deleted/edited row is left alone; idempotent per token.
- restoreUserSchedulesFromDeletion(userId, token) reverses it, re-enabling and
  re-arming only rows still carrying the exact attempt token and not independently
  deleted — so an owner-deleted or newer-attempt-suspended schedule is never
  resurrected.
- deleteUserController generates the attempt token, passes it to quiesce, and on
  any failure that cancels the user-deletion fence restores the suspended rows. A
  successful deletion hard-deletes them (and their snapshots) via the existing
  cascade and never restores.

Adds a `deletionSuspension` embedded field (excluded from the wire schedule),
data-method regression tests (suspend/restore/fence/idempotency/no-resurrect),
and controller tests (restore on drain-false and post-quiesce cascade failure,
no restore on success).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq

* fix(schedules): Wave B.2 — wire the interactive Stop persistence protocol

Thread r3804255932. The durable Stop primitives (requestRunAbort 'stop',
getScheduleRunAbortState, markRunAbortPersisted) already existed but production
only used requestRunAbort(..., 'deletion'). An interactive Stop flipped the job
to `aborted` and then persisted its partial message + checkpoint inside
`beforePublish`, without ever stamping the schedule Stop or acknowledging it —
so reconciliation, the generation owner, or a concurrent schedule/account
deletion could terminalize the run and release its capacity (and erase data)
mid-write.

Wire the request -> persist -> acknowledge -> settle barrier through the
schedule runtime (the route never touches raw Mongo):

- Expose beginScheduledStop / acknowledgeScheduledStopPersistence on the service.
- The abort route stamps the Stop BEFORE signalling abortJob (a serialized
  'in_progress' loser returns 409 STOP_IN_PROGRESS without a second abort),
  acknowledges only after beforePublish persistence succeeds, and on a
  persistence failure leaves the barrier unresolved so the run stays preserved
  (client retries; stale-owner timeout is the bounded recovery). A failed abort
  releases the stamp it placed so a replacement/retry is never blocked through
  its predecessor.
- recordScheduleOutcome (the owner settlement path) now waits, bounded, for the
  Stop acknowledgement before terminalizing; a resolved/non-stop/stale marker
  proceeds immediately. The paused Stop settles only after its own ack.

Adds service-layer barrier tests and route-level ordering/persistence-failure/
in-progress tests; the data-layer serialization is already covered.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq

* fix(schedules): Wave C.1 — reconcile durable trigger delivery with the reservation

Threads r3803826192 (manual limiter) and r3804255924 (PII/moderation), plus the
independently-found long-Retry-After race. fireSchedule reserves a `started` run
and a global capacity slot BEFORE the durable trigger delivery reaches the chat
route, where an interactive limiter (manual Run Now), PII, or moderation can
reject it before any generation job exists — dead-lettering the delivery while
the run sat `started` until the 30-minute orphan sweep mislabeled it interrupted.
And a valid delivery deferred by Retry-After (up to 24h) could be orphan-settled
and have its capacity released, then fire anyway.

Translate durable delivery state into the schedule outcome:

- Store the deterministic trigger deliveryKey on the ScheduleRun reservation
  (computed from the envelope BEFORE enqueue, so an ambiguous commit still has it).
- Add a getTriggerDelivery engine dep (wired to the merged trigger service's
  getDelivery) that reads the durable delivery by key.
- Schedule reconciliation, for a jobless `started` run: staging/pending/leased →
  admission is live, never orphan; dead → record `error` from the durable
  lastError and release capacity promptly (no 30-minute wait), through the
  ordinary outcome/auto-disable path; succeeded or no record → the existing
  legacy orphan policy (interrupted only past the cutoff); a delivery lookup
  failure defers rather than orphaning a possibly-live delivery. Limiter/PII/
  moderation middleware writes no schedule state.

Adds reconcile state-mapping tests (dead/pending/leased/staging/succeeded/none/
lookup-failure) and a fire test that the reservation's deliveryKey equals the
enqueued delivery's idempotency key.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq

* fix(stream): Wave C.2 — durable retry/ack for terminal host lifecycle actions

Thread r3804518375. Approval expiry won the `requires_action → aborted` CAS and
then invoked the host hook best-effort: `runApprovalExpiredHandler` swallowed a
failure, and because later sweeps enumerate only `requires_action` jobs, the now-
aborted job was never offered again. In the clustered entrypoint (no schedule
reconciler) the ScheduleRun stayed `requires_action` and its retained job
persisted indefinitely.

Make the host lifecycle work durable rather than schedule-specific:

- Add a generic `terminalHostActionPending` marker, set ATOMICALLY in the same
  terminal transition (ApprovalLifecycle.expireWithIdentity), only when a host
  adapter is installed.
- Retain and index such jobs: both stores keep them out of terminal reaping and
  expose getTerminalHostActionJobs(); Redis adds a set + extended (24h-bounded)
  TTL, in-memory a bounded 24h retention so a permanently-failing hook cannot leak.
- The manager clears the marker only after the adapter acknowledges success,
  fenced by generation identity (clearTerminalHostAction), so a replacement
  generation can neither clear nor execute its predecessor's action.
- cleanup()/expireStaleApprovals() enumerates unacknowledged terminal host actions
  across restarts and replicas and retries the idempotent hook; the relay only
  re-invokes while the marker is unacknowledged, so a successful ack prevents
  duplicate work. Store-won expiry marks it too, so a loser-replica relay still
  crosses the hook.
- Terminal SSE notification continues regardless of host-hook outcome.

Covers in-memory behavior (retry after failure, restart/other-replica retry, ack
prevents duplicates, identity fence, terminal notification on failure, no marker
accumulation for non-scheduled jobs) and updates the Redis cluster-membership
contract test for the new index.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq

* style(schedules): satisfy import sorting in fire.ts and fire.spec.ts

CI "Static checks" failed on IMPORT_SORT for the two files Wave C.1 added imports
to (the AgentTriggerEnvelope type import and getAgentTriggerIdempotencyKey).
Applied scripts/sort-imports.mts to exactly those files — imports-only reordering,
no behavior change. Other files reported by a repo-wide check are pre-existing on
dev and deliberately left untouched so this PR is not widened.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq

* fix(schedules): repair the deletion CLI and order restore before the fence release

Addresses three findings from the fresh Codex review.

P1 — config/delete-user.js called methods.disableUserSchedulesForDeletion, which
Wave B removed in favor of suspendUserSchedulesForDeletion. The file is
@ts-nocheck and its spec mocked the removed name, so neither typecheck nor tests
caught it; the real CLI would throw a TypeError before deleting anything and then
only unwind the fence. The CLI now uses the tokenized protocol: it mints a
suspension token, suspends with it, and restores that exact attempt's rows in its
finally block when the deletion does not commit. Its spec mocks the real methods,
so the breakage can no longer hide.

P2 — both the HTTP controller and the CLI released the user-deletion fence BEFORE
restoring schedules. That fence is what refuses new schedule writes/claims, so the
gap let an owner PATCH edit a still-suspended row and have its enabled/next-run
state overwritten by the older snapshot, and let a second deletion attempt
re-suspend under a new token — making the first restore a no-op and stranding the
disabled snapshot permanently. Restore now runs first, while writes are still
fenced.

Hardening for the same defect class across a crash: suspendUserSchedulesForDeletion
now ADOPTS an existing suspension's snapshot when re-suspending a row abandoned by
an earlier attempt, instead of re-capturing the row's current (already-suspended)
state. Without this, an attempt that died before restoring would have its
successor snapshot "disabled, no next run" and permanently strand the schedule.

Tests: CLI restore-before-fence ordering and no-restore-on-success; the same
ordering assertion on both controller post-quiesce failure paths; a data-method
regression that a second attempt adopts the abandoned snapshot and restores the
original enabled/next-run state.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq

* fix(schedules): converge dead deliveries in topology-safe maintenance

Codex finding: the `dead` delivery mapping added in Wave C.1 lives only inside
startScheduleEngine's reconciler, but the clustered entrypoint arms no engine — it
runs erasure-only maintenance. A delivery queued before a restart into clustered
mode and then rejected before generation creation (interactive limiter, PII,
moderation) dead-letters while its ScheduleRun stays `started`, holding a global
capacity slot indefinitely for an ordinary non-deleting schedule.

Add a dead-delivery convergence pass to the erasure sweep, so every topology that
runs schedule maintenance settles it. The pass is POSITIVE-EVIDENCE-ONLY and is
therefore safe where absence-based reconciliation is not: a `dead` delivery is
durable shared state proving no generation owns the reservation. It settles only
when the job is confirmed absent or identity-mismatched (an identity-matched job
still owns the run), defers on an unknown job lookup, on an in-flight abort, and
on an in-flight resume hand-off, ignores legacy reservations with no deliveryKey,
and applies a short grace so an accepted delivery still creating its generation is
never settled mid-handoff. Auto-disable policy is deliberately left to the armed
engine; this path records the failure and frees the slot.

Deliberately does NOT touch api/server/experimental.js — the clustered entrypoint
already starts this sweep, so the convergence arrives through the existing
initializer and the shared-file footprint stays as-is.

Tests: settles a dead delivery as error under an explicitly UNSAFE topology,
leaves live deliveries alone, never settles under an identity-matched running
generation (delivery is not even consulted), defers an in-flight abort, and
ignores a reservation with no deliveryKey.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq

* fix(stream): refresh host-action retention on each retry attempt

Codex finding: unacknowledged terminal host-action evidence was capped at the
24h pause TTL, so a host dependency (Mongo) unreachable for longer than that let
the Redis key — and its pending marker — expire with no generation-fenced
acknowledgement, stranding the ScheduleRun where no reconciler is armed.

Measure retention from the LAST retry rather than from the terminal transition:
enumerating a pending host action IS the retry attempt, so both stores refresh
its retention as they hand it to the hook (Redis re-EXPIREs the job key; the
in-memory store stamps terminalHostActionRefreshedAt and bounds from it). Evidence
therefore survives as long as some replica is still actively retrying, while a
deployment that stops sweeping entirely still lets it age out — so this does not
reintroduce the unbounded leak the cap existed to prevent.

Test: after a failed hook, a later cleanup pass keeps the marker pending and moves
its retention basis forward.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq

* fix(schedules): defer settlement when the Stop barrier times out

Codex finding: waitForStopPersistence returned after its 5s poll budget even when
the Stop was still fresh and unacknowledged, and recordScheduleOutcome then went
straight on to terminalize the run — releasing its capacity, deletion, and erasure
barriers while beforePublish may still have been writing. Slow checkpoint cleanup
is indistinguishable from a dead route on that signal, so the timeout was being
treated as if the barrier had been satisfied.

The poll budget now means "undecided", not "clear". On timeout with a fresh,
unacknowledged Stop the barrier DEFERS: recordScheduleOutcome returns false
without recording, leaving the run active/preserved. Settlement then happens
either when the route acknowledges, or once the existing stale-owner cutoff
(ABORT_OWNER_PRESUMED_ALIVE_MS) authorizes a later attempt — which the loop
already treats as clear-to-settle. Callers with durable retry (the approval-expiry
host action, reconciliation) re-drive it, so a deferral converges rather than
stranding the run.

Test: a fresh Stop that never acknowledges within the budget reports not-settled
and records no outcome.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq

* fix(schedules): require definite delivery failure, extract its message, converge deferred Stops

Third Codex round. Three findings in code from this closeout, plus one pre-existing
P1 that is a one-line operator-safety fix.

P1 — dead-delivery settlement demanded too little evidence. `dead` does not prove a
request was rejected: the trigger host marks response timeouts and invalid success
responses `certainty: 'ambiguous'`, and the engine dead-letters those once retries
are exhausted. The erasure sweep treated every dead letter as positive evidence, so
an ambiguous one sitting over a generation a peer had accepted could terminalize the
run and release its capacity mid-flight. It now settles only on a DEFINITE rejection,
unless job absence is deployment-authoritative (safe topology), where the
confirmed-absent job is itself the evidence.

P1 — `lastError` is an `AgentTriggerDeliveryFailure` object, not a string. A
duplicated local interface declared it `string` (against CLAUDE.md's no-duplicate-
types rule), so both the sweep and the engine reconciler passed the object into the
String-typed run/schedule `error` fields; Mongoose would reject the cast, the per-row
catch would swallow it, and the run would keep its global capacity slot. The dep type
now reuses the canonical `AgentTriggerDeliveryFailure` and both call sites pass
`.message`. Re-typing immediately surfaced a stale test that had asserted a string.

P1 (pre-existing) — the base-config global stop is honored in `getLimits` via
`isRuntimeDisabled` rather than a literal `=== false`. The stop has two shapes, and
deepMerge turns base `{ use: false }` plus a principal override of `true` into
`{ use: true }`, so the literal check reported the feature enabled and Run Now
dispatched straight through fireSchedule, bypassing the operator's emergency stop.
Now the same predicate the engine gate already uses.

P2 — a Stop whose settlement DEFERRED past the poll budget had no convergence path
where no reconciler is armed. `acknowledgeScheduledStopPersistence` now optionally
re-drives the terminal outcome once the barrier clears; `recordRunOutcome` is
match-guarded and idempotent, so an owner that already settled makes it a no-op. The
abort route passes it for a running generation; a paused job still settles explicitly.

Tests: ambiguous dead letters refused under unsafe topology but settled when absence
is authoritative, definite rejections settled either way, the failure message carried
through, and the abort route's re-drive present for running / absent for paused.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq

* fix(schedules): converge terminal runs in clustered workers, retry suspension restore

Fourth Codex round on 74f7af8. Both findings are in code from this closeout.

P2 - a clustered worker never settled a run whose generation finished but whose
outcome write failed. `recordScheduleOutcome` retries three times and then returns
false; the owner honors that by PRESERVING the terminal job as the only surviving
evidence, and the armed engine's reconciler replays exactly that. The clustered
entrypoint arms no engine, and its sweep only covered deleting schedules
(settleAbandonedRuns) and dead deliveries - an identity-matched job was skipped
outright. So an ordinary live schedule's run stayed `started`, held its GLOBAL
capacity slot, and kept a preserved job that carries no `completedAt` and is
therefore invisible to the store's finished-job sweep: both leaked until store
expiry.

The live-schedule pass now also converges from a retained terminal job, mirroring
the reconciler branch it stands in for: honor the owner's stamped outcome over the
generic status (so a balance refusal still walks its streak rather than resetting
it), clear the reserved conversationId when the generation never emitted its
created event, and delete the retained job only AFTER the outcome write is durable.

This stays positive-evidence-only and safe in every topology. Presence, not
absence, is the evidence: an identity-matched job is authoritative wherever it is
observed - a shared store shows the real generation, a process-local store can
only be showing this process's own - which is why it needs no
canInferOwnerDeathFromMissingJob fence, unlike the absence-based paths. The
in-flight abort/resume fences still defer, so an `aborted` job cannot settle a run
whose owner may still be persisting. Both cases now share one pass over one window
rather than two, and `retainedOutcome` moved to types.ts so the sweep reuses the
engine's mapping instead of duplicating it (and stays independent of the engine).

P2 - a failed restore stranded a live account's schedules. Cancelling an account
deletion restores the suspended rows while the deletion fence is still armed and
then releases that fence; nothing re-drives the restore afterwards, so one
transient write failure left the user with silently disabled, next-run-less
schedules. The restore is now retried at the single choke point both the HTTP
controller and the CLI share. Retrying is safe because each attempt re-reads only
the rows STILL carrying the token: a partially-applied unordered write converges
on exactly the stragglers, and a fully-applied one finds nothing.

The fence is still released when every attempt fails, deliberately: retaining it
would refuse the live account's schedule writes AND make beginAgentTriggerUserDeletion
report `in_progress` forever, blocking the retry that is the convergence path (a
later attempt adopts this snapshot, so its cancel restores these exact rows). Both
callers now log the user id and suspension token so the state stays recoverable by
hand if that never happens.

Tests: retained terminal job settled and its evidence released, stamped outcome
preferred over the generic status, stamped failure reason carried, reserved
conversationId cleared for a never-created conversation, identity-mismatched
terminal job ignored, abort-in-flight deferred; restore retried past a transient
write failure and converging on a partially applied restore.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq

* fix(schedules): converge unprojected pauses and release replayed bookkeeping jobs

Two of the four findings I flagged as pre-existing to my closeout commits. Both are
in code this PR introduces, so merging would have shipped them; both are the same
capacity/evidence-leak class the last four rounds have been closing. The remaining
two (slotless rows escaping the per-user cap, and the non-rotating `started`
reconciliation bucket) are genuinely latent and deliberately left alone.

An UNPROJECTED PAUSE held a global capacity slot forever. The pause projection is
what moves a run row off `started`; `recordScheduleOutcome` already retries it, but
the request controller discarded the result, so three failed attempts were dropped
silently. The armed engine's reconciler replays that state, but the clustered sweep
did not: a paused job is not terminal, so the retained-job path returned without
settling, and the dead-delivery path never inspects an identity-matched job at all.
The row stayed `started` with no cutoff that would ever clear it.

The call site now surfaces the failure, and the sweep converges it, mirroring the
reconciler's pause branch: project `requires_action` (which frees the slot) but do
NOT release the job's evidence — unlike a terminal job it is still live, awaiting an
approval. The resume hand-off fence still defers, so re-projecting cannot release a
slot a continuation just claimed.

The BOOKKEEPING REPLAY pass leaked its retained job. A run reaches that pass only
because its owner crashed before bookkeeping — which is also before it could release
the job it retained for exactly this recovery. The active-run pass clears its own;
this one never did, and a preserved job is deliberately kept WITHOUT `completedAt`
so the store's finished-job sweep cannot reap it early, so nothing else ever would.
It now clears after `finalizeBookkeeping` succeeds — identity-guarded, a no-op when
no job is retained, and skipped entirely when the replay itself failed, since the
retained job is the only surviving evidence in that case.

Tests: pause projected and the slot freed while the live job's evidence is kept,
paused job deferred during a resume hand-off, retained job released once replayed
bookkeeping is durable, and retained job kept when the replay fails.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq

* fix(schedules): fence the clustered pause replay against a concurrent resume claim

Codex round 5, on code I added in f735341. The finding is correct and the exposure
is one I introduced.

The pause replay lives in a sweep that runs in EVERY clustered replica, so several
sweepers can observe the same unprojected pause. Its guards were all derived from an
in-memory row SNAPSHOT: hasResumeHandoffInFlight reads the snapshot's
resumeClaimedAt, and recordRunOutcome's pause branch matched any row currently in
`started`/`requires_action` with no fence of its own. The race: sweeper A projects
the pause and frees the slot; the owner's approval then claims a fresh one
(markRunResumeClaimed takes the row to `started` WITH resumeClaimedAt in one write);
sweeper B, still holding the pre-projection snapshot, passes its hand-off check and
replays — `$unset: { capacitySlot, resumeClaimedAt }` under a continuation that is
already running. The run reverts to `requires_action` while its generation proceeds
outside global capacity.

The engine's reconciler makes the same call and has the same snapshot-derived guard,
but v1 arms exactly one engine, so it has no concurrent racer; the sweep is the first
thing to run this transition in parallel. Left the engine alone rather than widening
the change: its `requires_action` re-affirmation is deliberate and single-writer.

Fixed where the race is, in the write itself. `recordRunOutcome` takes an optional
`requireNoResumeClaim`, which adds `resumeClaimedAt: { $exists: false }` to the pause
filter, and the sweep sets it. Because the stamp is written in the SAME update that
moves the row to `started`, its absence is atomic proof no resume owns the row. The
flag is deliberately NOT set by the generation owner: its own re-pause legitimately
clears the stamp as the hand-off's completion signal.

The fence cannot block the recovery it exists to enable: markRunResumeClaimed only
matches `requires_action`, so a genuinely stuck `started` row can carry no resume
claim — it is in fact blocking its own approval until this replay frees it.

Tests: a replay from a stale snapshot leaves a resume-claimed row's status, slot, and
claim stamp intact, while a stuck row with no claim still recovers; and the sweep is
asserted to send the fence.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq

* fix(schedules): fence stale resume claims by age, not existence

Codex round 6, on the fence I added in ee00c60. Correct again, and the defect is
the mirror image of the one it fixed.

The fence was an EXISTENCE check (`resumeClaimedAt: { $exists: false }`) while its
caller's guard is a FRESHNESS check (hasResumeHandoffInFlight, bounded by
RESUME_HANDOFF_STALE_MS). They agree while a claim is fresh and disagree once it is
abandoned: a worker that dies after markRunResumeClaimed takes the row to `started`
and stamps resumeClaimedAt — but before the continuation resumes or
releaseRunResumeClaim rolls it back — leaves the stamp set forever. Past the bound
the sweep correctly stops deferring and tries to recover the row, but the write
rejected it purely because the field still existed. The row stayed `started` holding
its global capacity slot, and its approval was unresumable for good, since
markRunResumeClaimed only matches `requires_action`. That is exactly the stuck state
this replay exists to clear, so the fence had reintroduced it for the crashed-resume
case.

`requireNoResumeClaim: boolean` becomes `resumeClaimStaleBefore: Date`, and the
filter matches a row with no claim OR a claim older than that cutoff. The sweep
passes the SAME bound its in-flight check uses, so the two can no longer disagree.
A genuinely racing claim is by construction fresh — it is created after the sweeper's
snapshot — so the race from round 5 stays closed.

Tests: a row whose resume claim was abandoned by a dead worker now recovers (status,
slot and stamp all cleared), alongside the existing two — a fresh claim still repels
a stale-snapshot replay, and an unclaimed stuck row still recovers.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq

---------

Co-authored-by: Claude <noreply@anthropic.com>
2026-08-20 11:51:30 -04:00
Danny Avila
a5b10c78cf
🛡️ feat: Add Batched MCP Authority Proofs (#14688)
* feat(data-schemas): add MCP authority proof substrate

* feat(api): add default-off MCP authority fences
2026-08-07 12:23:25 -04:00
Danny Avila
c570517768
🐘 test: refresh FerretDB harness, registry-derived models, bulkWrite coverage (#14679)
* fix(data-schemas): refresh FerretDB harness model coverage, fix compile errors, add bulkWrite differentials

Track 2 of the search-stack plan (PLAN.md "FerretDB track"):

- Replace the three hand-rolled 29-model MODEL_SCHEMAS maps in
  multiTenancy/sharding/orgOperations.ferretdb.spec.ts with a shared
  getModelSchemas(mongoose) helper (misc/ferretdb/schemas.ts) derived from
  the live createModels() registry, so coverage tracks all 37 current
  models automatically instead of drifting. Matches the reference pattern
  in misc/documentdb/compat.documentdb.spec.ts.

- Fix the 3 compile-broken specs this uncovered: all three imported a
  `projectSchema` from '~/schema' that no longer exists (superseded by
  `chatProjectSchema`), which `tsc --noEmit` flags as TS2724 but the
  babel-based jest transform silently let through as `undefined`. Removing
  the hand-rolled maps removes the bad import as a side effect; verified
  clean with tsc across misc/ferretdb and misc/documentdb.

- Add misc/ferretdb/bulkWrite.ferretdb.spec.ts: differential specs for the
  five bulkWrite flows the plan names as actually at risk (import via
  bulkSaveConvos/bulkSaveMessages, bulkWriteAclEntries,
  bulkIncrementTagCounts, Transaction.insertMany, file-TTL bulkWrite via
  extendFilesTTL). Each flow runs identical operations against a real
  mongodb-memory-server (always) and, when FERRETDB_URI is set, against
  FerretDB, asserting normalized result equality. Multi-document
  transactions already degrade via the existing supportsTransactions
  probe — not duplicated here.

- Land the Spike A BSON-legibility findings (bson-legibility.md,
  bson-inventory.txt) from the bson-legibility-spike-6e38de worktree so
  decision 2's evidence is in-repo.

Verified against a real FerretDB 2.7.0 + postgres-documentdb 17 stack
(docker compose -f misc/ferretdb/docker-compose.ferretdb.yml): all 10
harness spec files pass individually, including all 10 bulkWrite.ferretdb
tests (5 mongodb-memory-server baselines + 5 FerretDB differentials). Full
packages/data-schemas src/ suite (1907 tests) unaffected.

* 📝 docs: make the BSON projection findings self-contained

The doc was written for readers who already knew the internal shorthand — it
opened on "Spike A executed, Spike B scoped" and referred to Options 1/2/3 and
"the handoff" without ever defining them, so a reader arriving from the repo
could not follow the argument or act on the recommendation.

Reframed around what the document actually investigates: the question is stated
up front, the three candidate mechanisms are named in a table before they are
compared, and the recommendation refers to them by name. No findings, numbers,
or SQL changed.

* 📝 docs: drop internal planning references from spec header

* fix(data-schemas): keep FerretDB harness schema derivation side-effect free

`getModelSchemas()` derived its map by calling `createModels(mongoose)`,
which carried three consequences the harness did not want:

- Model creation applies the tenant-isolation plugin to the module-level
  schema singletons, so every harness read and write inherited middleware
  that throws under `TENANT_ISOLATION_STRICT=true`.
- Registering on the default connection meant the benchmark's own
  `mongoose.connect()` auto-created 37 collections in the URI's base
  database, adding a database and dozens of collections to the very
  catalog metrics it measures.
- The unfiltered registry provisioned app-wide control-plane models
  (`SystemGrant`, `AuditLog`, `SkillSyncCredential`, `SkillSyncStatus`)
  into every org database.

The helper now builds the registry on a throwaway Mongoose instance,
returns schemas rebuilt from their own definition, options, and declared
indexes, skips the four app-wide models (validated against the registry so
a rename fails loudly), and memoizes the result.

Also in this pass:

- The "adds a new collection" migration test used `AuditLog`, which
  provisioning had already created, so it silently reused the production
  model and ignored its proposed schema. It now uses a fixture model absent
  from the registry and asserts the collection is missing beforehand and
  carries the proposed compound index afterwards.
- `bulkWrite` flows run inside `runAsSystem()`; they drive production
  methods unscoped, as a cross-tenant maintenance job does, and otherwise
  fail closed under strict tenant isolation.
- Phase 2's sparse-index assertion pinned a count the User schema no longer
  declares; it now checks that each index type round-trips.
2026-08-07 10:45:58 -04:00
Danny Avila
23d1ad473d
🍃 fix: Amazon DocumentDB Compatibility for Pipeline-Form Updates (#14495)
* 🍃 fix: Amazon DocumentDB Compatibility for Pipeline-Form Updates

- Rewrite acceptTerms without aggregation-pipeline update + $$NOW (null-guarded first-acceptance claim preserves the original timestamp under concurrent and repeat requests)
- Rewrite decrementTagCounts clamp-at-zero decrement as ordered two-op bulkWrite (clamp before guarded $inc)
- Rewrite extendFilesTTL TTL hold as projected read + per-doc guarded $set via tenantSafeBulkWrite, preserving only-widens/ceiling/cleared-stays-permanent semantics
- Log background index-build failures via Model 'index' listeners (previously swallowed silently, e.g. partialFilterExpression rejection on DocumentDB <5.0)
- Add misc/documentdb live-compatibility harness + assessment (AWS-cited)
- Fix latent file.spec helper bug: createdAt backdating was silently stripped by mongoose immutability

Closes #14488

* 🍃 fix: Harden DocumentDB-Safe Updates Against Cross-Call Races

Addresses Codex review on #14495:
- decrementTagCounts: normalize-null / $inc / clamp-negative op triple so
  interleaved decrements of the same tag converge on max(0, ...) exactly as
  the serialized pipeline did (clamp keys on count < 0, not count < amount)
- acceptTerms: guard the repeat-acceptance fallback with a non-null timestamp
  (exact complement of the claim guard) and retry the claim when a
  config/reset-terms.js reset races between the two updates, so acceptance
  never resurrects a reset cycle without a fresh audit timestamp

* 👷 ci: Suppress Ignored-File Warnings in Changed-File ESLint Run

Changed files under config-ignored paths (packages/data-schemas/misc/**)
emit "File ignored" warnings that fail --max-warnings=0.
2026-07-28 22:14:34 -04:00
Danny Avila
f7d59d3285
📦 chore: Update TypeScript Config for TS v7 (#12794)
- Enabled `esModuleInterop` in `client/tsconfig.json` for better module compatibility.
- Changed `moduleResolution` from `node` to `bundler` in `client/tsconfig.json`.
- Set `noEmit` to `true` in several `tsconfig.json` files to prevent output generation.
- Removed `baseUrl` from various `tsconfig.json` files to simplify path resolution.
- Updated path mappings in multiple packages to reflect new directory structures.

These changes aim to streamline TypeScript configurations and improve module resolution across the project.
2026-04-23 12:51:03 -04:00
Danny Avila
3bd2681272
🪐 fix: Replace $bitsAllSet ACL Queries for Azure Cosmos DB Compatibility (#12736)
* 🐛 fix: replace `$bitsAllSet` ACL queries for Cosmos DB compatibility

Azure Cosmos DB for MongoDB API does not implement the `$bitsAllSet`
query operator, so every permission check against Cosmos DB threw. The
five read paths in `aclEntry.ts` (`hasPermission`, `findAccessibleResources`,
`findPublicResourceIds`, and two sites in `getSoleOwnedResourceIds`) now
fetch candidate entries and apply the bitwise mask in application code.
This matches the existing FerretDB-compatible pattern.

Fixes #12729.

* 🐛 fix: delegate `findPubliclyAccessibleResources` to fixed DB method

`AccessControlService.findPubliclyAccessibleResources` inlined the same
`$bitsAllSet` query as the data-schemas layer, which fails on Azure
Cosmos DB for MongoDB. Delegate to `_dbMethods.findPublicResourceIds`
so a single implementation carries the Cosmos-compatible bitwise logic.

Refs #12729.

* 🐛 fix: move `$bitsAllSet` filter out of remote-agent aggregation

`enrichRemoteAgentPrincipals` used `$bitsAllSet` inside an aggregation
`$match`, which Azure Cosmos DB for MongoDB does not implement. Project
`permBits` through the pipeline and filter for `PermissionBits.SHARE` in
application code. The extra documents fetched are bounded by ACL entries
on a single agent resource, so the cost is negligible.

Refs #12729.

* 🧪 test: rename misleading public-dedup test and add real dedup coverage

The test previously named "returns deduplicated IDs even if the public
principal has multiple entries" only set up a single ACL entry, so it
did not actually exercise deduplication. Split into two tests: one for
the happy path (single entry with required bits), and one that bypasses
`grantPermission`'s upsert via `AclEntry.create` to confirm the
application-layer dedup Map handles genuine duplicates.

Refs #12729.

* 🧪 test: cover SHARE-bit filter in `enrichRemoteAgentPrincipals`

The `$bitsAllSet` match stage in `enrichRemoteAgentPrincipals` previously
guaranteed every aggregation row had SHARE; the Cosmos DB fix moved that
check into a JS `continue` branch with no direct coverage. Add a
dependency-injected unit test that stubs the aggregation with mixed
SHARE / non-SHARE / zero-bit rows and asserts only SHARE holders are
enriched and queued for backfill. Also includes a regression guard that
the `$match` pipeline stage no longer contains a `permBits` filter.

Refs #12729.

* ♻️ refactor: extract `filterByBitsAndDedup` helper for ACL reads

`findAccessibleResources` and `findPublicResourceIds` each inlined the
same bitmask-filter + `Map`-based dedup loop. Lift it into a private
`filterByBitsAndDedup(entries, requiredBits)` helper so the Cosmos-DB
compatible pattern lives in one place. Pure rename/extract — no
behavior change.

Refs #12729.

* 📝 docs: fix stale `\$bitsAllSet` references in FerretDB spec

The describe block and header comment in the FerretDB parity spec still
referenced `\$bitsAllSet queries` after the Cosmos DB compatibility fix
moved the bit mask into application code. Update the title to
\"Bitwise permission queries\" and rewrite the header comment to
describe the application-layer behavior being validated.

Refs #12729.

* ⚡ perf: push permission-mask filter back to the database via `$in`

The original fix for #12729 moved `$bitsAllSet` filtering into application
code, which meant every ACL read fetched the full set of rows for a
principal/resource and filtered in JS. For tenants with large ACL
collections this inflates wire transfer and heap.

Replace the JS filter with `permBits: { $in: permissionBitSupersets(X) }`.
For the 4-bit `PermissionBits` enum the `$in` list is at most 16 values
(8 for a single-bit mask like SHARE). `$in` is indexable and supported by
Azure Cosmos DB for MongoDB, so the filter runs on the server again —
restoring `.distinct('resourceId')` and `findOne()` semantics.

`permissionBitSupersets(requiredBits)` is memoized and exported from
`@librechat/data-schemas`. Callers restored:
  - `hasPermission`: back to `findOne` short-circuit
  - `findAccessibleResources` / `findPublicResourceIds`: back to `.distinct()`
  - `getSoleOwnedResourceIds`: back to the `$match` + `$group` aggregation
  - `enrichRemoteAgentPrincipals`: bit filter back in `$match`, JS `continue` removed

Refs #12729.

* 🧪 test: add `\$bitsAllSet` vs `\$in` parity + perf spec

Introduces `aclEntry.parity.spec.ts` — a side-by-side spec that runs the
legacy `\$bitsAllSet` query and the current `\$in`-based query against the
same `mongodb-memory-server` fixture and asserts identical output sets for
every affected method (`hasPermission`, `findAccessibleResources`,
`findPublicResourceIds`, `getSoleOwnedResourceIds`) across all 7 meaningful
permBits combinations.

Also logs median wall-clock time for the two query paths over 20 runs on
an 800-entry fixture, with a loose 3x guard against catastrophic
regressions. Initial local numbers: 1.05 ms vs 1.07 ms
(findAccessibleResources), 1.10 ms vs 1.05 ms (findPublicResourceIds).

Refs #12729.

* 🔒 hardening: freeze `permissionBitSupersets` cache + enum-shape guard

Two defensive changes from the comprehensive audit:

* Cached superset arrays are now `Object.freeze`d and the return type is
  `readonly number[]`. Previously the cached arrays were returned by
  mutable reference, so a caller that mutated the result would silently
  corrupt the process-wide cache for every subsequent permission check.
  `Object.freeze` turns that into a loud `TypeError` at the mutation
  site. All existing call sites pass the result directly to Mongoose's
  `$in`, which does not mutate.
* Added a module-load guard `if (MAX_PERM_BITS === 0) throw`. If
  `PermissionBits` is ever refactored to a `const` object or string
  enum, `Object.values(...).filter(isNumber)` would return `[]` and
  `MAX_PERM_BITS` would silently become 0, making every query match no
  rows and breaking every permission check. The guard fails loudly
  instead.

Also collapsed four identical JSDoc lines across `hasPermission`,
`findAccessibleResources`, `findPublicResourceIds`, and
`getSoleOwnedResourceIds` into a single `{@link permissionBitSupersets}`
reference.

Refs #12729.

* 🧪 test: add focused unit tests for `permissionBitSupersets`

The helper is the single point of correctness for every ACL read path
(every query uses `permBits: { $in: permissionBitSupersets(X) }`), so it
warrants direct coverage independent of the higher-level parity and
behavior specs. Six cases added:

* `requiredBits=0` returns all 16 values
* `requiredBits=15` returns `[15]` only
* every returned value is a bitwise superset of `requiredBits`
* full parity against the `$bitsAllSet` definition for every
  `required` in 0..15
* memoization: repeat calls return the same frozen reference
* frozen result throws `TypeError` on mutation attempts

Refs #12729.

* 🧪 test: tighten parity perf guard and document fixture constants

The `expect(currentMs).toBeLessThan(legacyMs * 3 + 50)` form was
dominated by the `+ 50` additive term at typical sub-ms query
latencies — at legacy=1ms, a 50x regression would still pass. Replace
with `Math.max(legacyMs * 5, 50)` so the multiplicative ceiling is
intact once the new path climbs out of the fixed noise floor.

Also added inline rationale for the `FIXTURE_SIZE = 800` and
`PERF_ITERATIONS = 20` constants.

Refs #12729.

* 🧹 chore: remove stale perf-guard comment, hoist rationale to describe

Commit a75f122 left a "Sanity check: A 3x multiplier" comment block
above the first perf guard, and 21852ac layered a second
"Multiplicative-dominant guard" block directly below it. The stale
block described the `legacyMs * 3 + 50` formula that no longer exists,
so both blocks coexisted and contradicted each other.

Delete the stale block from the first test, remove the redundant copy
from the second test, and lift the (now-single) rationale to the
enclosing `describe('performance')` block. Each test is now one
`expect` line — the rationale lives once, at the scope it applies to.

Refs #12729.

* 📝 docs: sharpen MAX_PERM_BITS guard message + test-sort consistency

Two NITs from the follow-up audit:

* The `MAX_PERM_BITS === 0` guard now names the specific refactors that
  could cause it (const object / string enum / all-string shape) and
  gives two concrete remediation paths (rewrite the reducer, or give
  `permissionBitSupersets` an explicit bit-width parameter). Previously
  the message just said "update permissionBitSupersets before
  continuing", which was vague.
* The `requiredBits=15` unit test now applies the same
  `[...result].sort((a, b) => a - b)` normalization as the other tests
  so the set-equality assertions are uniform. The function happens to
  return values in ascending order, but the helper's JSDoc does not
  promise ordering, so sorting before comparing is the correct
  defensive pattern.

Refs #12729.

* 🔒 fix: enforce `permBits <= MAX_PERM_BITS` at the schema level

Codex flagged a real silent regression: `permissionBitSupersets`
enumerates `$in` candidates only in `[0, MAX_PERM_BITS]` (currently 15),
so any ACL row with bits above the enum — e.g. `permBits = 31` from a
future role or a manual DB write — would be excluded from reads that
`$bitsAllSet` would have matched. Current write paths only produce
values in `[0, 15]` via the `PermissionBits` / `RoleBits` enums, so
no live data is affected, but the schema did not prevent out-of-range
writes, so the divergence was reachable.

Fix: add `min: 0`, `max: MAX_PERM_BITS`, and an `isInteger` validator to
the `permBits` field in the AclEntry schema. `MAX_PERM_BITS` is derived
from `PermissionBits` the same way `permissionBitSupersets` computes it,
so when a new bit is added to the enum both the read-side enumeration
and the write-side bound expand together.

Tests: four new cases cover the upper bound, over-limit, negative, and
non-integer inputs, each asserting `Mongoose.Error.ValidationError`.
Plus updated JSDoc on `permissionBitSupersets` to document the
invariant that the schema now enforces.

Refs #12729.

* ♻️ refactor: hoist `MAX_PERM_BITS` + enum-shape/ceiling guards to shared util

Addresses one MINOR DRY finding and one NIT defensive-guard finding from
the latest audit. `MAX_PERM_BITS` was duplicated in `methods/aclEntry.ts`
and `schema/aclEntry.ts` with identical computations; they could silently
diverge. Move the constant plus both module-load guards to
`src/common/permissions.ts`:

* `MAX_PERM_BITS === 0` guard — fail loudly if `PermissionBits` is ever
  refactored to a `const` object or string enum (applied in both sites
  now, not just methods).
* `MAX_PERM_BITS > 255` guard — circuit-breaker for the `$in` enumeration
  strategy. At 4 bits the list tops out at 16; at 8 bits 256. Beyond
  that the approach degrades, so fail at module load rather than emit an
  unusably-large `$in` list.

Both the schema and the methods file now import from the single
`src/common/permissions.ts`. `readonly number[]` return type and
`Object.freeze` cache-value protection from the prior commits are
preserved.

Refs #12729.

* 🔒 fix: reject out-of-range `requiredBits` without caching (DoS fix)

Codex flagged a real unbounded-cache DoS vector:
`api/server/controllers/agents/v1.js` parses `req.query.requiredPermission`
via `parseInt` and forwards it unvalidated to `findAccessibleResources`,
which eventually calls `permissionBitSupersets(requiredBits)`. Because
the old code memoized *every* distinct input in a process-global Map,
an attacker could flood the cache with unique integers and grow memory
without bound.

Fix: when `requiredBits` is not an integer, is negative, or has any bits
set above `MAX_PERM_BITS`, return a single shared frozen empty array and
do NOT touch the cache. An empty `$in` list correctly matches zero rows
(rows cannot satisfy a bit we do not support), so the response is also
semantically correct — previously it "worked" only because invalid
inputs coincidentally expanded to `[]` too, but at the cost of a cache
entry each time.

Five new tests exercise the rejection path: upper-bound overflow, mixed
in-range+out-of-range bits, shared-instance identity across rejected
inputs, a 2000-unique-value cache-growth probe, and a regression check
that legitimate in-range inputs still get memoized normally.

Refs #12729.

* 🧪 test: probe `Map.prototype.set` to prove cache doesn't grow on rejection

Strengthens the DoS cache-growth test the review pass flagged: reference
identity alone allows a hypothetical regression like
`supersetCache.set(requiredBits, EMPTY_SUPERSETS); return EMPTY_SUPERSETS;`
to pass while still leaking one entry per attacker request.

Add a second test that spies on `Map.prototype.set`, records the global
call count before and after a burst of 1500 rejected inputs (500 each of
over-MAX integers, negatives, and non-integers), and asserts the delta
is zero. Manually verified the test has teeth: injecting the
hypothetical regression in the implementation produced exactly 1500
extra `Map.set` calls and the new test failed as expected; reverting
restored the clean state.

Renames the existing identity-based test to `(reference identity)` so
its scope is clear alongside the new `(Map-write probe)` companion.

Refs #12729.
2026-04-19 22:28:48 -04:00
Danny Avila
58f128bee7
🗑️ chore: Remove Deprecated Project Model and Associated Fields (#11773)
* chore: remove projects and projectIds usage

* chore: empty line linting

* chore: remove isCollaborative property across agent models and related tests

- Removed the isCollaborative property from agent models, controllers, and tests, as it is deprecated in favor of ACL permissions.
- Updated related validation schemas and data provider types to reflect this change.
- Ensured all references to isCollaborative were stripped from the codebase to maintain consistency and clarity.
2026-03-21 14:28:53 -04:00
Danny Avila
38521381f4
🐘 feat: FerretDB Compatibility (#11769)
* feat: replace unsupported MongoDB aggregation operators for FerretDB compatibility

Replace $lookup, $unwind, $sample, $replaceRoot, and $addFields aggregation
stages which are unsupported on FerretDB v2.x (postgres-documentdb backend).

- Prompt.js: Replace $lookup/$unwind/$project pipelines with find().select().lean()
  + attachProductionPrompts() batch helper. Replace $group/$replaceRoot/$sample
  in getRandomPromptGroups with distinct() + Fisher-Yates shuffle.
- Agent/Prompt migration scripts: Replace $lookup anti-join pattern with
  distinct() + $nin two-step queries for finding un-migrated resources.

All replacement patterns verified against FerretDB v2.7.0.

* fix: use $pullAll for simple array removals, fix memberIds type mismatches

Replace $pull with $pullAll for exact-value scalar array removals. Both
operators work on MongoDB and FerretDB, but $pullAll is more explicit for
exact matching (no condition expressions).

Fix critical type mismatch bugs where ObjectId values were used against
String[] memberIds arrays in Group queries:
- config/delete-user.js: use string uid instead of ObjectId user._id
- e2e/setup/cleanupUser.ts: convert userId.toString() before query

Harden PermissionService.bulkUpdateResourcePermissions abort handling to
prevent crash when abortTransaction is called after commitTransaction.

All changes verified against FerretDB v2.7.0 and MongoDB Memory Server.

* fix: harden transaction support probe for FerretDB compatibility

Commit the transaction before aborting in supportsTransactions probe, and
wrap abortTransaction in try-catch to prevent crashes when abort is called
after a successful commit (observed behavior on FerretDB).

* feat: add FerretDB compatibility test suite, retry utilities, and CI config

Add comprehensive FerretDB integration test suite covering:
- $pullAll scalar array operations
- $pull with subdocument conditions
- $lookup replacement (find + manual join)
- $sample replacement (distinct + Fisher-Yates)
- $bit and $bitsAllSet operations
- Migration anti-join pattern
- Multi-tenancy (useDb, scaling, write amplification)
- Sharding proof-of-concept
- Production operations (backup/restore, schema migration, deadlock retry)

Add production retryWithBackoff utility for deadlock recovery during
concurrent index creation on FerretDB/DocumentDB backends.

Add UserController.spec.js tests for deleteUserController (runs in CI).

Configure jest and eslint to isolate FerretDB tests from CI pipelines:
- packages/data-schemas/jest.config.mjs: ignore misc/ directory
- eslint.config.mjs: ignore packages/data-schemas/misc/

Include Docker Compose config for local FerretDB v2.7 + postgres-documentdb,
dedicated jest/tsconfig for the test files, and multi-tenancy findings doc.

* style: brace formatting in aclEntry.ts modifyPermissionBits

* refactor: reorganize retry utilities and update imports

- Moved retryWithBackoff utility to a new file `retry.ts` for better structure.
- Updated imports in `orgOperations.ferretdb.spec.ts` to reflect the new location of retry utilities.
- Removed old import statement for retryWithBackoff from index.ts to streamline exports.

* test: add $pullAll coverage for ConversationTag and PermissionService

Add integration tests for deleteConversationTag verifying $pullAll
removes tags from conversations correctly, and for
syncUserEntraGroupMemberships verifying $pullAll removes user from
non-matching Entra groups while preserving local group membership.

---------
2026-03-21 14:28:49 -04:00