Commit graph

8 commits

Author SHA1 Message Date
Danny Avila
4e0d9145e0
⚛️ feat: Atomic Admin Config Changes Endpoint (set, unset, tombstone, entry upsert/remove/rename) (#17328)
* 🗄️ refactor: Portable Admin Config Store with Revisions, Key Encoding and Coherent Writes

Admin config writes went straight to MongoDB with dotted-path `$set`, `$unset`, `$pull` and `$addToSet`, and nothing checked `configVersion`. That caused six problems.

* ⚛️ feat: Atomic Admin Config Changes Endpoint (set, unset, tombstone, entry upsert/remove/rename)

Admin clients can change a principal's config overrides only one kind of change at a time:
- PATCH `/fields` for sets, at most 100 entries;
- POST `/fields/tombstone` for one tombstone;
- DELETE `/fields` for one reset.

* 🔁 fix: Reload librechat.yaml on Admin Config Writes

Base-only sections such as `filters` and `messageFilter` live only in librechat.yaml, and an admin
config write was how a running server picked up an edit to them. The coherent-cache change limited
admin-write invalidation to the override and MCP caches, so the YAML base was never re-read after
startup: the content-filter e2e suite, which activates filters by editing the YAML and writing a
reload sentinel, saw `filters: undefined` from GET /base and every activation timed out.

Admin config writes clear the YAML base config again, alongside the override and MCP caches, and
the clear is broadcast with the others, so every instance reloads its YAML. The reload keeps the
last good config on failure and republishes the tools derived from it, so the global tool cache is
left for it to overwrite rather than emptied first.

* 🧭 fix: Cap Dotted Field Path Depth and Hold Revision Preconditions on No-Op Writes

A dotted fieldPath was split without the 32-segment cap that segments already had, so a deep path reached the recursive path writer and failed with a 500. Both spellings now share MAX_CONFIG_PATH_DEPTH and answer 400 CONFIG_PATH_INVALID / too_deep on PATCH, tombstone and DELETE /fields.

Writes reduced to nothing (protected interface permissions, base-only or dedicated sections) answered 200 without checking expectedRevision. They now compare it with the stored document and answer 409 CONFIG_VERSION_CONFLICT with currentRevision on a mismatch, as the write itself would.

* 📡 fix: Retry the Config Invalidation Subscription When Redis Comes Back

A SUBSCRIBE that failed at startup (Redis unreachable, the client's retries spent) was only logged, and ioredis resubscribes only to channels it has joined, so the instance missed every other instance's config invalidation until it restarted. The subscription is now sent again each time the subscriber becomes ready until one succeeds; after that the client's own resubscribe takes over, and closing stops the retries.

* 🧷 fix: Tighten Config Change Batches Around Array Entries, Cancelled Changes and MCP Tombstones

- A tombstone with an index segment below a field configSchema does not define (an unknown key, or what an intersection or any holds) is refused as inside_array: the merge removes inherited values from objects only, so it was stored and reported applied while the element stayed visible.
- unset joins set and rename in the merged-by-key position check: unset endpoints.custom.0.headers answers 400 indexed_merge_key_write, since the position can name another endpoint by the time the batch applies. upsertEntry and removeEntry address entries by key.
- A batch whose changes cancel out (set interface.customWelcome, then unset interface) writes nothing: no empty document, revision bump, audit entry or invalidation. Its changes are reported as unchanged.
- A whole-section mcpServers tombstone reports each librechat.yaml stdio server it leaves in place as ignored (process_mcp_server).
- removeEntry no longer attributes the whole collection to itself, and existing issues of merged-by-key entries are matched by key rather than index, so removing an entry beside a legacy-invalid one (or shifting it) is no longer refused for that entry's issue.

* 🪪 fix: Keep a Requested Priority on New Principals and Report Ignored MCP Renames

POST …/changes discarded a broad manager's priority for a principal without a document when no change altered it: the batch answered written: false instead of creating the priority-only document PUT creates. An explicit priority on an absent principal now counts as a change.

A rename under an MCP server librechat.yaml runs as a process persisted an override the runtime merge drops, but was not reported in ignored. It now is, like set and tombstone. An unset there stays unreported: it asks for the YAML value, which is what runs.

* 🔁 fix: Drop YAML Reloads and Merges Overtaken by a Config Write

Admin config writes clear the YAML base config on every instance, so a write is how a running
server picks up an edit to librechat.yaml. Two races could undo that:

- A base reload that read the old YAML before a write's clear cached it after the clear, and the
  base has no TTL, so the old base was served until the next write.
- A request that read the old base before the clear, and was still resolving principals, cached
  its merge under the new override generation for the override TTL.

The config service now keeps a base-config generation that `clearAppConfigCache` bumps. A reload
publishes the base, the tools and the last good config only if its generation is still current,
a read after a clear never joins a reload started before it, and the newer reload starts once the
stale one settles so the loader's process-wide side effects end with the newest YAML. Merges
capture the generation, base included, before they read the base config.

Tests:
- service.spec: a clear during a reload (stale success, stale failure, a read waiting on the newer
  reload) and a merge built on the pre-clear base.
- invalidation.redis.spec: a write on one instance makes another reload librechat.yaml.
- admin-config-reload.spec (api): every admin config write route, through the real router,
  handlers, database and config service, reloads librechat.yaml; a request with nothing to write
  does not.
- invalidateConfigCaches.spec no longer asserts an order the parallel clears do not have.

* 🧪 test: Reload librechat.yaml After a Committed Change Batch and Not After an Unwritten One

* 🧮 fix: Validate Change Batches on Their End State and Report Base-Only Fields a Tombstone Keeps

A batch's validation scope was built from every intermediate write, located in the state each
change produced. A change a later one undid (set `balance.startBalance`, then restore `balance`)
still widened the scope, so an issue the stored document already had there rejected an
unrelated change, and an entry a later change moved (upsert one entry, then remove one before
it) pointed its written index at a neighbour with a legacy issue. The written paths are now
located in the batch's end state and kept only where that state differs from the stored
document, with entries of merged-by-key arrays compared by key.

A tombstone above a base-only field (`endpoints.agents`, `endpoints.agents.pullRequests`) is
stored and hides the rest of the subtree, while the runtime merge puts back the provider, host,
OAuth app and shared cache slots librechat.yaml sets. The response now lists those fields in
`ignored` with reason `base_only_path`, as it already does for the stdio MCP servers a
whole-section tombstone leaves in place.

* 🧬 fix: Encode Config Keys in Objects Built in Another Realm

The key encoder treated an object as plain only when its prototype was this realm's
Object.prototype, and returned any other object unchanged, keys and all. Under Jest,
structuredClone (which the secret pipeline uses) returns objects from the host realm, so the
items of an array the pipeline had cloned, such as Azure groups or custom endpoints, reached
the engine with their dotted keys raw. Node in production has one realm, but the encoding must
not depend on prototype identity: a skipped object stores its dotted keys raw without an error.

A plain object is now one whose prototype is null or a root Object.prototype from any realm.
The encoding spec covers an object built in a vm context.

* 🧪 test: Check Dotted Record Keys Reach the Store's Whole-Document Write

dev's admin config route test asserted the keys on patchConfigFields, which the handlers no
longer call; it now asserts them on the draft writeConfigDocument receives.

* 🚩 feat: Publish the expectedRevision, prunesEmptyParents and awaitedInvalidation Config Flags

GET /api/admin/config/meta now reports the three write-coherence behaviours this branch
implements, so an admin client can drop the workarounds each one replaces:

- expectedRevision: responses carry `revision`; PUT, PATCH /fields, POST /fields/tombstone and
  PATCH /active take `expectedRevision` in the body and both DELETE routes in the query. A
  mismatch answers 409 CONFIG_VERSION_CONFLICT with currentRevision, a malformed value 400
  INVALID_EXPECTED_REVISION; unpinned writes re-read and re-apply up to 3 times.
- prunesEmptyParents: a field delete or tombstone removes the objects it empties, stopping at
  arrays; shells-only documents count as empty for assign-only lifecycle calls.
- awaitedInvalidation: writes answer after the config caches are cleared (broadcast with Redis),
  GET /base bypasses the merged-config cache, and a merge read before a clear is never cached.

meta.spec probes each one against the behaviour: a stale and a malformed precondition on a
PATCH, the store operations' unset and tombstone on nested objects and an array element, and a
held invalidation, the GET /base read options and a merge raced by a clear.

* 🔀 fix: Keep Config Keys Raw Where the Engine Stores Them, So Older Servers Read Them During Upgrades

Servers before the key encoding (v0.8.9-rc1 among them) read every override key literally.
The Mongoose binding encoded every write, so during a rolling upgrade the first write from an
upgraded server turned a stored `gpt-4.1` or Bedrock inference profile into `gpt-4%2E1` for
every server not yet upgraded, and an older server's restore of a deleted group or role
config dropped the `keyEncoding` marker its schema lacks, after which upgraded servers read
the encoded keys literally.

The binding now sends keys raw first and encodes them only when the engine refuses a raw
field name (DocumentDB, Cosmos, FerretDB or MongoDB 4.4 for `$` names): the refused write
changed nothing, so it runs again encoded under the same revision precondition, and that
process encodes later writes holding such keys directly. MongoDB 4.4+ stores dotted names
(5.0+ also `$` names inside a value), so there an older server reads exactly what it wrote. A
document whose raw keys spell escape sequences while none holds a raw `.`, `$` or NUL is
still written encoded, since a raw read would decode it. A raw write clears the marker.

Reads decode a marked document (raw keys an older server added beside encoded ones, or wrote
under a marker it left, carry no escape sequence and read as stored) and an unmarked one
unless some key holds a raw `.`, `$` or NUL, which no encoded document does; that reads an
encoded document whose marker an older server dropped. Encoding stays in the binding for
every engine that needs it (decision #15); the residual cases are in documentdb-compat.md.

* ⏱️ fix: Remove Config Cache Writes a Clear Overtook in Flight

With an asynchronous APP_CONFIG store (Redis or a custom Keyv store), a clear can bump the
generation and finish its delete after a reload or merge checked the generation but before its
`set` landed. The older value then repopulated `_BASE_` until the next write, or a merged
config for the full override TTL.

The base reload, the last-good restoration and the merge cache now check the generation again
once their write lands and delete what they wrote if a clear overtook it. A deleted entry only
costs the next read a reload or a merge; a newer base load cannot be removed this way, since it
starts only after the stale one settles.

* 🔐 fix: Require the Stored Web Search Key Again When a Change Batch Moves Its URL

#17323 refuses a PUT, PATCH, tombstone or field delete that changes or removes a web search
URL beside a stored API key unless the key is entered again, so a stored key cannot be sent
to a destination it was not entered for. POST /changes skipped that check: a batch could set,
reset or tombstone `webSearch.searxngInstanceUrl` (or replace `webSearch` without the key,
which keeps the stored key) and point the stored key at another host.

A batch is now judged on the document it would store, like PATCH: a URL that ends up beside
a stored key it did not travel with answers 400 CONFIG_SECRET_REENTRY_REQUIRED and writes
nothing, dry runs included. Only keys the request sent count as entered again; a secret a set
kept from the stored document does not.

* 🚩 feat: Publish the changesBatch Flag and the maxChangeOps Limit in Config Meta

GET /api/admin/config/meta now reports `features.changesBatch: true` and
`limits.maxChangeOps` (1000, from MAX_CONFIG_CHANGE_OPS), so the admin panel can gate its
one-request save on the endpoint and size its batches from the server. The meta.spec probe
checks the behaviour itself: an oversized batch answers TOO_MANY_OPS, a batch with one invalid
op writes nothing, and a valid set plus tombstone lands as one write.
2026-10-22 13:38:06 +02:00
Danny Avila
0b4dc238b3
🗄️ refactor: Portable Admin Config Store with Revisions, Key Encoding and Coherent Writes (#17326)
* 🗄️ refactor: Portable Admin Config Store with Revisions, Key Encoding and Coherent Writes

Admin config writes went straight to MongoDB with dotted-path `$set`, `$unset`, `$pull` and `$addToSet`, and nothing checked `configVersion`. That caused six problems.

* 🔁 fix: Reload librechat.yaml on Admin Config Writes

Base-only sections such as `filters` and `messageFilter` live only in librechat.yaml, and an admin
config write was how a running server picked up an edit to them. The coherent-cache change limited
admin-write invalidation to the override and MCP caches, so the YAML base was never re-read after
startup: the content-filter e2e suite, which activates filters by editing the YAML and writing a
reload sentinel, saw `filters: undefined` from GET /base and every activation timed out.

Admin config writes clear the YAML base config again, alongside the override and MCP caches, and
the clear is broadcast with the others, so every instance reloads its YAML. The reload keeps the
last good config on failure and republishes the tools derived from it, so the global tool cache is
left for it to overwrite rather than emptied first.

* 🧭 fix: Cap Dotted Field Path Depth and Hold Revision Preconditions on No-Op Writes

A dotted fieldPath was split without the 32-segment cap that segments already had, so a deep path reached the recursive path writer and failed with a 500. Both spellings now share MAX_CONFIG_PATH_DEPTH and answer 400 CONFIG_PATH_INVALID / too_deep on PATCH, tombstone and DELETE /fields.

Writes reduced to nothing (protected interface permissions, base-only or dedicated sections) answered 200 without checking expectedRevision. They now compare it with the stored document and answer 409 CONFIG_VERSION_CONFLICT with currentRevision on a mismatch, as the write itself would.

* 📡 fix: Retry the Config Invalidation Subscription When Redis Comes Back

A SUBSCRIBE that failed at startup (Redis unreachable, the client's retries spent) was only logged, and ioredis resubscribes only to channels it has joined, so the instance missed every other instance's config invalidation until it restarted. The subscription is now sent again each time the subscriber becomes ready until one succeeds; after that the client's own resubscribe takes over, and closing stops the retries.

* 🔁 fix: Drop YAML Reloads and Merges Overtaken by a Config Write

Admin config writes clear the YAML base config on every instance, so a write is how a running
server picks up an edit to librechat.yaml. Two races could undo that:

- A base reload that read the old YAML before a write's clear cached it after the clear, and the
  base has no TTL, so the old base was served until the next write.
- A request that read the old base before the clear, and was still resolving principals, cached
  its merge under the new override generation for the override TTL.

The config service now keeps a base-config generation that `clearAppConfigCache` bumps. A reload
publishes the base, the tools and the last good config only if its generation is still current,
a read after a clear never joins a reload started before it, and the newer reload starts once the
stale one settles so the loader's process-wide side effects end with the newest YAML. Merges
capture the generation, base included, before they read the base config.

Tests:
- service.spec: a clear during a reload (stale success, stale failure, a read waiting on the newer
  reload) and a merge built on the pre-clear base.
- invalidation.redis.spec: a write on one instance makes another reload librechat.yaml.
- admin-config-reload.spec (api): every admin config write route, through the real router,
  handlers, database and config service, reloads librechat.yaml; a request with nothing to write
  does not.
- invalidateConfigCaches.spec no longer asserts an order the parallel clears do not have.

* 🧬 fix: Encode Config Keys in Objects Built in Another Realm

The key encoder treated an object as plain only when its prototype was this realm's
Object.prototype, and returned any other object unchanged, keys and all. Under Jest,
structuredClone (which the secret pipeline uses) returns objects from the host realm, so the
items of an array the pipeline had cloned, such as Azure groups or custom endpoints, reached
the engine with their dotted keys raw. Node in production has one realm, but the encoding must
not depend on prototype identity: a skipped object stores its dotted keys raw without an error.

A plain object is now one whose prototype is null or a root Object.prototype from any realm.
The encoding spec covers an object built in a vm context.

* 🧪 test: Check Dotted Record Keys Reach the Store's Whole-Document Write

dev's admin config route test asserted the keys on patchConfigFields, which the handlers no
longer call; it now asserts them on the draft writeConfigDocument receives.

* 🚩 feat: Publish the expectedRevision, prunesEmptyParents and awaitedInvalidation Config Flags

GET /api/admin/config/meta now reports the three write-coherence behaviours this branch
implements, so an admin client can drop the workarounds each one replaces:

- expectedRevision: responses carry `revision`; PUT, PATCH /fields, POST /fields/tombstone and
  PATCH /active take `expectedRevision` in the body and both DELETE routes in the query. A
  mismatch answers 409 CONFIG_VERSION_CONFLICT with currentRevision, a malformed value 400
  INVALID_EXPECTED_REVISION; unpinned writes re-read and re-apply up to 3 times.
- prunesEmptyParents: a field delete or tombstone removes the objects it empties, stopping at
  arrays; shells-only documents count as empty for assign-only lifecycle calls.
- awaitedInvalidation: writes answer after the config caches are cleared (broadcast with Redis),
  GET /base bypasses the merged-config cache, and a merge read before a clear is never cached.

meta.spec probes each one against the behaviour: a stale and a malformed precondition on a
PATCH, the store operations' unset and tombstone on nested objects and an array element, and a
held invalidation, the GET /base read options and a merge raced by a clear.

* 🔀 fix: Keep Config Keys Raw Where the Engine Stores Them, So Older Servers Read Them During Upgrades

Servers before the key encoding (v0.8.9-rc1 among them) read every override key literally.
The Mongoose binding encoded every write, so during a rolling upgrade the first write from an
upgraded server turned a stored `gpt-4.1` or Bedrock inference profile into `gpt-4%2E1` for
every server not yet upgraded, and an older server's restore of a deleted group or role
config dropped the `keyEncoding` marker its schema lacks, after which upgraded servers read
the encoded keys literally.

The binding now sends keys raw first and encodes them only when the engine refuses a raw
field name (DocumentDB, Cosmos, FerretDB or MongoDB 4.4 for `$` names): the refused write
changed nothing, so it runs again encoded under the same revision precondition, and that
process encodes later writes holding such keys directly. MongoDB 4.4+ stores dotted names
(5.0+ also `$` names inside a value), so there an older server reads exactly what it wrote. A
document whose raw keys spell escape sequences while none holds a raw `.`, `$` or NUL is
still written encoded, since a raw read would decode it. A raw write clears the marker.

Reads decode a marked document (raw keys an older server added beside encoded ones, or wrote
under a marker it left, carry no escape sequence and read as stored) and an unmarked one
unless some key holds a raw `.`, `$` or NUL, which no encoded document does; that reads an
encoded document whose marker an older server dropped. Encoding stays in the binding for
every engine that needs it (decision #15); the residual cases are in documentdb-compat.md.

* ⏱️ fix: Remove Config Cache Writes a Clear Overtook in Flight

With an asynchronous APP_CONFIG store (Redis or a custom Keyv store), a clear can bump the
generation and finish its delete after a reload or merge checked the generation but before its
`set` landed. The older value then repopulated `_BASE_` until the next write, or a merged
config for the full override TTL.

The base reload, the last-good restoration and the merge cache now check the generation again
once their write lands and delete what they wrote if a clear overtook it. A deleted entry only
costs the next read a reload or a merge; a newer base load cannot be removed this way, since it
starts only after the stale one settles.
2026-10-22 13:04:35 +02:00
Danny Avila
2591a15c10
🔑 feat: Keep Dotted Record Keys in Admin Config Writes and Publish the Config Contract (#17322)
* 🔑 feat: Keep Dotted Record Keys in Admin Config Writes and Publish the Config Contract

Keep dotted record keys in admin config write bodies behind a path-scoped
sanitizer, reject $-prefixed and prototype keys after auth, report dotted_key
issues with segments, and publish GET /api/admin/config/meta (features, rules,
limits, versions) so clients can gate on backend capabilities.

* 🧪 test: Expect Issue Segments in the Config Override e2e Scenario

Override validation issues now carry `segments`; assert the full issue shape.
2026-10-22 09:59:08 +02:00
Danny Avila
84851d8655
🍃 fix: Serialize Index Builds and the Agent Edge Cleanup for DocumentDB (#15670)
* 🍃 fix: Serialize Index Builds and the Agent Edge Cleanup for DocumentDB

Amazon DocumentDB admits one index build per collection at a time and rejects a second with code 40333. MongoDBSaver.setup() starts the compound and TTL builds of each checkpoint collection concurrently and returns the rejections, which buildMongoSaver logged and ignored, so the TTL index was the likely loser on every boot and checkpoints never expired. setupCheckpointIndexes re-runs the idempotent setup() while any rejection is a 40333. auditLog and the two index migrations bypassed createIndexesWithRetry; buildIndexWithRetry is the raw-collection form and they use it.

removeAgentIdsFromEdges pruned deleted agent ids with an aggregation-pipeline update, which DocumentDB rejects; the guard missed it because the pipeline reached updateMany as a function return value. It is now a read, a prune in code, and a compare-and-set bulk write, keeping the per-graph atomicity the pipeline had.

The static guard now carries the full 5.0 unsupported-operator matrix, rejects $set/$unset at stage position and the $count accumulator, follows calls to functions that return an array, and holds every index build to the retrying helpers.

* 🍃 fix: Keep Companion Setup Errors Past the Deadline and Trace Appended Stages

Codex round 1. setupCheckpointIndexes now carries the non-conflict errors of the last pass and returns them beside the conflict when the peer-build deadline passes, so a permanent failure on one collection is still logged when the other stays busy. The alias-stage detector also inspects the arguments of push, unshift and concat and both branches of a conditional element, since a pipeline assembled by push was invisible to it. Both tests fail without their fix.

* 🍃 fix: Page the Edge Cleanup, Count the Build Deadline From the First Conflict, and Widen the Guard

Codex round 2 and Copilot. buildWhenCollectionFree now starts its deadline at the first conflict rather than the first attempt, so a saver that admits one long build before reporting a companion 40333 still gets its rerun instead of caching with an index missing. removeAgentIdsFromEdges reads matching graphs a page at a time (sort by _id, limit) and uses the filter as its cursor, bounding application memory the way the server-side update did; the loop is bounded by passes without progress rather than a fixed attempt count. The guard follows array-returning class and object methods, resolves stages held in variables, and exempts the Meilisearch createIndex by its primaryKey option instead of by file. retryWithBackoff rejects NaN options instead of skipping every attempt. Every new test fails without its change.

* 🍃 fix: Walk the Edge Cleanup Behind an _id Cursor and Propagate Setup Rejections

Codex round 3. removeAgentIdsFromEdges pages behind an _id cursor so every pass resumes where the last one stopped instead of rescanning the cleaned prefix, with one sweep from the top once the cursor is exhausted to pick up compare-and-set misses; a test lands a concurrent edit on a first-page graph and checks the miss is cleaned. setupCheckpointIndexes rethrows a rejection of setup() itself so buildMongoSaver keeps its in-process fallback, converting only a build conflict that outlasts the deadline. The guard resolves a returned identifier against array-bound variables, so a pipeline built in a local and returned is still a pipeline update.

* 🍃 fix: Bound Cleanup Sweeps and the Audit Index Wait, Reuse the Removed-Id Set

Codex round 4. removeAgentIdsFromEdges gives up after a bounded number of sweeps from the top when references keep being added, since progress otherwise resets the stall bound forever (reachable in deleteUserAgents, which deletes the agents after cleanup); the removed-id set is built once per cleanup rather than per graph. The audit append index build on the request path now has a deadline, so a peer holding the collection makes the append fail open instead of holding the request; createAuditLogMethods takes optional index-build overrides so a test can shorten it. The guard treats a variable annotated with an array type as array-valued whatever its initializer, and states that dataflow is followed within a file only.
2026-09-06 15:02:43 -04:00
Danny Avila
c13990a398
🍃 test: Restore Sweep Coverage of Typed-Criteria Methods (#15609)
Some checks failed
Docker Dev Images Build / publish (push) Waiting to run
Sync Helm Chart Tags / Ignore non-main push (push) Waiting to run
Sync Helm Chart Tags / Sync chart tags (push) Waiting to run
Publish `@librechat/client` to NPM / pack (push) Has been cancelled
Publish `@librechat/client` to NPM / publish-npm (push) Has been cancelled
* 🍃 test: Restore Sweep Coverage of Typed-Criteria Methods

The typed-criteria refactor (#15580) validates query criteria key by key and
fails closed on anything unrecognized — correctly, since a silently accepted
bad criterion would widen a filter into an unscoped `find` or `deleteMany`.
The sweep synthesizes arguments from parameter names, so a parameter named
`query` received a string, which `buildFilter` read as a criteria object and
rejected at index '0'.

Six methods therefore went from adjudicated to un-driven with the suite still
green: `getActions`, `getAssistants`, `deleteAction(s)`, and
`deleteAssistant(s)`. Criteria-shaped parameters are now synthesized as an
empty object, which builds an empty filter and still reaches the engine.

Recovered 13 methods in total — the six regressions plus seven that had been
silently un-driven for the same reason (prompts, groups, conversation tags),
including through the runs recorded as authoritative. Baseline coverage rises
from 372 to 385 of 518 methods. The compatibility assessment now flags this
count as the thing to watch across releases: a coverage regression hides
behind a passing suite, and only a matrix diff surfaces it.

* 🍃 test: Preserve searchMessages' String Query in the Sweep

Criteria-shaped synthesis is name-based, so it also caught
`searchMessages(query: string, ...)`, whose parameter is a genuine search
string. On a Meili-enabled deployment that object is forwarded verbatim into
`meiliSearch`, so the recorded outcome would reflect a harness-generated
request rather than the method's behavior.

`searchMessages` gets an `ARG_OVERRIDES` entry supplying a real query string;
its failure is now the honest 'MeiliSearch plugin not registered' rather than
an invalid request. It is the only method in `src/methods` whose
`query`/`criteria`/`filter` parameter is string-typed, which the synthesis
comment records along with the symptom a future one would show.

An error-driven variant (repair only on 'Unknown query criterion') was tried
first and rejected: it protects string parameters automatically but loses
seven methods whose criteria parameter makes them early-return on a string
rather than throw a recognizable error, taking coverage from 385 back to 378.
Coverage holds at 385 of 518 driven, zero engine rejections.
2026-09-04 12:58:41 -04:00
Danny Avila
81338cef95
🍃 test: Sweep Every DB Schema Method Against a Live Engine (#15392)
* 🍃 test: Sweep Every data-schemas Method Against a Live Engine

Adds `misc/documentdb/sweep.documentdb.spec.ts`: drives all 510 exported
data-schemas methods against a real engine and records, per method, how many
driver queries it issued and whether the engine rejected any of them.

Every DocumentDB incompatibility found so far — pipeline updates, `$$REMOVE`,
`$facet`, the transaction-probe false negative, the Mongoose-compiled mixed
projection — was invisible until someone thought to look for its class. The
sweep inverts that: the engine adjudicates whatever each method emits, known
class or not. Query shapes are parsed before they are matched, so an empty
database adjudicates most entry paths.

Mechanics: arguments are synthesized from each function's parameter names,
then repaired adaptively from the validation errors they provoke (ObjectId and
number casts, required schema paths, object-parameter shapes) until the method
reaches the database; the driver prototype is instrumented to count calls and
capture server errors — including on cursor consumption, where parse errors
actually surface — and duplicate-key violations are classified as benign. A
method that never issues a query is reported un-adjudicated instead of
silently green: coverage is visible, not assumed.

Two modes with identical mechanics so matrices diff cleanly: in-memory MongoDB
(`SWEEP_BASELINE=true`, free, CI-able) and live DocumentDB (`DOCUMENTDB_URI`).
`SWEEP_REPORT_PATH` writes JSON for machine diffing;
`DOCUMENTDB_STRICT=true` turns rejections into failures.

First full run against live DocumentDB 5.0.0: 363 of 510 methods issued at
least one query, zero engine rejections, zero divergences from the MongoDB
baseline. The jest config gains the main config's transform-ignore patterns so
the full method bundle (which pulls ESM markdown tooling) loads.

* 🍃 fix: Adjudicate Index DDL, Real ACL Cascades, and a Run-Scoped Sweep Database

Review round 1, all four findings real:

- The driver instrumentation now wraps `createIndex`, `createIndexes`,
  `dropIndex`, `dropIndexes`, and `listIndexes` (cursor-wrapped), so an engine
  rejecting index DDL — which `recordAuditEntry` issues before any listed
  operation and then swallows — is adjudicated instead of surfacing as a
  false `no-queries` under STRICT.
- The `removeAllPermissions` no-op override is gone: `createMethods` supplies
  a real implementation delegating to `deleteAclEntries`, so driven deletion
  cascades exercise their ACL operations (not-driven drops 147 → 142).
- Each race timer is cleared when its attempt settles, instead of hundreds of
  referenced timers living until the shared deadline and the last one holding
  the process open after the report.
- The live run connects to a run-scoped database
  (`librechat_sweep_<runId>`), so its `dropDatabase` teardown can never erase
  fixtures out from under the sibling live suites sharing the URI, nor any
  retained state in the URI's own database.

* 🍃 fix: Attribute Straggler Queries by Async Context and Adjudicate Transaction Commits

Review round 2, all three findings real:

- Per-method attribution now rides AsyncLocalStorage instead of a module
  global: a method that outlives its timeout keeps recording queries and
  engine errors under ITS OWN row rather than corrupting whichever row the
  sweep advanced to, and a final reclassification pass upgrades any row whose
  engine error arrived after its outcome was assigned.
- The session prototype's `commitTransaction`/`abortTransaction`/
  `withTransaction` are instrumented: transaction-lifecycle rejections surface
  from the session, not a collection operation, and several methods convert
  them into domain errors — previously a real commit rejection classified as
  a mere `threw` and STRICT passed.
- The compatibility doc's published numbers now come from the
  corrected-harness baseline (368 driven / 142 un-driven) and state explicitly
  that the recorded live-run matrix predates the instrumentation widening, so
  the next live run is authoritative for the newly adjudicated surfaces.

* 🍃 fix: Settle Stragglers Before STRICT and Commit a Real Transaction on a Replica-Set Baseline

Review round 3, all four findings real:

- STRICT is now enforced against the FINALIZED rows: `afterAll` waits
  (bounded) for every timed-out invocation to settle, reclassifies rows whose
  engine errors arrived late, and asserts zero rejections — previously a
  straggler's rejection could arrive after its per-test assertion had already
  passed against the `timeout` outcome.
- The baseline runs on a single-node replica set with the live branch's exact
  connection options (`autoIndex`/`autoCreate` off, run-scoped `dbName`):
  a standalone baseline would reject driven transactions for topology reasons,
  and asymmetric model auto-creation would attribute DDL in one mode and not
  the other — both manufacturing false matrix divergences.
- `resolveMCPAuthorityProof` gets a seeded case (fixture ported from the
  compat suite, executed in tenant context via the new `within` wrapper on
  overrides): generic synthesis died on target validation before a session
  ever started, so the session instrumentation adjudicated nothing. It now
  drives 22 operations through the production path, transaction commit
  included, on both engines.

* 🍃 docs: Sync the Published Baseline Counts With the Seeded Harness

* 🍃 fix: Materialize Transaction-Read Collections and Settle Stragglers Before Teardown

Review round 4, three findings real (the fourth — stale doc totals — was
already fixed on the pushed head):

- Straggler settlement moved BEFORE engine teardown: dropping the database or
  closing the client would abort outstanding work with closed-client errors
  that are not engine verdicts, leaving a timed-out row unadjudicated while
  STRICT passes.
- The authority fixtures (sweep AND compat) materialize PluginAuth and Token
  before the transaction: `loadCurrentProof` reads both inside it, and
  DocumentDB rejects any in-transaction statement touching a non-existent
  collection — the rejection surfaces as `proof_unavailable` before the commit
  is reached. This is the mechanism behind the compat authority test failing
  on every live cluster run while passing against MongoDB; with the
  collections materialized, the next live run adjudicates the transaction
  itself instead of the missing namespace.
- Engine errors are recognized by `instanceof MongoServerError` instead of a
  `name` equality, so server-originated subclasses (MongoWriteConcernError,
  MongoBulkWriteError) can no longer slip through the session wrapper as
  domain throws.

* 🍃 fix: Materialize Group and Config for the Authority Transaction and Exclude Constructor Exports

Review round 5, both findings real. The proof transaction also reads Group and
Config; the sweep fixture materializes both (the compat fixture already
creates a Config record and now materializes Group). SessionError — an error
class exported alongside the session methods — is excluded from the method
map, correcting the totals to 509 methods / 369 driven / 140 un-driven; the
published baseline is synced in the same commit.

* 🍃 fix: Build a Fresh Method Bundle per Case and Emit a Normalized Matrix

Review round 6, both findings real:

- Every row now invokes a freshly constructed method bundle. Factory-local
  caches leaked across alphabetically ordered rows — the session methods cache
  `sessionIndexesPromise`, so `ensureSessionIndexes` returned that settled
  promise and was misreported as `no-queries`. It now issues its 3 index
  operations, making the matrix order-independent (370 driven / 139 un-driven).

- The report separates run metadata from a canonical `rows` payload with
  run-specific values normalized out (run id, generated ObjectIds and UUIDs),
  so the two engines' matrices diff cleanly instead of differing on every row
  that embeds a synthesized value in its error text.

* 🍃 fix: Materialize Authority Snapshot Namespaces Before Its Transaction

Amazon DocumentDB rejects any statement inside a transaction that touches a
collection which does not exist, and `asMCPError` converts that server
rejection into `proof_unavailable` — so on a deployment where no PluginAuth
or Token row has ever been written, every MCP authority proof fails with an
error naming nothing about the real cause. This is the mechanism behind the
live-harness authority probe failing on every real cluster run since July
while passing against MongoDB.

`loadAuthoritativeSnapshot` now materializes the nine namespaces its snapshot
reads before opening the transaction, once per process. MongoDB tolerates the
in-transaction read either way, so nothing changes there; a create failure is
logged at debug and left to surface from the transaction itself.

The regression test drops the credential namespaces to reproduce the state and
asserts the resolve recreates them and succeeds; it fails without the fix.

* 🍃 docs: Record the Authoritative Hardened Live Run

The hardened harness ran against a live DocumentDB 5.0.0 cluster: 370 of 509
methods drove at least one query, zero engine rejections, and the matrix is
byte-identical to the MongoDB baseline — every outcome and query count
matches. This run adjudicated index DDL, transaction commits and aborts, ACL
cascades, and the authority snapshot transaction, none of which the first
pre-hardening run could see.

The same run turned the compatibility suite's MCP authority probe green for
the first time on any real cluster, confirming the namespace-materialization
diagnosis for the `proof_unavailable` failures seen since July.

* 🍃 fix: Retry Snapshot Namespace Creation After a Transient Failure

The catch suppressed every `createCollection` error, so a transient failure
settled the once-per-process memo as successful: creation was skipped for the
rest of the process, the transaction kept failing on the missing collection,
and every later proof returned `proof_unavailable` even after the condition
cleared. The existing reset-on-reject path could never fire because the inner
promise never rejected.

Only `NamespaceExists` (including a concurrent creator winning the race) is
suppressed now; every other failure propagates, clears the memo, and lets the
next request retry. The regression test rejects the first creation with a
transient error, asserts the proof fails, then asserts the following request
recreates the namespace and succeeds — it fails against the swallow-all form.

* 🍃 fix: Throttle Snapshot Namespace Preflight Retries With a Bounded Cooldown

Clearing the memo on every rejection made a persistent failure — an
authorization error, say — turn each authority request into another serial
collection preflight: a DDL retry storm precisely while the database is
unhealthy, since the single-flight promise only coalesces requests while an
attempt is pending.

A failure is now recorded with its timestamp and replayed without touching the
database until a bounded cooldown elapses (30s, overridable through the
existing hooks for tests). Transient failures stay retryable, just not on
every request. Both sides of the boundary are covered: one test asserts a
persistent failure issues no further creations inside the cooldown, and the
existing retry test runs with the cooldown disabled to assert recovery still
works. The throttle test fails with the cooldown gate removed.

* 🍃 fix: Instrument Db-Level Namespace DDL in the Sweep

The authority path's namespace preflight calls `Model.createCollection()`,
which delegates to `Db.createCollection` — not to any `Collection` operation
the sweep wrapped. A DocumentDB rejection of that DDL would therefore be
laundered by `asMCPError` into `proof_unavailable`, leaving the row
classified as a mere `threw` while `DOCUMENTDB_STRICT=true` passed: the exact
false-green class this suite exists to close, reintroduced by the preflight
itself.

`Db.createCollection`/`dropCollection`/`renameCollection` are now
instrumented, and `NamespaceExists` (48) joins the benign server codes since
an idempotent preflight against an existing collection is the expected
outcome, not an incompatibility. The authority row's adjudicated query count
rises from 22 to 31, which is those nine preflight operations becoming
visible.
2026-08-31 09:54:32 -04:00
Danny Avila
9684b2478f
🍃 fix: Restore Amazon DocumentDB Compatibility Across Trigger and Subagent Writes (#15375)
* 🍃 fix: Restore Amazon DocumentDB Compatibility Across Trigger and Subagent Writes

The README advertises Amazon DocumentDB 5.0+ support, but six sites added
between 2026-07-29 and 2026-08-30 reintroduced constructs the engine rejects,
and the transaction probe silently reported a false negative. All seven are
adjudicated against a live DocumentDB 5.0.0 cluster, not against AWS's
omission-based documentation.

Rejected by every engine version, now rewritten with supported operators:

- `claimNextAgentTriggerDelivery` used a pipeline-form update. It reads the
  globally oldest candidate under the same sort, then claims that exact `_id`
  and lifecycle state with a compare-and-swap. Ownership is still granted only
  by the fenced write. An empty queue costs the one read it costs today; a
  successful claim costs one additional round trip.
- `renewAgentTriggerDeliveryProducerLease` used a `$cond` to keep the later
  deadline. `$max` is the exact classic-operator form and evaluates no
  expression.
- `claimBackgroundToolResults` and `releaseBackgroundToolResultClaims` rewrote
  the entire `content` array through `$map`. The filtered positional operator
  addresses the matching parts directly, so each write touches only those parts
  and still needs no read-modify-write.
- The bounded subagent thread-view projections used `$$REMOVE` to omit keys.
  They now emit `null` and are pruned back to absent inside the pass that
  already materializes each record, preserving the absent-vs-present contract.
- The selected-task read used `$facet`. It is now two concurrent reads, which
  also removes the 16 MiB single-document risk the surrounding code already
  avoids for the same reason.

`supportsTransactions` probed a canary collection that does not exist, and
DocumentDB rejects a transaction touching a non-existent collection. The probe
returned `false` on an engine that fully supports transactions, so every caller
silently took the non-transactional path. It now materializes the canary first,
once per process.

Adds `documentdb.spec.ts`, a static guard over the whole class, because no
runtime test can catch it: every suite runs `mongodb-memory-server`, which is
real MongoDB and accepts all of these constructs. Adds
`misc/documentdb/audit.documentdb.spec.ts` to re-adjudicate against a real
cluster, and records the live verdicts and connection requirements in the
compatibility assessment.

* 🍃 fix: Address Review — CAS Exhaustion, AST Guard, Single-Snapshot Selected Read

Addresses all three review findings, and the strengthened guard immediately
found a seventh incompatible site the original character-level scan missed.

- Exhausting the claim's compare-and-swap budget now throws instead of
  returning null. Every lost round read a claimable row, so the queue is not
  empty; null is the engine's confirmed-empty signal and advances its idle
  backoff, which would leave queued work waiting. A throw lands on the engine's
  retryable claim-failure path, which holds the polling cadence. Covered by a
  new test asserting the bounded budget and that the loser owns nothing.

- The static guard now walks the TypeScript AST: it catches pipelines bound to
  a variable first, cast with `as`, chosen by a conditional, or carried in a
  `bulkWrite` operation's `update` property (including shorthand). Every shape
  is proven against fixtures in the suite itself, so the detectors can never
  silently go inert. The stronger scan flagged `updateToolCallResult`, whose
  pipeline lived in a `stages` variable: the backgrounded tool-call settle path
  was still broken on DocumentDB. Rewritten with the filtered positional
  operator — subfield sets preserve an existing `resultClaim` without per-part
  branching, an explicit `$unset` keeps the old whole-object replacement's
  wakeup-disarm semantics, and the attachments merge becomes `$pull` plus
  `$push` (two atomic writes that converge under the retry the claim fencing
  already provides). New tests lock the disarm and supplied-claim semantics,
  and the live audit suite now drives this method too.

- The selected-task read is a single aggregation again: each selected row
  yields both its bounded message fields and its source payload from the same
  document version, restoring the `$facet` snapshot guarantee the concurrent
  split lost, and returning the endpoint to three reads.

* 🍃 fix: Close Settle Atomicity Windows and Spread Claim Contention

Self-review pass (eight independent review angles over the branch diff).

Correctness:

- The attachments merge is a single fenced write again: read the array, merge
  with the exact semantics the old `$filter`/`$concatArrays` had, and write it
  back with the settle patch in one update guarded on the array being unchanged
  — the same compare-and-swap this file already uses for control receipts. The
  `$pull`-then-`$push` split could strip attachments on a failure between the
  writes (the throw path retires the delivery, so nothing healed it) and
  duplicated them under concurrent re-application.
- A supplied `resultClaim` is stamped BEFORE the settle write instead of after
  it. The claim-last order left a terminal, claimable, unclaimed part visible
  to the ~1s wakeup poll, which could acquire and deliver a result a manual
  poll had already delivered; claim-first leaves only inert intermediate states
  (a claim on a part that is not yet terminal admits no consumer). The stamp's
  guard is `resultClaim: null`, restoring the old `$ifNull` treatment of a
  stored null.
- `releaseBackgroundToolResultClaims` filters on `content: { $type: 'array' }`:
  the filtered positional operator errors on a row without the array where the
  old pipeline no-op'd, and that throw surfaced as retryable release failures
  that left claims stamped forever.
- The claim loop reads a BATCH of candidates per round, so the engine's
  parallel claimers spread across the batch instead of convoying on one
  head-of-queue row; exhaustion now throws the typed
  `AgentTriggerClaimContentionError`. The dead-capability adoption fence also
  pins `availableAt`, which the write copies.
- The transaction probe probes first and creates the canary only after a
  failure — plain MongoDB never pays the DDL or gains the collection — logs a
  failed create instead of swallowing it, and concurrent first probes share one
  in-flight promise.

Structure and guard:

- One `partScope(prefix)` builder yields the `$elemMatch` document filter and
  the arrayFilters element filter from a single predicate, deleting the
  aggregation `$expr` twin, and the agent-ownership rule shared by settling and
  claiming lives in one `agentOwnershipFilter` helper.
- The static guard scans all three backend workspaces (`packages/data-schemas`,
  `packages/api`, `api`), parses each file once, and shares the tree between
  both detectors.
- The audit suite asserts each probe's returned verdict, the compat suite's run
  recipe carries the connection parameters the audit header documents, and the
  `$facet` pipeline assertion covers every captured aggregation in one loop.

* 🍃 fix: Keep Attachment Contention Retryable and the Selected Snapshot Authoritative

Second codex round, both findings real:

- Exhausting the attachment-merge fence now returns an unmatched result with a
  warn log instead of throwing. The settle retry loop treats unmatched as
  retry-then-heal-later and degrades to false; a throw escaped it onto
  ambiguous-failure handling that can retire the completion, losing a healthy
  completed tool result to mere contention. Locked by a test asserting the
  bounded rounds resolve to `{ matched: false }` rather than a rejection.

- When the selected task's rows also appear in the normal page window, the
  dedup pass kept the page copy — read by an independent concurrent aggregate —
  and discarded the single-snapshot version, letting a page row claim a
  transcript that `selectedSources` does not carry. Duplicates are now replaced
  in place with the snapshot version (page order kept), so the selected
  message/source pair stays per-document coherent end to end.

* 🍃 fix: Treat a Stored-Null Claim Stamp as Unclaimed in the Claim Fence

Third codex round. The subfield-preserving settle write keeps a persisted
`resultClaim: null` that the old whole-object rewrite dropped, and the claim
path's outer `$elemMatch` fence still required the field to be absent via
`$exists: false`. The in-memory claimable scan, the claim arrayFilters, and the
settle stamp all already treat a null stamp as unclaimed, so a part carrying
one became terminal but permanently unclaimable: every manual poll and wakeup
read `ready` in memory, missed the fence, and returned `not_ready` forever.

The fence now matches missing-or-null like every other layer. The regression
test persists an actual null stamp — asserted present before claiming, so the
test cannot pass vacuously — and claims it.
2026-08-30 18:33:25 -04:00
Danny Avila
23d1ad473d
🍃 fix: Amazon DocumentDB Compatibility for Pipeline-Form Updates (#14495)
* 🍃 fix: Amazon DocumentDB Compatibility for Pipeline-Form Updates

- Rewrite acceptTerms without aggregation-pipeline update + $$NOW (null-guarded first-acceptance claim preserves the original timestamp under concurrent and repeat requests)
- Rewrite decrementTagCounts clamp-at-zero decrement as ordered two-op bulkWrite (clamp before guarded $inc)
- Rewrite extendFilesTTL TTL hold as projected read + per-doc guarded $set via tenantSafeBulkWrite, preserving only-widens/ceiling/cleared-stays-permanent semantics
- Log background index-build failures via Model 'index' listeners (previously swallowed silently, e.g. partialFilterExpression rejection on DocumentDB <5.0)
- Add misc/documentdb live-compatibility harness + assessment (AWS-cited)
- Fix latent file.spec helper bug: createdAt backdating was silently stripped by mongoose immutability

Closes #14488

* 🍃 fix: Harden DocumentDB-Safe Updates Against Cross-Call Races

Addresses Codex review on #14495:
- decrementTagCounts: normalize-null / $inc / clamp-negative op triple so
  interleaved decrements of the same tag converge on max(0, ...) exactly as
  the serialized pipeline did (clamp keys on count < 0, not count < amount)
- acceptTerms: guard the repeat-acceptance fallback with a non-null timestamp
  (exact complement of the claim guard) and retry the claim when a
  config/reset-terms.js reset races between the two updates, so acceptance
  never resurrects a reset cycle without a fresh audit timestamp

* 👷 ci: Suppress Ignored-File Warnings in Changed-File ESLint Run

Changed files under config-ignored paths (packages/data-schemas/misc/**)
emit "File ignored" warnings that fail --max-warnings=0.
2026-07-28 22:14:34 -04:00