LibreChat/packages/data-schemas/misc/documentdb/documentdb-compat.md
Danny Avila 4e0d9145e0
⚛️ feat: Atomic Admin Config Changes Endpoint (set, unset, tombstone, entry upsert/remove/rename) (#17328)
* 🗄️ refactor: Portable Admin Config Store with Revisions, Key Encoding and Coherent Writes

Admin config writes went straight to MongoDB with dotted-path `$set`, `$unset`, `$pull` and `$addToSet`, and nothing checked `configVersion`. That caused six problems.

* ⚛️ feat: Atomic Admin Config Changes Endpoint (set, unset, tombstone, entry upsert/remove/rename)

Admin clients can change a principal's config overrides only one kind of change at a time:
- PATCH `/fields` for sets, at most 100 entries;
- POST `/fields/tombstone` for one tombstone;
- DELETE `/fields` for one reset.

* 🔁 fix: Reload librechat.yaml on Admin Config Writes

Base-only sections such as `filters` and `messageFilter` live only in librechat.yaml, and an admin
config write was how a running server picked up an edit to them. The coherent-cache change limited
admin-write invalidation to the override and MCP caches, so the YAML base was never re-read after
startup: the content-filter e2e suite, which activates filters by editing the YAML and writing a
reload sentinel, saw `filters: undefined` from GET /base and every activation timed out.

Admin config writes clear the YAML base config again, alongside the override and MCP caches, and
the clear is broadcast with the others, so every instance reloads its YAML. The reload keeps the
last good config on failure and republishes the tools derived from it, so the global tool cache is
left for it to overwrite rather than emptied first.

* 🧭 fix: Cap Dotted Field Path Depth and Hold Revision Preconditions on No-Op Writes

A dotted fieldPath was split without the 32-segment cap that segments already had, so a deep path reached the recursive path writer and failed with a 500. Both spellings now share MAX_CONFIG_PATH_DEPTH and answer 400 CONFIG_PATH_INVALID / too_deep on PATCH, tombstone and DELETE /fields.

Writes reduced to nothing (protected interface permissions, base-only or dedicated sections) answered 200 without checking expectedRevision. They now compare it with the stored document and answer 409 CONFIG_VERSION_CONFLICT with currentRevision on a mismatch, as the write itself would.

* 📡 fix: Retry the Config Invalidation Subscription When Redis Comes Back

A SUBSCRIBE that failed at startup (Redis unreachable, the client's retries spent) was only logged, and ioredis resubscribes only to channels it has joined, so the instance missed every other instance's config invalidation until it restarted. The subscription is now sent again each time the subscriber becomes ready until one succeeds; after that the client's own resubscribe takes over, and closing stops the retries.

* 🧷 fix: Tighten Config Change Batches Around Array Entries, Cancelled Changes and MCP Tombstones

- A tombstone with an index segment below a field configSchema does not define (an unknown key, or what an intersection or any holds) is refused as inside_array: the merge removes inherited values from objects only, so it was stored and reported applied while the element stayed visible.
- unset joins set and rename in the merged-by-key position check: unset endpoints.custom.0.headers answers 400 indexed_merge_key_write, since the position can name another endpoint by the time the batch applies. upsertEntry and removeEntry address entries by key.
- A batch whose changes cancel out (set interface.customWelcome, then unset interface) writes nothing: no empty document, revision bump, audit entry or invalidation. Its changes are reported as unchanged.
- A whole-section mcpServers tombstone reports each librechat.yaml stdio server it leaves in place as ignored (process_mcp_server).
- removeEntry no longer attributes the whole collection to itself, and existing issues of merged-by-key entries are matched by key rather than index, so removing an entry beside a legacy-invalid one (or shifting it) is no longer refused for that entry's issue.

* 🪪 fix: Keep a Requested Priority on New Principals and Report Ignored MCP Renames

POST …/changes discarded a broad manager's priority for a principal without a document when no change altered it: the batch answered written: false instead of creating the priority-only document PUT creates. An explicit priority on an absent principal now counts as a change.

A rename under an MCP server librechat.yaml runs as a process persisted an override the runtime merge drops, but was not reported in ignored. It now is, like set and tombstone. An unset there stays unreported: it asks for the YAML value, which is what runs.

* 🔁 fix: Drop YAML Reloads and Merges Overtaken by a Config Write

Admin config writes clear the YAML base config on every instance, so a write is how a running
server picks up an edit to librechat.yaml. Two races could undo that:

- A base reload that read the old YAML before a write's clear cached it after the clear, and the
  base has no TTL, so the old base was served until the next write.
- A request that read the old base before the clear, and was still resolving principals, cached
  its merge under the new override generation for the override TTL.

The config service now keeps a base-config generation that `clearAppConfigCache` bumps. A reload
publishes the base, the tools and the last good config only if its generation is still current,
a read after a clear never joins a reload started before it, and the newer reload starts once the
stale one settles so the loader's process-wide side effects end with the newest YAML. Merges
capture the generation, base included, before they read the base config.

Tests:
- service.spec: a clear during a reload (stale success, stale failure, a read waiting on the newer
  reload) and a merge built on the pre-clear base.
- invalidation.redis.spec: a write on one instance makes another reload librechat.yaml.
- admin-config-reload.spec (api): every admin config write route, through the real router,
  handlers, database and config service, reloads librechat.yaml; a request with nothing to write
  does not.
- invalidateConfigCaches.spec no longer asserts an order the parallel clears do not have.

* 🧪 test: Reload librechat.yaml After a Committed Change Batch and Not After an Unwritten One

* 🧮 fix: Validate Change Batches on Their End State and Report Base-Only Fields a Tombstone Keeps

A batch's validation scope was built from every intermediate write, located in the state each
change produced. A change a later one undid (set `balance.startBalance`, then restore `balance`)
still widened the scope, so an issue the stored document already had there rejected an
unrelated change, and an entry a later change moved (upsert one entry, then remove one before
it) pointed its written index at a neighbour with a legacy issue. The written paths are now
located in the batch's end state and kept only where that state differs from the stored
document, with entries of merged-by-key arrays compared by key.

A tombstone above a base-only field (`endpoints.agents`, `endpoints.agents.pullRequests`) is
stored and hides the rest of the subtree, while the runtime merge puts back the provider, host,
OAuth app and shared cache slots librechat.yaml sets. The response now lists those fields in
`ignored` with reason `base_only_path`, as it already does for the stdio MCP servers a
whole-section tombstone leaves in place.

* 🧬 fix: Encode Config Keys in Objects Built in Another Realm

The key encoder treated an object as plain only when its prototype was this realm's
Object.prototype, and returned any other object unchanged, keys and all. Under Jest,
structuredClone (which the secret pipeline uses) returns objects from the host realm, so the
items of an array the pipeline had cloned, such as Azure groups or custom endpoints, reached
the engine with their dotted keys raw. Node in production has one realm, but the encoding must
not depend on prototype identity: a skipped object stores its dotted keys raw without an error.

A plain object is now one whose prototype is null or a root Object.prototype from any realm.
The encoding spec covers an object built in a vm context.

* 🧪 test: Check Dotted Record Keys Reach the Store's Whole-Document Write

dev's admin config route test asserted the keys on patchConfigFields, which the handlers no
longer call; it now asserts them on the draft writeConfigDocument receives.

* 🚩 feat: Publish the expectedRevision, prunesEmptyParents and awaitedInvalidation Config Flags

GET /api/admin/config/meta now reports the three write-coherence behaviours this branch
implements, so an admin client can drop the workarounds each one replaces:

- expectedRevision: responses carry `revision`; PUT, PATCH /fields, POST /fields/tombstone and
  PATCH /active take `expectedRevision` in the body and both DELETE routes in the query. A
  mismatch answers 409 CONFIG_VERSION_CONFLICT with currentRevision, a malformed value 400
  INVALID_EXPECTED_REVISION; unpinned writes re-read and re-apply up to 3 times.
- prunesEmptyParents: a field delete or tombstone removes the objects it empties, stopping at
  arrays; shells-only documents count as empty for assign-only lifecycle calls.
- awaitedInvalidation: writes answer after the config caches are cleared (broadcast with Redis),
  GET /base bypasses the merged-config cache, and a merge read before a clear is never cached.

meta.spec probes each one against the behaviour: a stale and a malformed precondition on a
PATCH, the store operations' unset and tombstone on nested objects and an array element, and a
held invalidation, the GET /base read options and a merge raced by a clear.

* 🔀 fix: Keep Config Keys Raw Where the Engine Stores Them, So Older Servers Read Them During Upgrades

Servers before the key encoding (v0.8.9-rc1 among them) read every override key literally.
The Mongoose binding encoded every write, so during a rolling upgrade the first write from an
upgraded server turned a stored `gpt-4.1` or Bedrock inference profile into `gpt-4%2E1` for
every server not yet upgraded, and an older server's restore of a deleted group or role
config dropped the `keyEncoding` marker its schema lacks, after which upgraded servers read
the encoded keys literally.

The binding now sends keys raw first and encodes them only when the engine refuses a raw
field name (DocumentDB, Cosmos, FerretDB or MongoDB 4.4 for `$` names): the refused write
changed nothing, so it runs again encoded under the same revision precondition, and that
process encodes later writes holding such keys directly. MongoDB 4.4+ stores dotted names
(5.0+ also `$` names inside a value), so there an older server reads exactly what it wrote. A
document whose raw keys spell escape sequences while none holds a raw `.`, `$` or NUL is
still written encoded, since a raw read would decode it. A raw write clears the marker.

Reads decode a marked document (raw keys an older server added beside encoded ones, or wrote
under a marker it left, carry no escape sequence and read as stored) and an unmarked one
unless some key holds a raw `.`, `$` or NUL, which no encoded document does; that reads an
encoded document whose marker an older server dropped. Encoding stays in the binding for
every engine that needs it (decision #15); the residual cases are in documentdb-compat.md.

* ⏱️ fix: Remove Config Cache Writes a Clear Overtook in Flight

With an asynchronous APP_CONFIG store (Redis or a custom Keyv store), a clear can bump the
generation and finish its delete after a reload or merge checked the generation but before its
`set` landed. The older value then repopulated `_BASE_` until the next write, or a merged
config for the full override TTL.

The base reload, the last-good restoration and the merge cache now check the generation again
once their write lands and delete what they wrote if a clear overtook it. A deleted entry only
costs the next read a reload or a merge; a newer base load cannot be removed this way, since it
starts only after the stale one settles.

* 🔐 fix: Require the Stored Web Search Key Again When a Change Batch Moves Its URL

#17323 refuses a PUT, PATCH, tombstone or field delete that changes or removes a web search
URL beside a stored API key unless the key is entered again, so a stored key cannot be sent
to a destination it was not entered for. POST /changes skipped that check: a batch could set,
reset or tombstone `webSearch.searxngInstanceUrl` (or replace `webSearch` without the key,
which keeps the stored key) and point the stored key at another host.

A batch is now judged on the document it would store, like PATCH: a URL that ends up beside
a stored key it did not travel with answers 400 CONFIG_SECRET_REENTRY_REQUIRED and writes
nothing, dry runs included. Only keys the request sent count as entered again; a secret a set
kept from the stored document does not.

* 🚩 feat: Publish the changesBatch Flag and the maxChangeOps Limit in Config Meta

GET /api/admin/config/meta now reports `features.changesBatch: true` and
`limits.maxChangeOps` (1000, from MAX_CONFIG_CHANGE_OPS), so the admin panel can gate its
one-request save on the endpoint and size its batches from the server. The meta.spec probe
checks the behaviour itself: an oversized batch answers TOO_MANY_OPS, a batch with one invalid
op writes nothing, and a valid set plus tombstone lands as one write.
2026-10-22 13:38:06 +02:00

436 lines
26 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

# Amazon DocumentDB Compatibility Assessment (issue #14488)
> **Update 2026-08-30 — adjudicated live.** Everything below was originally
> decided from AWS's documentation, which omits unsupported operators rather
> than listing them. It has now been run against a real DocumentDB **5.0.0**
> cluster (the version this project supports, and the one the reference
> deployment runs). `audit.documentdb.spec.ts` records the verdicts; the
> corrections are in "Live verdicts" at the end of this document. Two claims
> below were wrong: pipeline-form updates had returned in new code, and the
> transaction probe reported a false negative.
Adjudicated against official AWS documentation on 2026-07-28. Engine columns
throughout: DocumentDB **3.6 / 4.0 / 5.0 / 8.0 instance-based** and **elastic
clusters**. AWS's supported-APIs page states that unsupported operators are
_omitted_ from its tables, so several verdicts below are implicit-by-omission
and flagged as such.
## Executive summary
- The login blocker was real: three aggregation-pipeline updates (one using
`$$NOW`) existed in the codebase. DocumentDB documents no support for
pipeline-form updates on any engine version. All three are now rewritten
with plain update operators that work on every engine, including elastic.
- Partial-index creation fails on DocumentDB < 5.0 and elastic — and the
failure was **provably silent** (no log, no crash, uniqueness quietly
unenforced). Model creation now attaches an `index` listener so failed
builds log loudly.
- Transactions already degrade gracefully via a runtime probe. GridFS is
unreachable dead code. `retryWrites=false` is a deployment-docs item.
- Recommendation: **DocumentDB 5.0+ instance-based is a supportable target**;
4.0 runs but with degraded uniqueness enforcement (now logged); **elastic
clusters should be documented as unsupported** (no unique indexes at all).
## Proven incompatibilities — fixed in this PR
### 1. `acceptTerms` pipeline update + `$$NOW` (P0 — blocked login)
`packages/data-schemas/src/methods/user.ts:303` used a pipeline-form
`findByIdAndUpdate` with `$ifNull`/`$$NOW` (introduced by PR #10810, matching
the reporter's regression window). When Terms gating is on, every login hits
this and DocumentDB rejects it.
AWS evidence: the [supported APIs page](https://docs.aws.amazon.com/documentdb/latest/developerguide/mongo-apis.html)
lists only classic update operators (no pipeline form anywhere); the `$set`/
`$unset` _stage_ operators are marked unsupported for 3.6/4.0/5.0; `$$NOW` is
absent from the System variables table entirely (`$$CURRENT` and `$$REMOVE`
are explicitly "No"). Implicit-by-omission, but consistent with the reported
`Failed to parse update: field must be of BSON type object` class of error
(AWS documents no exact error string).
Fix: null-guarded first-acceptance claim (`termsAcceptedAt: null` matches both
the schema's explicit `null` default and missing legacy fields — a
`$exists: false` guard would never fire because of that default), with a
plain-`$set` fallback for repeat acceptance. First-acceptance timestamp
preservation, concurrency convergence, and the `IUser | null` contract are
covered by tests, including a raw-inserted legacy document without the field.
### 2. `decrementTagCounts` pipeline update (P1 — silent tag-count drift)
`packages/data-schemas/src/methods/conversationTag.ts:47` used a
`$max`/`$subtract`/`$ifNull` pipeline inside `bulkWrite`, wrapped in a
try/catch that only logs — on DocumentDB, conversation deletion succeeded
while tag counts silently drifted.
Fix: two mutually exclusive plain ops per tag in one ordered `bulkWrite` —
clamp-to-zero (`count` below the decrement amount, or null/missing) first,
then a guarded `$inc`. Clamp-at-zero, missing-count tolerance, and
variable-amount semantics all preserved; now covered by a new test block
(previously untested).
### 3. `extendFilesTTL` pipeline update (P1 — `/files/usage` TTL holds fail)
`packages/data-schemas/src/methods/file.ts:607` — **not in the reporter's
list; found by sweeping the codebase** (`rg` for pipeline-shaped update args
and `$$NOW`; these three sites were the only hits).
Fix: read the candidate files (one projected query), compute each file's
`min(now + renewMs, createdAt + maxLifetimeMs)` ceiling client-side, then
issue per-document guarded `$set`s (`expiresAt: { $exists: true, $lt: next }`)
through `tenantSafeBulkWrite`. Only-widens, per-file ceiling, and
cleared-TTL-stays-permanent semantics are preserved under concurrency by the
write guard. Cost: one extra read round trip on this path — unavoidable
without a schema change, because the ceiling is per-document.
## Proven, made loud — partial indexes (P1 on < 5.0 / elastic)
Four unique partial indexes exist:
- `packages/data-schemas/src/schema/user.ts:192` and `:199` — OAuth ids
(`googleId`, `openidId`, …) with `partialFilterExpression: { $exists: true }`
- `packages/data-schemas/src/schema/file.ts:173` — `execute_code` files
(`$eq`-shaped filter)
- `packages/data-schemas/src/schema/group.ts:56` — group source ids
(`$exists: true`)
AWS: [partial-index.html](https://docs.aws.amazon.com/documentdb/latest/developerguide/partial-index.html)
— "The partial index feature is supported in Amazon DocumentDB 5.0
instance-based clusters"; the index-properties table marks Partial as
No/No/Yes/Yes/No across 3.6/4.0/5.0/8.0/elastic. The `$exists` and `$eq`
filter shapes used here are inside DocumentDB 5.0's supported operator list
(`$eq, $exists, $and, $gt/$gte/$lt/$lte`), so on 5.0+ these indexes build.
On 3.6/4.0/elastic the builds fail — and empirically (probe: unique index over
pre-seeded duplicates, mongoose 8, autoIndex) the failure is **completely
silent**: no unhandled rejection, no log, the index simply doesn't exist and
duplicate inserts succeed. Mongoose only surfaces build errors through a
`Model.on('index')` listener, which nothing attached. `createModels` now
attaches one that logs every failed build (`packages/data-schemas/src/models/index.ts`).
Operational consequence on < 5.0 remains: OAuth-account uniqueness is not
DB-enforced — documented, loud, but not fixable in application code.
## Proven compatible — no action needed
- **Transactions**: supported on 4.0+ instance-based ("Amazon DocumentDB …
supports transactions in Amazon DocumentDB 4.0 and later" —
[transactions.html](https://docs.aws.amazon.com/documentdb/latest/developerguide/transactions.html));
unsupported on 3.6 and elastic. LibreChat already probes at runtime
(`packages/data-schemas/src/utils/transactions.ts`, cached in
`api/server/services/PermissionService.js`) and falls back to
non-transactional writes — the same mode as standalone MongoDB without a
replica set. DocumentDB's restrictions (1-minute execution limit, no cursors
in transactions, no retryable commit/abort) don't intersect LibreChat's
usage.
- **GridFS**: `packages/api/src/cache/keyvMongo.ts` only constructs a
`GridFSBucket` when `useGridFS` is set — no caller ever sets it and the
class isn't exported (the singleton uses a plain `logs` collection). Dead
code. Moot regardless: AWS lists GridFS as supported on instance-based
clusters (elastic: no).
- **TTL indexes**: supported everywhere including elastic. AWS warns deletion
is best-effort ("Documents are not guaranteed to be deleted within any
specific period") — acceptable, since LibreChat treats TTL as cleanup, not
as a security boundary.
- **`$ifNull`**: supported on all versions (only its pipeline-update context
was the problem).
## Deployment requirements (documentation, not code)
- **`retryWrites=false` is mandatory in `MONGO_URI`.** AWS: "Amazon DocumentDB
does not currently support retryable writes"; the failure mode is
`{"ok":0,"errmsg":"Unrecognized field: 'txnNumber'","code":9}`
([functional-differences.html](https://docs.aws.amazon.com/documentdb/latest/developerguide/functional-differences.html)).
`api/db/connect.js` passes `MONGO_URI` through verbatim, so this belongs in
the deployment docs (and the live harness flags a URI missing it).
- TLS with the AWS CA bundle; clusters are VPC-only (tunnel/bastion for
external access).
## Document as unsupported — elastic clusters
[Elastic cluster limitations](https://docs.aws.amazon.com/documentdb/latest/developerguide/docdb-using-elastic-clusters.html):
no **unique indexes** (any), no partial indexes, no ACID transactions, no
GridFS, no change streams, `$expr` unsupported, and the cursor-methods table
even lists `sort()`/`skip()`/`limit()` as "No". The `email + tenantId` unique
index alone disqualifies elastic clusters. Recommend stating this explicitly
in the docs.
## Undetermined — honest gaps
- **The reporter's engine version and cluster type** — still unknown; it
decides whether the partial-index caveat applies to them (5.0+: it doesn't).
Worth asking directly on the issue.
- **DocumentDB 8.0 pipeline-update acceptance** — 8.0 added `$set`/`$unset`
aggregation _stages_, but AWS never documents pipeline-form updates; the
harness probe answers this live.
- **`collMod` is only "Partial"** on every version — `Model.syncIndexes()`
may issue `collMod` beyond the documented `expireAfterSeconds`, so the static
guard rejects it outright.
- **Aggregation operators — now enforced, not observed.** 29 `aggregate`
calls across 11 files (`insights.ts`, `message.ts`, `mcpAuthority.ts`,
`agent.ts`, `aclEntry.ts`, `triggerDelivery.ts`, `queuedTurn.ts`,
`prompt.ts`, `conversation.ts`, `agentCategory.ts`,
`packages/api/src/code/lifecycle.ts`). An earlier inventory here named three
files because it was built from `.aggregate(`, which never matches the
generically typed `.aggregate<T>(` form the newer code uses. Every operator
in use (`$strLenBytes`, `$let`, `$map`, `$reduce`, `$regexMatch`, `$convert`,
`$anyElementTrue`, …) is in AWS's 5.0 supported list, and
`methods/documentdb.spec.ts` now rejects the full unsupported matrix,
`$set`/`$unset` at stage position, and the `$count` accumulator. Several
operators are 4.0+/5.0-only (`$expr`, `$switch`, `$convert`, `$regexMatch`,
array-form `$first`), which reinforces the 5.0+ recommendation.
- **No faithful local emulator exists.** The `documentdb-local` Docker image
is the PostgreSQL-based Linux Foundation project — AWS's own OSS blog
confirms "a different engine than the one used in Amazon DocumentDB." Live
regression testing must run against a real cluster.
## Proven, fixed — edge cleanup pipeline update (every version)
`removeAgentIdsFromEdges` (`methods/agent.ts`, since #14428) pruned deleted
agent ids out of every graph's `edges` with an aggregation-pipeline update —
the class fixed everywhere else in #15375. The static guard missed it because
the pipeline reached `updateMany` as a function's return value rather than an
array literal or an array-bound variable; the guard now follows calls to
functions declared or annotated to return an array, and independently rejects
`$set`/`$unset` at stage position. The rewrite reads each matching graph,
prunes in code, and writes the result back behind a compare-and-set on the
edges it read (`tenantSafeBulkWrite`, one round trip for every graph), so a
concurrent edit is never overwritten — the atomicity the pipeline provided, at
the cost of one extra read on the agent-delete path.
## Proven, fixed — concurrent index builds (every version)
DocumentDB admits one index build per collection at a time and rejects a
second with code 40333 where MongoDB would serialize it. `utils/retry.ts`
exists for exactly this and the boot builds go through it — but three callers
bypassed it, and one sits on the agent hot path:
- **`MongoDBSaver.setup()`** (`@langchain/langgraph-checkpoint-mongodb`) starts
the compound and TTL builds of each checkpoint collection in a single
`Promise.allSettled` and returns the rejections instead of throwing;
`buildMongoSaver` logged them and proceeded. On DocumentDB one build per
collection lost that race on every boot — the TTL index is pushed second,
making it the likely loser, so checkpoints accumulated without bound.
`setupCheckpointIndexes` now re-runs the idempotent `setup()` while any
rejection is a 40333; each pass admits one more build until all exist.
- `methods/auditLog.ts` called `model.createIndexes()` raw before the chain's
first append; `migrations/mcpAuthorityIndexes.ts` and
`migrations/mcpServerNames.ts` called `collection.createIndex()` raw. All
three now go through the helpers (`buildIndexWithRetry` is the
raw-collection form of `createIndexesWithRetry`).
`methods/documentdb.spec.ts` rejects any `createIndex` / `createIndexes` /
`syncIndexes` / `ensureIndexes` outside `utils/retry.ts` unless it is the
argument of `buildIndexWithRetry`, so the class cannot recur silently. The
method sweep cannot see this class at all — it drives exported methods, and
these builds happen inside a third-party `setup()` and in migrations — which
is how it survived three audits.
## Regression strategy
1. **In-repo (CI today)**: the behavioral tests added in
`user.methods.spec.ts`, `conversationTag.methods.spec.ts`, and
`file.spec.ts` lock the pipeline-free implementations' semantics
(first-acceptance preservation, clamp-at-zero, per-file ceiling).
2. **Live harness (this directory)**: `compat.documentdb.spec.ts` exercises
the exact operations behind #14488 against a real cluster and prints a
capability matrix (pipeline updates, `$$NOW`, transactions, partial unique
indexes, TTL, `retryWrites`). Gated on `DOCUMENTDB_URI`; verified green
against real MongoDB as a baseline. Suggested cadence: before releases and
whenever update-operator code in `data-schemas` changes; optionally a
scheduled GitHub Action on a runner with VPC access to a dev cluster.
## Support matrix and recommendation
| Capability (LibreChat dependency) | 3.6 | 4.0 | 5.0 | 8.0 | Elastic |
| ---------------------------------------- | --- | --- | --- | --- | ------- |
| Pipeline updates (**no longer used**) | ✗ | ✗ | ✗ | ? | ✗ |
| Plain update operators (all writes now) | ✓ | ✓ | ✓ | ✓ | ✓ |
| Unique indexes | ✓ | ✓ | ✓ | ✓ | ✗ |
| Partial unique indexes (OAuth ids) | ✗ | ✗ | ✓ | ✓ | ✗ |
| Transactions (runtime-probed) | ✗ | ✓ | ✓ | ✓ | ✗ |
| TTL indexes | ✓ | ✓ | ✓ | ✓ | ✓ |
| Concurrent index builds (now serialized) | ✗ | ✗ | ✗ | ✗ | ✗ |
**Recommendation**: support **DocumentDB 5.0+ instance-based** with
`retryWrites=false` documented as required. 4.0 functions with
partial-unique-index loss (now logged loudly at startup) — "works, with a
documented caveat". Elastic clusters: unsupported, full stop.
## Live verdicts (DocumentDB 5.0.0, 2026-08-30)
Probed by `audit.documentdb.spec.ts`, which drives the production methods
themselves rather than re-implementations.
| Construct | Verdict | Server error |
| ---------------------------------------------------- | -------- | ----------------------------------------------------------- |
| Aggregation-pipeline update | rejected | `Failed to parse update: field must be of BSON type object` |
| `$$REMOVE` | rejected | `Feature not supported: $$REMOVE` |
| `$facet` | rejected | `Aggregation stage not supported: '$facet'` |
| `$max`, `$set`/`$unset` | accepted | — |
| Filtered positional `$[<id>]` | accepted | — |
| `$regexMatch`, `$switch`, `$let`, `$convert` | accepted | — |
| `$strLenBytes`, `$substrCP`, `$mergeObjects`, `$map` | accepted | — |
| Partial unique indexes, TTL indexes | accepted | — |
### Correction 1 — pipeline updates returned after this document was written
Six sites reintroduced unsupported constructs between 2026-07-29 and
2026-08-30, all in code added with the durable trigger and background-task
work. Nothing caught them: every unit suite runs `mongodb-memory-server`, which
is real MongoDB and accepts all of it. `src/methods/documentdb.spec.ts` is now
a static guard against the whole class.
### Correction 2 — the transaction probe reported a false negative
`supportsTransactions` read a canary collection that does not exist, and
DocumentDB rejects a transaction touching a non-existent collection
(`Feature not supported: non-existent collection in transaction`). The probe
therefore returned `false` on an engine that fully supports transactions, and
every caller silently took the non-transactional path. Verified directly: with
the collection materialized, both read-only and multi-write transactions
commit. The probe now creates the canary first.
### Connection requirements for the live suites
Established against the real cluster; all four are load-bearing through a
tunnel and none were documented before:
- `authSource=admin` — the user lives in `admin`; a database in the URI path
otherwise becomes the auth source and authentication fails
- `authMechanism=SCRAM-SHA-1` — DocumentDB rejects SCRAM-SHA-256
(`Unsupported mechanism [ -301 ]`)
- `directConnection=true` — replica-set discovery returns internal cluster
hostnames that are unreachable through a tunnel
- `tlsAllowInvalidHostnames` — the tunnel endpoint never matches the certificate
## Method sweep (2026-08-31)
`sweep.documentdb.spec.ts` drives every exported data-schemas method (509 at
the time of writing; constructor-valued exports are excluded) against a real engine, auto-synthesizing arguments,
repairing them from validation errors, and counting the driver queries each
method actually issues — a method that issues none is reported un-adjudicated
instead of silently green. Run once against in-memory MongoDB
(`SWEEP_BASELINE=true`) and once against DocumentDB, then diff the JSON
matrices (`SWEEP_REPORT_PATH`).
Corrected-harness baseline (replica-set MongoDB, real ACL cascades, index DDL
and transaction-lifecycle instrumentation, per-invocation async attribution,
seeded authority-transaction case, constructor exports excluded, a fresh
method bundle per case, criteria-shaped parameters synthesized as objects):
**385 of 518 methods issue at least one query; zero rejections on MongoDB**.
The remaining 133 issue none (validation
rejected the synthesized arguments, or a guard short-circuited); they are
listed in the matrix and shrink by adding `ARG_OVERRIDES` entries. Watch this count across releases: the typed
criteria refactor (#15580) silently took six adjudicated methods to
un-driven because the sweep synthesized a string for a `query` parameter and
`buildFilter` fails closed on it — a coverage regression that a green suite
hides. Diffing matrices, not reading the pass line, is what catches it. The
report's `rows` payload is normalized for
cross-engine diffing (`diff <(jq .rows a.json) <(jq .rows b.json)`); run
metadata lives outside it.
**Authoritative live run (2026-08-31, hardened harness, DocumentDB 5.0.0):
370 of 509 methods drove at least one query; zero engine rejections; the
matrix is byte-identical to the MongoDB baseline** — every row's outcome and
query count matches, so there is no engine divergence anywhere the sweep can
reach. This run adjudicated index DDL, transaction commits and aborts, ACL
cascades, and the authority snapshot's own transaction, none of which the
first (pre-hardening) run could see.
The same run turned the compatibility suite's `MCP authority snapshot` probe
green for the first time on any real cluster. It had failed since July with
`proof_unavailable`: `loadAuthoritativeSnapshot` reads nine collections inside
its transaction, DocumentDB rejects in-transaction statements against
namespaces that do not exist, and `asMCPError` converted that server rejection
into a reason naming nothing about the cause. The method now materializes
those namespaces before opening the transaction.
This sweep exists because every incompatibility found so far was invisible
until someone thought to look for its class; here the engine adjudicates
whatever each method emits, known class or not.
## Admin config overrides — key encoding (2026-10-21)
Admin config override documents (`configs.overrides`) are free-form JSON whose
record keys legitimately contain `.` (Bedrock inference profiles such as
`us.anthropic.claude-sonnet-4-5`, Azure model names such as `gpt-4.1`, MIME
patterns, header names) and can start with `$`. MongoDB 5+ stores such field
names but cannot address them in an update; DocumentDB, Cosmos DB (MongoDB
API) and FerretDB restrict them further. Whether DocumentDB 5.0 and Cosmos
accept dotted names inside a Mixed subdocument was never adjudicated live.
The question no longer needs an answer. Config persistence goes through the
portable store contract (`src/app/store.ts`, conformance suite
`src/app/conformance.ts`), and the Mongoose binding (`src/methods/config.ts`)
writes every document whole with plain `$set`. Its keys go raw when the engine
stores them, and reversibly percent-escaped (`%`, `.`, `$` and NUL in every
object key, `src/methods/encoding.ts`) when the engine refuses a raw name. An
engine that cannot store a dotted or `$`-prefixed field name therefore never
holds one, so DocumentDB and Cosmos are covered whichever way their field-name
rules turn out. Values are never rewritten.
- Write form: the binding sends the keys raw first. When the engine refuses a
field name (`isFieldNameRejection`), the refused write changed nothing, and
it is sent again encoded under the same revision precondition; that process
then encodes every later write that holds such a key. A document is also
written encoded when its keys spell escape sequences (`%2E`, `%24`, `%25`,
`%00`) while none holds a raw `.`, `$` or NUL, since a raw read would decode
them. An encoded document carries `keyEncoding: 1`; a raw write clears it.
- Reads decode a marked document, and an unmarked one unless some key holds a
raw `.`, `$` or NUL (no encoded document does). The decoder rewrites only the
four sequences above, so raw keys an older server adds to an encoded
document, or writes under a marker it left in place, read as stored.
- Rolling upgrades and rollback: servers before the encoding read every key
literally. On MongoDB 4.4+ (dotted names) and 5.0+ (`$` names inside a
value) the binding keeps writing raw keys, so an older server reads exactly
what it wrote. Only on an engine that refuses raw names does an older server
meet encoded keys, and those are entries it could not store itself (it
answered 422 `CONFIG_KEY_NOT_STORABLE`): it reads them as
`us%2Eanthropic%2Eclaude-sonnet-4-5` until it is upgraded. An older server's
group or role deletion that restores a config document drops the
`keyEncoding` field its schema lacks; the read rule above still decodes that
document. The one key a read can misread is a raw key that literally spells
one of the four escape sequences, in a document an older server wrote with no
dotted or `$` key.
- No dotted-path `$set`/`$unset` or `$pull`/`$addToSet` touches `overrides` or
`tombstones`; every mutation is a pure transform followed by one
compare-and-swap on `{_id, configVersion}`, which every engine supports.
The batch route (`POST /api/admin/config/:type/:id/changes`, `src/app/changes.ts`)
applies any mix of set, unset, tombstone and entry changes (upsert, remove, rename)
to the decoded document and stores the result with that same single write, so it
needs nothing from the engine beyond the contract.
- Pre-release check on a real cluster: run the conformance runner
(`misc/ferretdb/configStore.ferretdb.spec.ts`, any MongoDB-compatible URI in
`FERRETDB_URI`) against DocumentDB 5.0 and Cosmos. It passes 24/24 against
mongod 8.2 and mongodb-memory-server; FerretDB and the managed engines were
not available when this was written.
- The runner is destructive: it deletes every document in the target
database's `configs` collection before each case and drops the collection
at teardown. Give it a throwaway database on the cluster (for example
`.../config_store_test`), never a database a deployment uses. It refuses to
start when `configs` already holds documents.
- `compat.documentdb.spec.ts` › "admin config overrides with dotted record keys"
writes through `patchConfigFields` and `upsertConfig`, which store keys raw
or, on an engine that refuses them, encoded, so on DocumentDB and Cosmos it
checks whichever round trip the engine takes:
`npx jest --config misc/documentdb/jest.documentdb.config.mjs -t 'dotted record keys'`.
- The 422 `CONFIG_KEY_NOT_STORABLE` answer (`isFieldNameRejection`) remains
for a key name the engine refuses in both forms, such as an empty key.
### Before the encoding: raw dotted keys (2026-10-20)
Before the store encoded keys, dotted record keys reached the engine verbatim
through a dotted `$set` path and a whole-object `$set`. Documents written then
carry no `keyEncoding` marker and still hold raw dotted names, which the
binding reads verbatim and keeps raw on engines that store them. Local runs of
the compat suite against a single-node replica set of each server (16 of 16
passing on every version):
| Engine | Dotted keys, `$set` path | Dotted keys, whole `$set` | `$` key inside a value |
| --------------------------- | ------------------------ | ------------------------- | ---------------------- |
| MongoDB 4.4.29 (local) | supported | supported | rejected |
| MongoDB 5.0.31 (local) | supported | supported | stored |
| MongoDB 6.0.26 (local) | supported | supported | stored |
| MongoDB 7.0.14 (local) | supported | supported | stored |
| MongoDB 8.2.6 (local) | supported | supported | stored |
| Amazon DocumentDB 5.0 | **not yet run** | **not yet run** | **not yet run** |
| Azure Cosmos DB for MongoDB | **not yet run** | **not yet run** | **not yet run** |