LibreChat/packages/data-schemas/misc/documentdb/documentdb-compat.md
Danny Avila 81338cef95
🍃 test: Sweep Every DB Schema Method Against a Live Engine (#15392)
* 🍃 test: Sweep Every data-schemas Method Against a Live Engine

Adds `misc/documentdb/sweep.documentdb.spec.ts`: drives all 510 exported
data-schemas methods against a real engine and records, per method, how many
driver queries it issued and whether the engine rejected any of them.

Every DocumentDB incompatibility found so far — pipeline updates, `$$REMOVE`,
`$facet`, the transaction-probe false negative, the Mongoose-compiled mixed
projection — was invisible until someone thought to look for its class. The
sweep inverts that: the engine adjudicates whatever each method emits, known
class or not. Query shapes are parsed before they are matched, so an empty
database adjudicates most entry paths.

Mechanics: arguments are synthesized from each function's parameter names,
then repaired adaptively from the validation errors they provoke (ObjectId and
number casts, required schema paths, object-parameter shapes) until the method
reaches the database; the driver prototype is instrumented to count calls and
capture server errors — including on cursor consumption, where parse errors
actually surface — and duplicate-key violations are classified as benign. A
method that never issues a query is reported un-adjudicated instead of
silently green: coverage is visible, not assumed.

Two modes with identical mechanics so matrices diff cleanly: in-memory MongoDB
(`SWEEP_BASELINE=true`, free, CI-able) and live DocumentDB (`DOCUMENTDB_URI`).
`SWEEP_REPORT_PATH` writes JSON for machine diffing;
`DOCUMENTDB_STRICT=true` turns rejections into failures.

First full run against live DocumentDB 5.0.0: 363 of 510 methods issued at
least one query, zero engine rejections, zero divergences from the MongoDB
baseline. The jest config gains the main config's transform-ignore patterns so
the full method bundle (which pulls ESM markdown tooling) loads.

* 🍃 fix: Adjudicate Index DDL, Real ACL Cascades, and a Run-Scoped Sweep Database

Review round 1, all four findings real:

- The driver instrumentation now wraps `createIndex`, `createIndexes`,
  `dropIndex`, `dropIndexes`, and `listIndexes` (cursor-wrapped), so an engine
  rejecting index DDL — which `recordAuditEntry` issues before any listed
  operation and then swallows — is adjudicated instead of surfacing as a
  false `no-queries` under STRICT.
- The `removeAllPermissions` no-op override is gone: `createMethods` supplies
  a real implementation delegating to `deleteAclEntries`, so driven deletion
  cascades exercise their ACL operations (not-driven drops 147 → 142).
- Each race timer is cleared when its attempt settles, instead of hundreds of
  referenced timers living until the shared deadline and the last one holding
  the process open after the report.
- The live run connects to a run-scoped database
  (`librechat_sweep_<runId>`), so its `dropDatabase` teardown can never erase
  fixtures out from under the sibling live suites sharing the URI, nor any
  retained state in the URI's own database.

* 🍃 fix: Attribute Straggler Queries by Async Context and Adjudicate Transaction Commits

Review round 2, all three findings real:

- Per-method attribution now rides AsyncLocalStorage instead of a module
  global: a method that outlives its timeout keeps recording queries and
  engine errors under ITS OWN row rather than corrupting whichever row the
  sweep advanced to, and a final reclassification pass upgrades any row whose
  engine error arrived after its outcome was assigned.
- The session prototype's `commitTransaction`/`abortTransaction`/
  `withTransaction` are instrumented: transaction-lifecycle rejections surface
  from the session, not a collection operation, and several methods convert
  them into domain errors — previously a real commit rejection classified as
  a mere `threw` and STRICT passed.
- The compatibility doc's published numbers now come from the
  corrected-harness baseline (368 driven / 142 un-driven) and state explicitly
  that the recorded live-run matrix predates the instrumentation widening, so
  the next live run is authoritative for the newly adjudicated surfaces.

* 🍃 fix: Settle Stragglers Before STRICT and Commit a Real Transaction on a Replica-Set Baseline

Review round 3, all four findings real:

- STRICT is now enforced against the FINALIZED rows: `afterAll` waits
  (bounded) for every timed-out invocation to settle, reclassifies rows whose
  engine errors arrived late, and asserts zero rejections — previously a
  straggler's rejection could arrive after its per-test assertion had already
  passed against the `timeout` outcome.
- The baseline runs on a single-node replica set with the live branch's exact
  connection options (`autoIndex`/`autoCreate` off, run-scoped `dbName`):
  a standalone baseline would reject driven transactions for topology reasons,
  and asymmetric model auto-creation would attribute DDL in one mode and not
  the other — both manufacturing false matrix divergences.
- `resolveMCPAuthorityProof` gets a seeded case (fixture ported from the
  compat suite, executed in tenant context via the new `within` wrapper on
  overrides): generic synthesis died on target validation before a session
  ever started, so the session instrumentation adjudicated nothing. It now
  drives 22 operations through the production path, transaction commit
  included, on both engines.

* 🍃 docs: Sync the Published Baseline Counts With the Seeded Harness

* 🍃 fix: Materialize Transaction-Read Collections and Settle Stragglers Before Teardown

Review round 4, three findings real (the fourth — stale doc totals — was
already fixed on the pushed head):

- Straggler settlement moved BEFORE engine teardown: dropping the database or
  closing the client would abort outstanding work with closed-client errors
  that are not engine verdicts, leaving a timed-out row unadjudicated while
  STRICT passes.
- The authority fixtures (sweep AND compat) materialize PluginAuth and Token
  before the transaction: `loadCurrentProof` reads both inside it, and
  DocumentDB rejects any in-transaction statement touching a non-existent
  collection — the rejection surfaces as `proof_unavailable` before the commit
  is reached. This is the mechanism behind the compat authority test failing
  on every live cluster run while passing against MongoDB; with the
  collections materialized, the next live run adjudicates the transaction
  itself instead of the missing namespace.
- Engine errors are recognized by `instanceof MongoServerError` instead of a
  `name` equality, so server-originated subclasses (MongoWriteConcernError,
  MongoBulkWriteError) can no longer slip through the session wrapper as
  domain throws.

* 🍃 fix: Materialize Group and Config for the Authority Transaction and Exclude Constructor Exports

Review round 5, both findings real. The proof transaction also reads Group and
Config; the sweep fixture materializes both (the compat fixture already
creates a Config record and now materializes Group). SessionError — an error
class exported alongside the session methods — is excluded from the method
map, correcting the totals to 509 methods / 369 driven / 140 un-driven; the
published baseline is synced in the same commit.

* 🍃 fix: Build a Fresh Method Bundle per Case and Emit a Normalized Matrix

Review round 6, both findings real:

- Every row now invokes a freshly constructed method bundle. Factory-local
  caches leaked across alphabetically ordered rows — the session methods cache
  `sessionIndexesPromise`, so `ensureSessionIndexes` returned that settled
  promise and was misreported as `no-queries`. It now issues its 3 index
  operations, making the matrix order-independent (370 driven / 139 un-driven).

- The report separates run metadata from a canonical `rows` payload with
  run-specific values normalized out (run id, generated ObjectIds and UUIDs),
  so the two engines' matrices diff cleanly instead of differing on every row
  that embeds a synthesized value in its error text.

* 🍃 fix: Materialize Authority Snapshot Namespaces Before Its Transaction

Amazon DocumentDB rejects any statement inside a transaction that touches a
collection which does not exist, and `asMCPError` converts that server
rejection into `proof_unavailable` — so on a deployment where no PluginAuth
or Token row has ever been written, every MCP authority proof fails with an
error naming nothing about the real cause. This is the mechanism behind the
live-harness authority probe failing on every real cluster run since July
while passing against MongoDB.

`loadAuthoritativeSnapshot` now materializes the nine namespaces its snapshot
reads before opening the transaction, once per process. MongoDB tolerates the
in-transaction read either way, so nothing changes there; a create failure is
logged at debug and left to surface from the transaction itself.

The regression test drops the credential namespaces to reproduce the state and
asserts the resolve recreates them and succeeds; it fails without the fix.

* 🍃 docs: Record the Authoritative Hardened Live Run

The hardened harness ran against a live DocumentDB 5.0.0 cluster: 370 of 509
methods drove at least one query, zero engine rejections, and the matrix is
byte-identical to the MongoDB baseline — every outcome and query count
matches. This run adjudicated index DDL, transaction commits and aborts, ACL
cascades, and the authority snapshot transaction, none of which the first
pre-hardening run could see.

The same run turned the compatibility suite's MCP authority probe green for
the first time on any real cluster, confirming the namespace-materialization
diagnosis for the `proof_unavailable` failures seen since July.

* 🍃 fix: Retry Snapshot Namespace Creation After a Transient Failure

The catch suppressed every `createCollection` error, so a transient failure
settled the once-per-process memo as successful: creation was skipped for the
rest of the process, the transaction kept failing on the missing collection,
and every later proof returned `proof_unavailable` even after the condition
cleared. The existing reset-on-reject path could never fire because the inner
promise never rejected.

Only `NamespaceExists` (including a concurrent creator winning the race) is
suppressed now; every other failure propagates, clears the memo, and lets the
next request retry. The regression test rejects the first creation with a
transient error, asserts the proof fails, then asserts the following request
recreates the namespace and succeeds — it fails against the swallow-all form.

* 🍃 fix: Throttle Snapshot Namespace Preflight Retries With a Bounded Cooldown

Clearing the memo on every rejection made a persistent failure — an
authorization error, say — turn each authority request into another serial
collection preflight: a DDL retry storm precisely while the database is
unhealthy, since the single-flight promise only coalesces requests while an
attempt is pending.

A failure is now recorded with its timestamp and replayed without touching the
database until a bounded cooldown elapses (30s, overridable through the
existing hooks for tests). Transient failures stay retryable, just not on
every request. Both sides of the boundary are covered: one test asserts a
persistent failure issues no further creations inside the cooldown, and the
existing retry test runs with the cooldown disabled to assert recovery still
works. The throttle test fails with the cooldown gate removed.

* 🍃 fix: Instrument Db-Level Namespace DDL in the Sweep

The authority path's namespace preflight calls `Model.createCollection()`,
which delegates to `Db.createCollection` — not to any `Collection` operation
the sweep wrapped. A DocumentDB rejection of that DDL would therefore be
laundered by `asMCPError` into `proof_unavailable`, leaving the row
classified as a mere `threw` while `DOCUMENTDB_STRICT=true` passed: the exact
false-green class this suite exists to close, reintroduced by the preflight
itself.

`Db.createCollection`/`dropCollection`/`renameCollection` are now
instrumented, and `NamespaceExists` (48) joins the benign server codes since
an idempotent preflight against an existing collection is the expected
outcome, not an incompatibility. The authority row's adjudicated query count
rises from 22 to 31, which is those nine preflight operations becoming
visible.
2026-08-31 09:54:32 -04:00

17 KiB

Amazon DocumentDB Compatibility Assessment (issue #14488)

Update 2026-08-30 — adjudicated live. Everything below was originally decided from AWS's documentation, which omits unsupported operators rather than listing them. It has now been run against a real DocumentDB 5.0.0 cluster (the version this project supports, and the one the reference deployment runs). audit.documentdb.spec.ts records the verdicts; the corrections are in "Live verdicts" at the end of this document. Two claims below were wrong: pipeline-form updates had returned in new code, and the transaction probe reported a false negative.

Adjudicated against official AWS documentation on 2026-07-28. Engine columns throughout: DocumentDB 3.6 / 4.0 / 5.0 / 8.0 instance-based and elastic clusters. AWS's supported-APIs page states that unsupported operators are omitted from its tables, so several verdicts below are implicit-by-omission and flagged as such.

Executive summary

  • The login blocker was real: three aggregation-pipeline updates (one using $$NOW) existed in the codebase. DocumentDB documents no support for pipeline-form updates on any engine version. All three are now rewritten with plain update operators that work on every engine, including elastic.
  • Partial-index creation fails on DocumentDB < 5.0 and elastic — and the failure was provably silent (no log, no crash, uniqueness quietly unenforced). Model creation now attaches an index listener so failed builds log loudly.
  • Transactions already degrade gracefully via a runtime probe. GridFS is unreachable dead code. retryWrites=false is a deployment-docs item.
  • Recommendation: DocumentDB 5.0+ instance-based is a supportable target; 4.0 runs but with degraded uniqueness enforcement (now logged); elastic clusters should be documented as unsupported (no unique indexes at all).

Proven incompatibilities — fixed in this PR

1. acceptTerms pipeline update + $$NOW (P0 — blocked login)

packages/data-schemas/src/methods/user.ts:303 used a pipeline-form findByIdAndUpdate with $ifNull/$$NOW (introduced by PR #10810, matching the reporter's regression window). When Terms gating is on, every login hits this and DocumentDB rejects it.

AWS evidence: the supported APIs page lists only classic update operators (no pipeline form anywhere); the $set/ $unset stage operators are marked unsupported for 3.6/4.0/5.0; $$NOW is absent from the System variables table entirely ($$CURRENT and $$REMOVE are explicitly "No"). Implicit-by-omission, but consistent with the reported Failed to parse update: field must be of BSON type object class of error (AWS documents no exact error string).

Fix: null-guarded first-acceptance claim (termsAcceptedAt: null matches both the schema's explicit null default and missing legacy fields — a $exists: false guard would never fire because of that default), with a plain-$set fallback for repeat acceptance. First-acceptance timestamp preservation, concurrency convergence, and the IUser | null contract are covered by tests, including a raw-inserted legacy document without the field.

2. decrementTagCounts pipeline update (P1 — silent tag-count drift)

packages/data-schemas/src/methods/conversationTag.ts:47 used a $max/$subtract/$ifNull pipeline inside bulkWrite, wrapped in a try/catch that only logs — on DocumentDB, conversation deletion succeeded while tag counts silently drifted.

Fix: two mutually exclusive plain ops per tag in one ordered bulkWrite — clamp-to-zero (count below the decrement amount, or null/missing) first, then a guarded $inc. Clamp-at-zero, missing-count tolerance, and variable-amount semantics all preserved; now covered by a new test block (previously untested).

3. extendFilesTTL pipeline update (P1 — /files/usage TTL holds fail)

packages/data-schemas/src/methods/file.ts:607 — not in the reporter's list; found by sweeping the codebase (rg for pipeline-shaped update args and $$NOW; these three sites were the only hits).

Fix: read the candidate files (one projected query), compute each file's min(now + renewMs, createdAt + maxLifetimeMs) ceiling client-side, then issue per-document guarded $sets (expiresAt: { $exists: true, $lt: next }) through tenantSafeBulkWrite. Only-widens, per-file ceiling, and cleared-TTL-stays-permanent semantics are preserved under concurrency by the write guard. Cost: one extra read round trip on this path — unavoidable without a schema change, because the ceiling is per-document.

Proven, made loud — partial indexes (P1 on < 5.0 / elastic)

Four unique partial indexes exist:

  • packages/data-schemas/src/schema/user.ts:192 and :199 — OAuth ids (googleId, openidId, …) with partialFilterExpression: { $exists: true }
  • packages/data-schemas/src/schema/file.ts:173 — execute_code files ($eq-shaped filter)
  • packages/data-schemas/src/schema/group.ts:56 — group source ids ($exists: true)

AWS: partial-index.html — "The partial index feature is supported in Amazon DocumentDB 5.0 instance-based clusters"; the index-properties table marks Partial as No/No/Yes/Yes/No across 3.6/4.0/5.0/8.0/elastic. The $exists and $eq filter shapes used here are inside DocumentDB 5.0's supported operator list ($eq, $exists, $and, $gt/$gte/$lt/$lte), so on 5.0+ these indexes build.

On 3.6/4.0/elastic the builds fail — and empirically (probe: unique index over pre-seeded duplicates, mongoose 8, autoIndex) the failure is completely silent: no unhandled rejection, no log, the index simply doesn't exist and duplicate inserts succeed. Mongoose only surfaces build errors through a Model.on('index') listener, which nothing attached. createModels now attaches one that logs every failed build (packages/data-schemas/src/models/index.ts). Operational consequence on < 5.0 remains: OAuth-account uniqueness is not DB-enforced — documented, loud, but not fixable in application code.

Proven compatible — no action needed

  • Transactions: supported on 4.0+ instance-based ("Amazon DocumentDB … supports transactions in Amazon DocumentDB 4.0 and later" — transactions.html); unsupported on 3.6 and elastic. LibreChat already probes at runtime (packages/data-schemas/src/utils/transactions.ts, cached in api/server/services/PermissionService.js) and falls back to non-transactional writes — the same mode as standalone MongoDB without a replica set. DocumentDB's restrictions (1-minute execution limit, no cursors in transactions, no retryable commit/abort) don't intersect LibreChat's usage.
  • GridFS: packages/api/src/cache/keyvMongo.ts only constructs a GridFSBucket when useGridFS is set — no caller ever sets it and the class isn't exported (the singleton uses a plain logs collection). Dead code. Moot regardless: AWS lists GridFS as supported on instance-based clusters (elastic: no).
  • TTL indexes: supported everywhere including elastic. AWS warns deletion is best-effort ("Documents are not guaranteed to be deleted within any specific period") — acceptable, since LibreChat treats TTL as cleanup, not as a security boundary.
  • $ifNull: supported on all versions (only its pipeline-update context was the problem).

Deployment requirements (documentation, not code)

  • retryWrites=false is mandatory in MONGO_URI. AWS: "Amazon DocumentDB does not currently support retryable writes"; the failure mode is {"ok":0,"errmsg":"Unrecognized field: 'txnNumber'","code":9} (functional-differences.html). api/db/connect.js passes MONGO_URI through verbatim, so this belongs in the deployment docs (and the live harness flags a URI missing it).
  • TLS with the AWS CA bundle; clusters are VPC-only (tunnel/bastion for external access).

Document as unsupported — elastic clusters

Elastic cluster limitations: no unique indexes (any), no partial indexes, no ACID transactions, no GridFS, no change streams, $expr unsupported, and the cursor-methods table even lists sort()/skip()/limit() as "No". The email + tenantId unique index alone disqualifies elastic clusters. Recommend stating this explicitly in the docs.

Undetermined — honest gaps

  • The reporter's engine version and cluster type — still unknown; it decides whether the partial-index caveat applies to them (5.0+: it doesn't). Worth asking directly on the issue.
  • DocumentDB 8.0 pipeline-update acceptance — 8.0 added $set/$unset aggregation stages, but AWS never documents pipeline-form updates; the harness probe answers this live.
  • collMod is only "Partial" on every version — avoid Model.syncIndexes() against DocumentDB (it may issue collMod beyond the documented expireAfterSeconds).
  • Read-side aggregations (3 files: methods/prompt.ts, methods/aclEntry.ts, methods/agentCategory.ts) were not audited stage-by-stage; no exotic stages ($facet, $setWindowFields, $unionWith, $graphLookup) are used anywhere.
  • No faithful local emulator exists. The documentdb-local Docker image is the PostgreSQL-based Linux Foundation project — AWS's own OSS blog confirms "a different engine than the one used in Amazon DocumentDB." Live regression testing must run against a real cluster.

Regression strategy

  1. In-repo (CI today): the behavioral tests added in user.methods.spec.ts, conversationTag.methods.spec.ts, and file.spec.ts lock the pipeline-free implementations' semantics (first-acceptance preservation, clamp-at-zero, per-file ceiling).
  2. Live harness (this directory): compat.documentdb.spec.ts exercises the exact operations behind #14488 against a real cluster and prints a capability matrix (pipeline updates, $$NOW, transactions, partial unique indexes, TTL, retryWrites). Gated on DOCUMENTDB_URI; verified green against real MongoDB as a baseline. Suggested cadence: before releases and whenever update-operator code in data-schemas changes; optionally a scheduled GitHub Action on a runner with VPC access to a dev cluster.

Support matrix and recommendation

Capability (LibreChat dependency) 3.6 4.0 5.0 8.0 Elastic
Pipeline updates (no longer used) ✗ ✗ ✗ ? ✗
Plain update operators (all writes now) ✓ ✓ ✓ ✓ ✓
Unique indexes ✓ ✓ ✓ ✓ ✗
Partial unique indexes (OAuth ids) ✗ ✗ ✓ ✓ ✗
Transactions (runtime-probed) ✗ ✓ ✓ ✓ ✗
TTL indexes ✓ ✓ ✓ ✓ ✓

Recommendation: support DocumentDB 5.0+ instance-based with retryWrites=false documented as required. 4.0 functions with partial-unique-index loss (now logged loudly at startup) — "works, with a documented caveat". Elastic clusters: unsupported, full stop.

Live verdicts (DocumentDB 5.0.0, 2026-08-30)

Probed by audit.documentdb.spec.ts, which drives the production methods themselves rather than re-implementations.

Construct Verdict Server error
Aggregation-pipeline update rejected Failed to parse update: field must be of BSON type object
$$REMOVE rejected Feature not supported: $$REMOVE
$facet rejected Aggregation stage not supported: '$facet'
$max, $set/$unset accepted —
Filtered positional $[<id>] accepted —
$regexMatch, $switch, $let, $convert accepted —
$strLenBytes, $substrCP, $mergeObjects, $map accepted —
Partial unique indexes, TTL indexes accepted —

Correction 1 — pipeline updates returned after this document was written

Six sites reintroduced unsupported constructs between 2026-07-29 and 2026-08-30, all in code added with the durable trigger and background-task work. Nothing caught them: every unit suite runs mongodb-memory-server, which is real MongoDB and accepts all of it. src/methods/documentdb.spec.ts is now a static guard against the whole class.

Correction 2 — the transaction probe reported a false negative

supportsTransactions read a canary collection that does not exist, and DocumentDB rejects a transaction touching a non-existent collection (Feature not supported: non-existent collection in transaction). The probe therefore returned false on an engine that fully supports transactions, and every caller silently took the non-transactional path. Verified directly: with the collection materialized, both read-only and multi-write transactions commit. The probe now creates the canary first.

Connection requirements for the live suites

Established against the real cluster; all four are load-bearing through a tunnel and none were documented before:

  • authSource=admin — the user lives in admin; a database in the URI path otherwise becomes the auth source and authentication fails
  • authMechanism=SCRAM-SHA-1 — DocumentDB rejects SCRAM-SHA-256 (Unsupported mechanism [ -301 ])
  • directConnection=true — replica-set discovery returns internal cluster hostnames that are unreachable through a tunnel
  • tlsAllowInvalidHostnames — the tunnel endpoint never matches the certificate

Method sweep (2026-08-31)

sweep.documentdb.spec.ts drives every exported data-schemas method (509 at the time of writing; constructor-valued exports are excluded) against a real engine, auto-synthesizing arguments, repairing them from validation errors, and counting the driver queries each method actually issues — a method that issues none is reported un-adjudicated instead of silently green. Run once against in-memory MongoDB (SWEEP_BASELINE=true) and once against DocumentDB, then diff the JSON matrices (SWEEP_REPORT_PATH).

Corrected-harness baseline (replica-set MongoDB, real ACL cascades, index DDL and transaction-lifecycle instrumentation, per-invocation async attribution, seeded authority-transaction case, constructor exports excluded, a fresh method bundle per case): 370 of 509 methods issue at least one query; zero rejections on MongoDB. The remaining 139 issue none (validation rejected the synthesized arguments, or a guard short-circuited); they are listed in the matrix and shrink by adding ARG_OVERRIDES entries. The report's rows payload is normalized for cross-engine diffing (diff <(jq .rows a.json) <(jq .rows b.json)); run metadata lives outside it.

Authoritative live run (2026-08-31, hardened harness, DocumentDB 5.0.0): 370 of 509 methods drove at least one query; zero engine rejections; the matrix is byte-identical to the MongoDB baseline — every row's outcome and query count matches, so there is no engine divergence anywhere the sweep can reach. This run adjudicated index DDL, transaction commits and aborts, ACL cascades, and the authority snapshot's own transaction, none of which the first (pre-hardening) run could see.

The same run turned the compatibility suite's MCP authority snapshot probe green for the first time on any real cluster. It had failed since July with proof_unavailable: loadAuthoritativeSnapshot reads nine collections inside its transaction, DocumentDB rejects in-transaction statements against namespaces that do not exist, and asMCPError converted that server rejection into a reason naming nothing about the cause. The method now materializes those namespaces before opening the transaction.

This sweep exists because every incompatibility found so far was invisible until someone thought to look for its class; here the engine adjudicates whatever each method emits, known class or not.