LibreChat/UPGRADING.md
Danny Avila 8a7fe7b9f3
🗃️ fix: Restore Authorization Uniqueness and Revoke Duplicate Grants (#18258)
* fix: repair duplicate authorization records and revoke every matching grant

* test: verify standalone authorization repair command and keep bootstrap isolated

* test: distinguish supported aggregate disk options in DocumentDB guard
2026-11-17 09:52:22 -05:00

268 lines
15 KiB
Markdown

# Upgrading LibreChat
## Native-only web search after v0.8.8-rc2
Since v0.8.8-rc3 (#15875), `WEB_SEARCH.USE` authorizes both LibreChat's
external search pipeline and provider-native search. `interface.webSearch: false`
sets that permission to `false` for `USER` and `ADMIN`. Removing the YAML field
preserves the stored denial; it does not restore the default grant.
For native search without LibreChat's external pipeline:
1. Explicitly set `interface.webSearch: true` and restart the API to restore
`USER` and `ADMIN` grants. This enables both roles, not just the affected user.
For selective access or custom roles, leave the field unset and enable
`WEB_SEARCH.USE` only for the intended roles in the permissions editor.
2. Remove only `web_search` from your explicit `endpoints.agents.capabilities`
list, preserving the other capabilities. Omitting the list restores defaults,
including external search. This capability does not control native search.
3. In each relevant model specification, keep top-level `webSearch: false` for
the external tool and `preset.web_search: true` for native search. Keep
`preset.useResponsesApi: true` for OpenAI-compatible native search.
4. Saved agents use their own model parameters. Enable native Web Search there.
5. Reload the client and verify the effective role permission and provider search
request. Responses API being enabled alone does not mean search is allowed.
Example fields to merge into an existing configuration:
```yaml
interface:
webSearch: true # Shared role grant, including native search.
endpoints:
agents:
# Preserve your existing list; remove only web_search.
capabilities: [deferred_tools, execute_code, file_search, artifacts, subagents, actions, context, skills, memory, ask_user_question, tools, chain, ocr]
```
An explicit YAML grant is reapplied on restart. After recovery, remove the field
if the permissions editor should own future role changes. Stored grants and
denials are preserved when the field is absent. No role reset is required.
The composer palette and its pinned chips share the external capability gate in
the updated composer. In v0.8.8, an old pinned Search badge may remain visible after
external search is disabled; the backend still refuses to load that tool (#16673).
## Redis streaming now coalesces deltas by default
Redis-backed streams now batch model and tool-argument deltas in a **25 ms**
window when `STREAM_DELTA_COALESCE_MS` is unset. Explicit `0` keeps per-delta
publication; existing explicit values retain their behavior. In-memory streams
are unchanged, including deployments using `USE_REDIS_STREAMS=false`.
The same window batches durable appends and publications. It reduces Redis
round trips and repeated guard/TTL work, but not Pub/Sub message count at the cost of up to one window of buffering latency and possible loss
of unflushed deltas on process crash. Terminal and non-coalescable barriers still
flush pending batches before proceeding.
Coalescing batches Redis requests, not the subscriber wire format: each event
still publishes as an individually sequenced `chunk` frame. Subscribers from
before batch-frame support can read these publications without a preparatory
configuration change. New subscribers also retain `chunk_batch` decoding for
interoperation with existing opt-in batching producers.
This removes the new default's batch-frame compatibility hazard; it does not
promise compatibility across unrelated generation-protocol changes. If an
existing producer already emits `chunk_batch` frames through explicit opt-in,
keep those producers away from subscribers that predate batch-frame support,
or disable coalescing on those producers before introducing older subscribers.
## Tenant index migration (v0.8.7 and earlier databases)
Upgrading an existing database can log `Index build failed` for User, Role,
Preset, AccessRole, MCPServer, AgentCategory, Message, or Conversation. Older
unique indexes (for example `email_1` or `name_1`) conflict with current non-unique
indexes of the same name. Tenant-scoped compound indexes now enforce uniqueness.
This also affects single-tenant deployments. See #15759 (successor to #14826).
The tenant-index migration is an explicit maintenance command; startup does not
run it automatically. Use the updated code/image containing this command. For a
source checkout, install dependencies and build the packages first (`npm run
build:packages`).
1. Back up MongoDB and stop all LibreChat API replicas and workers that write to
the database. Keep MongoDB available throughout the migration.
2. Preview known legacy unique indexes using the deployment's `MONGO_URI` and
connection settings from `.env`:
```sh
npm run migrate:tenant-indexes:dry-run
```
3. Apply the migration:
```sh
npm run migrate:tenant-indexes
```
With Docker Compose, stop the API and run a one-off container from the updated
image while MongoDB remains running. Use the same Compose file flags as your
deployment (for example, `docker compose -f deploy-compose.yml`). Set the
working directory to `/app`, where the root npm scripts live; the production
API image otherwise defaults to `/app/api`:
```sh
docker compose stop api
docker compose run --rm --no-deps -w /app api npm run migrate:tenant-indexes:dry-run
docker compose run --rm --no-deps -w /app api npm run migrate:tenant-indexes
docker compose up -d api
```
4. Restart writers only after the command exits successfully (status 0). Verify
that startup no longer reports these index conflicts.
The command first builds tenant-scoped unique indexes, then drops only known
superseded unique indexes, and explicitly creates current schema indexes for the
affected collections. It preserves custom indexes and current non-unique indexes;
it does not delete documents or use `syncIndexes`/`dropIndexes`. Automatic index
creation is disabled in the maintenance process to prevent races, even when the
server normally enables it. Current indexes are explicitly built even if the
server uses `MONGO_AUTO_INDEX=false`.
The dry run only lists removals: it does not validate replacement builds, database
permissions, or duplicate data. Any listing, dropping, or building error makes
the apply command fail. If replacement creation fails (for example due to
existing duplicates or unsupported database index options), no old constraints
have been dropped. Keep writers stopped, correct the reported problem, and rerun.
A later failure can leave a partial migration; rerunning safely completes it.
Index builds may take time on large collections. Do not drop all indexes to
resolve a failure.
### MCP name reservations
The original durable reservation implementation omitted persisted tenant identity.
The corrected release stores `tenantId` and uses a unique `{ name, tenantId }` index.
Existing tenantless reservations remain global tombstones because their original
ownership cannot be recovered safely. Never delete or reassign them.
For deployments that need to keep chat available, use the targeted migration below
instead of the broad offline tenant-index migration. It touches only `mcpnames`:
no document rewrites, no changes to other collections, and no startup flag.
1. Back up the database and rehearse against the same DocumentDB engine/version.
Tests cover real MongoDB; a dry run is not a DocumentDB compatibility test.
2. Temporarily reject MCP creation/deletion and account-erasure requests at your
deployment edge (including base-path aliases). Upstream routes include
`POST /api/mcp/servers`, `DELETE /api/mcp/servers/:serverName`, and
`DELETE /api/user/delete`. Also pause CLI/admin/background erasure and any
direct reservation writers; audit fork-specific paths. Let already admitted
operations finish. This is an operational gate, not a built-in application toggle.
Leave chat, reads and ordinary API traffic running.
3. Roll every API replica and other writer onto this corrected release while
keeping the gate closed. Verify image digests and terminate/drain old processes,
jobs and scripts. Do not rely on a rolling deployment having merely started.
Avoid rollback or automatic scale-up onto old images during this procedure.
4. From a single maintenance container using the updated image, the deployment's
writer `MONGO_URI`, credentials and TLS settings, run from `/app`:
```sh
npm run migrate:mcp-reservations:dry-run
npm run migrate:mcp-reservations -- --writers-upgraded
```
The acknowledgement asserts that **all writers** now persist tenant identity
and honor legacy tombstones; the command cannot discover or fence old replicas.
Do not use it to skip the preceding rollout. Automatic model index creation is
disabled in the maintenance process. Index operations run serially and use the
existing DocumentDB index-build retry/polling helper. Run only one migrator.
5. Require exit status zero. Inspect `db.mcpnames.getIndexes()`: the full unique
`name_1_tenantId_1` index must exist and the legacy `name_1` must be absent.
Test cross-tenant same-title creation and single/bulk deletion in staging,
then remove the temporary gate. Keep all writers on the corrected code.
The command builds and verifies the compound constraint **before** dropping the
old global constraint. It preserves custom indexes and legacy documents. The dry
run only reports index presence; it does not build indexes or validate permissions.
If a build fails, the old constraint remains. If dropping fails, both constraints
may remain; leave the gate closed and rerun after resolving the error. A completed
run is idempotent. Database index builds may still consume resources or briefly
block operations, depending on the engine; this avoids an API shutdown, not every
possible latency spike. Do not use the broad migration on a live database.
The original offline procedure remains available when all writers can be stopped:
`npm run migrate:tenant-indexes`. It also covers collections beyond `mcpnames`.
### Repairing duplicate roles, system grants, and agent categories
A failed unique-index build can leave legacy duplicate authorization/configuration
records. Updating the application does not remove those records or repair the
missing constraint. The corrected grant revocation removes every matching grant
in the exact requested tenant/platform scope, even before the index is repaired.
It does not revoke grants for a different tenant or remove platform grants when
revoking a tenant grant. Deploy the correction to every API replica first.
Use the targeted repair below for `systemgrants`, `roles`, and `agentcategories`.
It does **not** repair files, messages, MCP names, or all tenant indexes. It runs
only when invoked explicitly, never at startup.
1. Take a database snapshot and rehearse on a restored copy using the deployment's
actual DocumentDB engine/version. Local regression tests use real MongoDB;
they do not establish DocumentDB compatibility. Check archive, CRUD, aggregate,
and index-build privileges before the production window. Inspect which regional
deployments share a database; run once per independent writer database, not
once per API region connected to the same database.
2. From one maintenance container with the corrected image, the writer `MONGO_URI`
and the deployment's TLS settings, run from `/app`:
```sh
node config/repair-authorization-duplicates.js --dry-run
```
Preview is also the default for `node config/repair-authorization-duplicates.js`. Output contains
aggregate counts and index status only. Conflicting settings produce exit code
2 in preview; operational failures produce exit code 1. A live preview is not
a consistent snapshot and is not permission or index-build validation.
3. Gate and drain **all writes to these three collections**, including admin grant
changes, role creation/edit/rename/deletion and their grant cascades, category
changes, tenant onboarding/seeding, account/group erasure and maintenance
scripts. Audit fork-specific writers. Freeze rollouts, restarts and automatic
scale-up that could run `initializeRoles`, `seedSystemGrants` or
`ensureDefaultCategories`. Role reads can create a missing built-in role:
existing chat can continue only when its role records already exist. Gate new
tenant provisioning and requests that would create missing roles as well.
This is a deployment-controlled gate; there is no built-in maintenance toggle.
Blocking only `/api/admin/grants` is insufficient. If the deployment cannot
isolate these writers, stop writers for the repair instead of assuming a safe
online migration. Reads and unrelated message/chat writes need not be stopped.
4. Run one migrator, keeping the gate closed until every index is verified:
```sh
node config/repair-authorization-duplicates.js --apply --writers-paused
```
`--writers-paused` acknowledges the operational gate; it does not enforce it.
The command inspects all three collections before cleanup and refuses conflicting
records. It retains the lowest `_id` in each equivalent group without modifying
it, archives every complete removed record in `authorization_duplicate_archive`,
then deletes the duplicate. Role/category settings and unknown fields must agree;
only `_id`, `__v`, `createdAt`, and `updatedAt` are ignored for comparison. Grants
also ignore `grantedAt` and `grantedBy` provenance, which the archive preserves;
different `expiresAt` values or tenant-field representations are conflicts.
No permissions are unioned and no tenant is reassigned. Conflict resolution
requires a separate reviewed decision; do not simply keep the most permissive
role or restore every archived grant.
5. Require exit zero and all three `uniqueIndex` values to be true. Verify full,
non-sparse, non-partial unique indexes on `(principalType, principalId,
capability, tenantId)`, `(name, tenantId)`, and `(value, tenantId)` respectively.
The repair requests background index builds serially and drops no existing/custom
indexes. Amazon DocumentDB defaults to foreground builds; background builds
allow reads to continue ([AWS index creation guidance](https://docs.aws.amazon.com/documentdb/latest/devguide/index-creation.html)).
Rehearse and watch database load/latency; background does not mean zero impact.
Re-run preview, verify zero duplicate groups, smoke-test grant/revoke plus role
and category operations in staging, then reopen the affected writes.
Every removal is archived with majority acknowledgement and read back before
source deletion. Keep that collection access-restricted with the source data;
there is no TTL or automatic restore. A database snapshot is still required.
A failed/interrupted run may have archived/deleted some records or built some
indexes. Keep the gate closed and rerun preview before resuming apply. The archive
is idempotent; an existing archive that differs from its source blocks deletion.
Apply takes a single-migrator lock in `authorization_repair_locks`. A hard-killed
process leaves the `authorization` lock in place intentionally. Before removing
that lock manually, verify the previous process and its database operations have
finished; do not expire or steal it while an index build may still be running.
Duplicate scans use no disk spill, a 60-second server timeout, and a 1,000-record
per-group bound. Oversized/slow scans fail rather than performing an unbounded
repair. These bounds are maintenance-library options, not request-path settings.