LibreChat/examples/sdk-runtime/README.md
Danny Avila dd83596e70
🚔 fix: Enforce Host Model Policies for Responses and OpenAI Callers (#18400)
* fix: enforce host model policies for API SDK callers

* refactor: share API model policy assembly in TypeScript
2026-11-18 08:31:19 -05:00

179 lines
12 KiB
Markdown

# A remote SDK agent in real LibreChat
This example runs an actual `@librechat/agents` Run in another Node process and sends its native
SDK events through LibreChat's existing handlers, message persistence, approvals, and Agents UI.
The default model is deterministic so the example requires no provider key. The LibreChat server,
management API, browser UI, graph execution, tools, and checkpoint transport are real.
Build the shared packages from the repository root (`npm run build:data-provider`,
`npm run build:data-schemas`, `npm run build:api`) and build/start LibreChat normally. Merge the
configuration fragment alongside your existing endpoints; do not replace your complete config.
Set the same random `SDK_RUNTIME_SECRET` (at least 32 characters) in LibreChat and the worker.
```sh
node examples/sdk-runtime/server.cjs
```
Register a saved agent using the existing Management API or update an existing agent's binding:
```json
{
"name": "Remote graph example",
"provider": "openAI",
"model": "gpt-4.1-mini",
"sdk_runtime": { "id": "remote-example", "protocolVersion": 1 }
}
```
The host provider/model must be permitted by your existing LibreChat configuration. The worker
owns its own model configuration; the host does not forward its provider credentials. Select the
saved agent in the normal Agents picker and send a message. Approve `inspect_example` when asked.
The graph plans, runs an SDK agent with a real tool, reviews, then refines through a conditional
loop. Reload during execution or while awaiting approval, resume, stop, and inspect saved history.
Stop cancels this example's execution; it does not automatically undo completed side effects.
Set `SDK_EXAMPLE_GRAPH=simple` for the minimal one-agent example in `minimal.cjs`.
The default `agent.cjs` demonstrates a custom StateGraph with planning, SDK tool execution,
review, and conditional refinement. For actual provider calls,
set `SDK_EXAMPLE_LIVE_MODEL=true`, `OPENAI_API_KEY`, and optionally `SDK_EXAMPLE_MODEL` **on the
worker**. Set `SDK_EXAMPLE_STEP_DELAY_MS=10000` to make reconnect/stop easy to observe. Set `SDK_RUNTIME_AGENT_IDS` to a comma-separated allowlist of saved agent IDs for an
additional deployment restriction.
Use the supplied Mongo checkpointer configuration for approvals that survive rebuilding a worker Run.
Serve production workers through TLS (`wss://`) and enforce network access to the deployment.
Only loopback workers may use unencrypted `ws://`. Browser clients never receive the runtime secret.
The bridge is an execution transport, not a second renderer. Browser reconnect uses LibreChat's
existing job stream. By default, a lost host-to-worker connection aborts execution and fails the turn.
Enable the opt-in recovery configuration below to reattach the same live Run and replay committed
events without repeating execution. Approval resumes rebuild a
fresh worker Run against host-owned checkpoints, without sharing database credentials.
Remote runtimes can execute their own operator-installed tools and the saved root agent's
authorized host schema tools. Install `input.hostTools.toolDefinitions` on the SDK agent as the
examples do; the supplied SDK handler dispatches those tools on the host. Provider credentials
and MCP tokens remain host-owned. Configure worker-owned domain access separately and authorize
it using the signed caller identity. Host policy hooks still gate SDK tool execution.
Host-tool capability 2 requires matching host and worker upgrades and propagates per-invocation
cancellation without closing the run's socket; older versions fail before host-tool execution.
## Measuring transport overhead
With the worker running, run `node examples/sdk-runtime/benchmark.cjs` with the same secret.
It compares five warm executions of the identical deterministic custom graph locally and in the
separate process, reporting first text event and completion medians. It includes Run construction
and the socket handshake, but excludes browser rendering, TLS, WAN latency, and provider inference.
Use the browser test `e2e/specs/mock/remote-sdk.spec.ts` for native approval/reload/stop verification.
Its test attachment also records send-to-approval and approval-to-visible-output timings.
## Capability status
This example validates the initial transport. It does **not** establish all main-chat features.
Read the [parity matrix and rollout gates](../../docs/sdk/agents.md#main-chat-parity-and-rollout-gates)
before enabling a product workflow. In particular, host model/PII protection blocks remote
execution, and model/tool builder settings, files/artifacts, steering, nested usage/billing, and
host tracing policies are not automatically inherited by the worker.
## Recovering a dropped host-worker connection
Enable `remote.recovery: {}` in the host registration, then start the example with
`SDK_RUNTIME_RECOVERY=true SDK_RUNTIME_REDIS_URI=redis://127.0.0.1:6379`.
Both sides must run the new bridge version. The worker owns this Redis connection; it never
receives the LibreChat database credentials. `SDK_RUNTIME_JOURNAL_PREFIX` separates deployments.
The production API is `createRemoteSdkServer({recovery:{journal,maxDisconnectMs,ownerId}, ...})`.
`createRedisRuntimeJournal(redis, limits)` accepts an operator-owned Redis-compatible `eval`
client (optionally `evalsha` to avoid resending Lua on every operation) and explicit prefix, retention, lease, frame, byte and batch limits. The example values
are illustrative: choose retention and Redis persistence/replication for your data policy.
The host YAML defaults are a 15-second disconnect window, 250-ms retry delay and 64-MiB
request-local outgoing journal. Keep both sides' disconnect windows aligned and shorter than
`rpcTimeoutMs`. Recovery preserves the same Run and pending RPC/policy replies; it does not
re-execute the factory or a completed tool after a transient connection loss. Cancellation is
an acknowledged control message and is resent after reconnect.
A completed/paused terminal result is committed atomically with its terminal journal state,
then published. It can replay after a worker restart. A restart of an **in-flight** graph instead
replays committed output and fails safely once the old lease expires: arbitrary side effects
cannot be resumed safely by retrying execution. Live reconnect requires routing to the same
worker process. For a worker pool, set each worker's stable `SDK_RUNTIME_OWNER_ID` and configure
operator-owned direct routes on the host:
```yaml
recovery:
routes:
worker-a: wss://worker-a.internal/sdk
worker-b: wss://worker-b.internal/sdk
```
The initial connection may use the pool URL. After its authenticated ready handshake, the host
reconnects directly to that allowlisted owner, binding the owner ID in the signed reconnect claim.
A worker cannot supply an arbitrary destination; missing routes, missing owner identities with a
route map, and mismatched owners fail closed. Owner identities and routes must remain stable for
the lifetime of a session. A route must identify one worker, not another balanced pool. Single-worker
deployments can omit both the identity and map. Host process restart adoption and cross-worker live execution migration are
not supported. The existing host checkpointer still supports explicit approval continuation.
Recovery batches queued frames within the configured journal byte/frame limits, persists cumulative
ACKs separately from control-frame dispatch, and uses EVALSHA with a NOSCRIPT-only fallback when
supported. Writes still commit before publication; cancellation does not wait behind ACK writes.
Final snapshots use bounded ordered chunks and an atomic terminal receipt, including offline
completion/replay. The host's `maxBufferedBytes` bounds snapshot reassembly as well as queued traffic.
Journal keys retain sensitive event/checkpoint data until their TTL; delivery acknowledgements
do not delete history. Operators must restrict and encrypt Redis access, configure durability,
and integrate `journal.purge({sessionId,scope})` with any earlier deletion requirements. Automatic
account/conversation deletion does not yet purge worker journals; use the required retention
policy before rollout. Storage failure, overflow, lease loss, expired/missing replay state, and
identity mismatches fail closed. They never trigger a fresh execution of a started session.
Run `npm run test:cache-integration:sdk --workspace=@librechat/api` against local Redis for
fault-injection tests, and the Redis Playwright lane's `remote-sdk-recovery.spec.ts` for a real
LibreChat custom-graph approval, wire outage, reconnection and saved-history test. Set
`SDK_RUNTIME_RECOVERY=true` for `benchmark.cjs` to include Redis journal overhead.
## Protected provider-text benchmark
With SDK 4.2.7 and this bridge on both sides, run
`SDK_BENCHMARK_PROTECTION=true node examples/sdk-runtime/benchmark.cjs` against the
example worker. Add `SDK_RUNTIME_RECOVERY=true` to include Redis recovery. Both
local and remote executions use the same deterministic pass-through output policy;
the remote path adds host inspection through the SDK contract. This benchmark does
not implement a production PII detector or enable protected saved-agent chat.
See [host provider-text services](../../docs/sdk/agents.md#host-provider-text-service)
for admission ownership, encrypted journal payloads, and the remaining model/input
policy guards. The existing factories retain `input.config`, including the new
provider-text protection adapter, for both the minimal and custom-graph examples.
## Host-owned model benchmark
With SDK 4.2.8 and this bridge on both sides, start the example worker as above, then run:
```bash
SDK_BENCHMARK_HOST_MODEL=true node examples/sdk-runtime/benchmark.cjs
```
The model now executes in the host benchmark process. The separate worker still owns the custom
plan/inspect/review/refine graph and its example tool. Both the local and remote runs use the same
SDK host-model facade and deterministic model. Add `SDK_BENCHMARK_PROTECTION=true` for the existing
text-policy path, and `SDK_RUNTIME_RECOVERY=true` for Redis recovery. These are benchmark options,
not saved-agent rollout toggles. The minimal and complex factories both retain `input.hostModel`
when supplied; neither needs a provider credential for a host-owned model.
See [host model transport](../../docs/sdk/agents.md#host-model-transport) for streaming credit,
encryption, cancellation, callback ownership, and the remaining builder/policy rollout gates.
### Use the saved agent's host model
Enable `endpoints.agents.sdkHostModels: {}` and `remote.hostModel: true` on the operator's
runtime registration to select the saved agent's primary model and parameters. Both examples
consume `input.agent.model` and retain `input.hostModel`; model credentials stay in LibreChat.
The custom graph keeps its plan/review/refine nodes and uses the host-authorized tool schemas.
This mode requires updated workers with `builderModel: 1` and, for initial persistence admission,
`modelAdmission: 1`. Chat, Responses and OpenAI-compatible callers use native host input callbacks.
Direct programmatic callers must supply those callbacks; local graph tools and provider fallbacks
remain gated. Combining initial admission with explicit provider-text output protection requires
`modelAdmissionWithProviderText: 1`, advertised
only with SDK 4.2.9 invocation-context support. Upgrade separately deployed workers to SDK 4.2.9
and this bridge version; older workers stay gated. See [the parity matrix](../../docs/sdk/agents.md#main-chat-parity-and-rollout-gates)
and [configuration](../../docs/sdk/agents.md#saved-agent-builder-models) before enabling it.