mirror of
https://github.com/danny-avila/LibreChat.git
synced 2026-11-18 14:08:21 +00:00
33 commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
f07750b089
|
🧱 ci: Block the PWA Service Worker Across the Mock E2E Profiles (#18063)
* 🧱 ci: Block the PWA Service Worker Across the Mock E2E Profiles Every mock test opens a fresh browser context, so the app's service worker installs and claims the page during each test's first load. A `page.route` installed within a few milliseconds of that claim is never consulted: the request it should hold goes to the server, and a test waiting on the route times out. 113 mock spec files install routes and seven blocked the worker themselves. Set `serviceWorkers: 'block'` in the mock profile's shared `use`, which every profile derived from it inherits. No spec asserts worker behavior; one that does can opt back in with `test.use({ serviceWorkers: 'allow' })`. * 🧪 test: Keep the Sandpack Preview Worker in the Plain Text Artifact Spec `serviceWorkers: 'block'` stops every service worker in the browser context, not only the app's. Sandpack's static template serves an artifact's files from a service worker inside its preview iframe, so with the profile blocking workers the preview stays empty and the spec that reads it fails every attempt. Opt that one spec back in and say in the profile's comment that the setting reaches third-party workers too. |
||
|
|
c6b7a5ac12
|
🎲 ci: Fail Playwright Runs When a Test Passes Only on a Retry (#18051)
Playwright reports a test that fails and then passes on a retry as "flaky" and exits 0, so a regression that fails two attempts in three merges green. Set `failOnFlakyTests` in CI on the mock profile (inherited by the Redis transport, registered-email and other derived profiles) and on the base profile. Retry counts are unchanged: the later attempts still record the trace and video and show whether a failure repeats. A run failed this way still prints "N flaky" beside a passed count, so add a reporter that names each retried pass in the job log and the job summary and says that it is what failed the run. |
||
|
|
c3d5815d50
|
🛰️ fix: Poll the Flux Cluster Returned by the Global Endpoint (#17907)
* fix: Poll the Flux Cluster polling_url Returned by the Global Endpoint * fix: Only Forward the Flux Key to Same-Origin or BFL Polling URLs * test: Cover Flux polling_url Routing in the Mock Harness |
||
|
|
5a633524a2
|
🔑 fix: Remember Tool Approvals for MCP Servers Using Env References (#17889)
* fix: Remember Tool Approvals for MCP Servers Using Env References * fix: Resolve Approval Env Only for Tiers That Expand It, Add E2E Scenarios * fix: Reject User Placeholders in Resolved Env Values and Take the Env Source as a Parameter * test: Expect the Resume Reply After Approving the Env MCP Tool * fix: Check Fully Expanded Config Values for User Placeholders * fix: Resolve Approval Env Into a Prototype-Free Map From Own Properties |
||
|
|
ea584cd519
|
🧾 fix: Count Google Native Media Blocks in Precomputed Token Budgets (#17890)
* fix: Count Google Native Media Blocks in Precomputed Token Budgets * test: Cover Gemini PDF context accounting in the mock harness |
||
|
|
27e8930d52
|
⏱️ ci: Balance Playwright Shards by Recorded Duration (#17019)
Plan the complete mock suite's shards from recorded per-file durations instead of Playwright's equal-count split, widen the memory lanes from six to ten, and give the nightly full suite ten shards per store so it fits its 30-minute job timeout again. |
||
|
|
5de60f36b9
|
🪣 feat: Look Up Bitbucket Cloud Pull Requests (#16987)
* feat: Look Up Bitbucket Cloud Pull Requests * fix: Keep the Pull Request Host Out of Overrides and Per-Host Keys A role, group, user or tenant override can no longer change or remove pullRequests.host, so feature gating and link recording see the operator's host. Links are recorded only from the configured host, and the batch share key and the rate limit cooldown are scoped by host. * fix: Compare Pull Request Link Hosts Through the Host Resolver * test: Add Pull Request Scenarios Against a Fake GitHub Enterprise Host * fix: Keep the Pull Request Provider Out of Overrides and the Batch Share Key A role, group, user or tenant override can no longer change pullRequests.provider, so feature gating and link recording agree with the lookup, and a batch lookup for one provider is never joined by a request for another. * fix: Read GitLab History Apart From Open Merge Requests Merged and closed merge requests are listed on their own, so open fork merge requests on the branch name cannot crowd them out. Rate limit cooldowns are scoped by provider, and a workspace repository with a dot segment is rejected. * fix: Revalidate Bitbucket Pull Requests and Skip Ones Without a Source Commit Read a full closed page so a newer closed fork cannot hide an older pull request from the repository, recheck the source branch and fork status on the detail, and treat a pull request without a source commit as unmatchable instead of failing the lookup. * fix: Follow Bitbucket's Next Link When Listing Pull Requests * fix: Record the Lane Pull Request Without an Aggregation Pipeline Update * fix: Match an Empty Lane Repository as Unknown * fix: List Locked GitLab Merge Requests With the Open Ones * fix: Replace an Expiring Fake Git Host Certificate |
||
|
|
92432e95d5
|
🔐 feat: Enforce Required Two-Factor Authentication (#14731)
Some checks are pending
Backend Unit Tests / Build packages (push) Waiting to run
Backend Unit Tests / Codegraph select (push) Waiting to run
Backend Unit Tests / TypeScript type checks (push) Blocked by required conditions
Backend Unit Tests / Circular dependency checks (push) Waiting to run
Backend Unit Tests / Tests: api (shard 1/3) (push) Blocked by required conditions
Backend Unit Tests / Tests: api (shard 2/3) (push) Blocked by required conditions
Backend Unit Tests / Tests: api (shard 3/3) (push) Blocked by required conditions
Backend Unit Tests / Tests: data-provider (push) Blocked by required conditions
Backend Unit Tests / Tests: data-schemas (push) Blocked by required conditions
Backend Unit Tests / Tests: @librechat/api (shard 1/4) (push) Blocked by required conditions
Backend Unit Tests / Tests: @librechat/api (shard 2/4) (push) Blocked by required conditions
Backend Unit Tests / Tests: @librechat/api (shard 3/4) (push) Blocked by required conditions
Backend Unit Tests / Tests: @librechat/api (shard 4/4) (push) Blocked by required conditions
Codegraph E2E Votes / vote (full suite) (push) Waiting to run
Docker Dev Branch Images Build / publish (push) Waiting to run
Frontend Unit Tests / Client build recovery regression (push) Waiting to run
Frontend Unit Tests / Codegraph select (push) Waiting to run
Frontend Unit Tests / Build packages (push) Waiting to run
Frontend Unit Tests / TypeScript type checks (client) (push) Blocked by required conditions
Frontend Unit Tests / Tests: @librechat/client (push) Blocked by required conditions
Frontend Unit Tests / Tests: Ubuntu (shard 1/2) (push) Blocked by required conditions
Frontend Unit Tests / Tests: Ubuntu (shard 2/2) (push) Blocked by required conditions
Frontend Unit Tests / Vite build verification (push) Blocked by required conditions
Sync Locize Translations & Create Translation PR / Sync Translation Keys with Locize (push) Waiting to run
Sync Locize Translations & Create Translation PR / Create Translation PR on Version Published (push) Blocked by required conditions
* feat: enforce required two-factor authentication Require local and LDAP accounts to enroll in TOTP before they can use the app when ENFORCE_TWO_FACTOR_AUTHENTICATION is set. Sign-in, refresh, bearer verification, optional share auth and image authorization all route an unenrolled account into a staged setup flow, and every bearer credential minted before enrollment or a password reset is retired. Password reset retirement uses the existing credentialsChangedAt stamp, and commitPasswordReset clears any staged enrollment in the same write. Passkey sign-in follows the same enrollment gate as password sign-in. * test: cover passkey sign-in handing off to required enrollment * refactor: keep Mongoose user types out of two-factor auth contracts * fix: withdraw a minted session when its retirement recheck fails * fix: hold Agent API keys and the reset CLI to the required two-factor policy * fix: adopt a setup token handed off after the setup screen mounted * fix: move every tab that receives an enrollment hand-off into setup * fix: read the enrollment gate's retirement cutoff in system scope * fix: clear minted session cookies before the fallible session deletion * fix: adopt a setup token persisted before the setup screen subscribed |
||
|
|
04461d4cdb |
⛴️ fix: Keep MCP App and Speech Limits in One E2E Config
(cherry picked from commit
|
||
|
|
f18db2e84e |
🧪 test: Cover MCP App Operation Limits in Authenticated Startup Config (#16429)
(cherry picked from commit
|
||
|
|
7b27e7941e |
🧩 feat: MCP Apps support (#13831)
* feat: add MCP Apps support
* fix: preserve MCP App results and enforce sandbox policy
* fix: bind persisted MCP Apps to their server configuration
* fix: complete MCP Apps runtime integration
* fix: type the optional MCP App binding codec explicitly
* test: provide MCP Apps policy in bridge mocks
* fix: pass admitted App policy to the tool call owner
* fix: align MCP Apps CI contracts
* test: preserve MCP abort reasons in recovery mocks
* fix: preserve MCP compatibility across policy profiles
* test: use profiled MCP catalog generations
* test: expect the standard MCP session profile
* fix: remove the unused MCP abort import
* test: expect profiled MCP fixture arguments
* test: seed profiled MCP manager fixtures
* Fix request-header bearer resolution after canary merge
* Preserve MCP App views across canary message updates
* Document message-owned MCP App placement
* ci: Recheck MCP Apps Against Canary Base
---------
Co-authored-by: Lia <lia@librechat.ai>
(cherry picked from commit
|
||
|
|
4617302116 |
🔑 feat: Passkey Authentication with WebAuthn Sign-In and Management (#14715)
* feat: passkey authentication with WebAuthn sign-in and management
* refactor: tighten passkey account protections
Require password confirmation before adding or removing a passkey, and rate
limit those confirmations per user in their own namespace. Restrict passkey
enrollment and sign-in to local accounts, and re-evaluate account status once
the credential owner is resolved. Require user verification on the sign-in
ceremony. Invalidate outstanding access tokens on password reset. Tighten
redirect target validation and sanitize ceremony failure logs.
Adds coverage across the passkey controller, the WebAuthn helpers, the step-up
limiter, both JWT strategies, the redirect utility and the passkey settings UI.
* fix: localize passkey default name and respect router basename
Send a localized default credential label at enrollment so non-English
locales no longer persist the server's English fallback, and route the
passkey post-login and 2FA navigations through the deployment base path
so subdirectory installs stay inside the router.
Also mock syncStaticTools in the server boot specs; without it the
post-listen MCP initialization calls undefined and exits the worker.
* fix: harden passkey counter checks and reach stranded credentials
Make the signature counter write a compare-and-swap so two assertions
that verified against the same stored counter cannot both commit, and
reject the losing one; a cloned authenticator is otherwise accepted
inside that window. Counterless authenticators keep restamping at 0.
Keep the passkey panel reachable for identity-provider accounts, which
already get passwordless removal server-side but had no way in, while
hiding the add control they cannot use. Cascade passkey deletion from
the admin user-deletion path so credential IDs are not stranded.
* fix: reject stale 2FA temp tokens and surface passkey load failures
A temporary 2FA token proves only the first factor, so honouring one
minted before a password reset let anyone holding the second factor
survive account recovery. Apply the same credentials-changed cutoff the
JWT strategy uses.
Also distinguish a failed passkey query from an empty list; the error
state previously rendered as "no passkeys yet", which misreports what
is protecting the account.
* fix: fail closed when the passkey counter write cannot commit
Returning success on a storage error bypassed the compare-and-swap it
exists to enforce, so a cloned authenticator could be accepted while the
stored counter stayed stale. A database that cannot take this write
cannot mint the session either, so little is gained by failing open.
Counterless credentials carry no clone signal and keep their old path.
* fix: keep the passkey 2FA handoff inside the router
A full-page navigation to the 2FA screen remounted AuthContextProvider,
which silent-refreshes on mount; no refresh session exists yet at that
point, so it bounced back to login before the code could be entered.
Route through the router like the password flow does.
Also reveal the removal step-up field when the server refuses a
passwordless attempt, so an account migrated to an identity provider
without its old hash cleared can still clear its credentials; drop the
staged password when the dialog closes; and revoke passkeys in the CLI
password reset, which bypassed the HTTP controller's cleanup.
* fix: apply one email-verification policy across login methods
Accounts predating mandatory verification are marked verified as they
sign in when no email is configured, but only the password strategy did
it, so the same legacy account was admitted by password and refused by
passkey. Extract the rule into a shared helper and call it from both.
* fix: refuse a passkey sign-in when the ban check cannot complete
checkBan defaults next to a no-op, so calling it bare discarded any
internal failure and left req.banned unset, letting the request fall
through and be issued tokens. Capture the error the middleware chain
would have propagated and reject instead.
* fix: refine passkey management dialogs
* test: stabilize passkey tooltip assertions
* test: reconcile passkey specs with the canary reset and deletion flows
The credentialsChangedAt stamp now lives in commitPasswordReset, so its
assertion moves to the passwordReset unit spec. Restores the describe
blocks the rebase merged together.
* test: mock passkey routes in the cross-site login suite
* refactor: move passkey handlers and credential checks into packages/api
The passkey controller, the credentialsChangedAt predicate and the legacy
email-verification grandfathering now live in TypeScript with injected
dependencies; /api keeps only the wiring that builds the handlers.
* style: align passkey UI with Tailwind v4 order and design-rule lint
* fix: revoke earlier access tokens on a CLI password reset
passwordVersion is not a user schema field, so the write was dropped and
tokens issued before the reset kept verifying.
* test: cover passkey sign-in and credential revocation end to end
* fix: hide passkey sign-in when email login is disabled
The passkey endpoints already refuse under ALLOW_EMAIL_LOGIN=false, so
advertising the button only led to a 403.
* fix: let only one concurrent verification consume a passkey challenge
Without a native getDel the challenge was read then deleted, so two
concurrent verifications could both use it. The delete now arbitrates:
only the caller whose delete removed the key gets the challenge.
* test: run passkey ceremonies on the localhost relying party
WebAuthn refuses an IP address as the RP ID, and the harness serves the
app on 127.0.0.1. The reset scenario runs once per project, so the mock
harness lifts the per-IP reset request limit as it does for forks.
* fix: keep passkey management when email login is disabled
Gate only the sign-in button on emailLoginEnabled, since only the sign-in
routes sit behind that gate; list, rename and remove stay reachable.
* fix: build the passkey credential index before the first registration
With MONGO_AUTO_INDEX=false the unique credentialId index was never
created, so concurrent registrations could store the same credential.
* test: enable password reset in the mock harness and cover email-login-off passkeys
* fix: stop re-exporting the passkey cap from data-schemas
The bundler left every import of methods/passkey.ts unlinked while that
module re-exported MAX_PASSKEYS_PER_USER from librechat-data-provider, so
the index build and the counter-failure log threw ReferenceError at
runtime. Consumers now import the cap from librechat-data-provider.
* test: close nested settings dialogs before logging out in passkey scenarios
* fix: reject cross-site passkey assertions
The verify route accepted a form posted from another site, which let an
attacker sign a victim into the attacker's account. It now runs the same
requireSameOrigin guard as password login and the 2FA step.
* fix: end refresh sessions on a CLI password reset
A surviving refresh session could mint an access token issued after the
credentialsChangedAt stamp, so the CLI now deletes sessions like the web
reset does.
* fix: keep the passkey button locked when it supersedes autofill
Starting the modal ceremony aborts the pending autofill one, and the
autofill error path cleared the shared lock and busy state while the
modal ceremony was still running.
(cherry picked from commit
|
||
|
|
ec1304f684
|
🛅 fix: Preserve Agent Management Upload IDs (#15840)
* fix: preserve agent management upload IDs with E2E coverage * test: clean up agent management upload fixtures |
||
|
|
fed09dacb6
|
📎 test: Verify Stateful Sandbox Upload Continuity (#14363)
Covers the one leg the credential-free mock suite cannot fake: a file attached through the UI's Code Environment target must land in the live Code API, get delivered into the sandbox, and stay readable across turns. - fake-model: E2E_EXEC_UPLOADED:/E2E_EXEC_PERSIST: markers emit real bash_tool calls through the production tool pipeline (the sandbox surface stateful agents register; execute_code stays host-wired and never appears in the tool registry), plus an env-gated dump of each run's advertised tools for future spec authoring - spec: attach uploads a CSV to the Code API, turn 1 reads it back from /mnt/data and drops a proof file no upload contained, turn 2 (sent without an attachment) reads both — only possible when the run reuses the same stateful runtime session - e2e config: agents capabilities gain stateful_code_sessions, and the mock profile pins CHECK_BALANCE=false so a developer's local .env can never fail every send with zero-balance users - skips unless LIBRECHAT_CODE_BASEURL is provided (kept out of CI) |
||
|
|
ec15075dc8
|
🚚 feat: Route Attachments and Provision Tools Lazily (#15763) | ||
|
|
1c7a172321
|
🕹️ feat: Configure BYOM Tool Permissions (#15504)
* feat: add configurable BYOM permissions * test: cover BYOM permission settings live * fix: harden BYOM permission settings * perf: reuse principals for environment discovery * chore: normalize BYOM settings style * fix: Align BYOM Discovery and Admission * fix: Harden BYOM Permission Boundaries * fix: Align BYOM Skill Approval Routing * fix: Preserve Lazy BYOM Approval Capabilities * perf: Parallelize Cached Environment Reads |
||
|
|
dfb264811c
|
🪢 test: Prove MCP OAuth Resume End to End (#15485) | ||
|
|
a9ccac8656
|
🧲 feat: Enable Secure Attached Environment Pairing (#15355)
* feat: add secure code environment pairing * fix: satisfy code environment type checks * fix: secure code environment administration * fix: isolate code pairing control plane * fix: validate code pairing control responses * fix: secure code pairing transport * fix: validate code pairing wire format * fix: harden pairing secret lookup |
||
|
|
c1cb591d49
|
🗿 feat: Add Attached Stateful Code Environments (#15352)
* feat: add attached stateful code environments * fix: include code environment in lazy agent type * fix: harden stateful environment routing * fix: complete code environment route isolation * fix: declare code environment map type * fix: preserve configured code execution routes * fix: harden stateful environment updates * test: assert route-scoped sandbox readiness * fix: harden stateful environment lifecycle * fix: isolate migrated code sessions * test: assert route-qualified code sessions |
||
|
|
1489623fa3
|
🎬 test: Record-Once/Replay-Forever Model Fixtures for Mock E2E (#15210)
* 🎬 test: Record-Once/Replay-Forever Model Fixtures for Mock E2E The mock e2e lane's only credential-free model is a hand-authored script: `fake-model.js` decides responses from ~60 `E2E_*` prompt markers. That covers scripted shapes well, but no scenario replays a *real* recorded provider conversation through the assembled chain, so real streaming shapes — provider chunk cadence, reasoning deltas, usage metadata — are only ever approximated. This adds a record-once/replay-forever tier alongside the marker routing. Record (`E2E_MODEL_FIXTURES=record`, needs a provider key): the run hook appends a LangChain callback handler to every agent context's `clientOptions.callbacks` instead of overriding the model, so the REAL provider streams while each invocation's `AIMessageChunk`s serialize to `e2e/fixtures/model-replay/<name>.jsonl` — text deltas, tool_call_chunks, reasoning kwargs, and genuine usage metadata. Only the latest human text is recorded for binding; system prompts and tool schemas never enter the fixture. Replay (default, keyless): `fake-model.js` consults `tryBindReplay` ahead of marker routing, binding a conversation whose prompt matches the next unconsumed invocation. The replaying model is not hand-assigned — it is registered as SDK provider `librechat-e2e-replay` via `registerProvider` and constructed through the SDK's own `initializeModel`, so registry lookup, constructor clientOptions, and real `bindTools` all run the way a live provider's would. Recorded chunks therefore stream through the same createRun → graph → SSE → persistence chain. Consumption is enforced rather than assumed: every invocation re-checks its prompt against the recording, an invocation past the end of the script throws, and a per-fixture ledger lets the spec assert at teardown that every recorded invocation and chunk was drained. Streaming incrementality is asserted from that ledger, not by sampling transient DOM, which is a race by construction. The credential-free profile is unchanged when not recording: the record provider and its selector entry are template markers that stay comments, and no existing spec's routing is touched (a fixture only binds on an exact prompt match; everything else falls through). Verified: record vs the real DeepSeek API 1 passed (15.2s); keyless replay 1 passed twice (11.4s, 12.7s) with the ledger fully drained (2/2 invocations, 10/10 chunks, no overruns or mismatches); app-load, completion, and chat 10 passed unchanged. * fix(e2e): rebind a replay fixture from the top for a new conversation The replay cursor is process-global while the web server outlives a Playwright retry, so a fully consumed fixture left the retry unable to bind its first prompt: it fell through to marker routing and failed deterministically, burning every configured CI retry. A partially consumed attempt failed the same way. Binding now restarts the fixture when the incoming prompt matches its first recorded invocation, resetting the ledger with the cursor so the new attempt is judged on its own consumption instead of accumulating the previous one's counts. Continuing an in-progress binding still outranks restarting, so a fixture whose opening prompt repeats later in the script advances rather than rewinding. The over-consumption guard is untouched — it fires inside the stream when the cursor passes the end, not at bind time. * fix(e2e): close three replay-lane gaps found in review Restart the recorder on a retry. Its state is process-global like the replay cursor, so a failed attempt that had already recorded invocations left the counter advanced: the retry appended 2/3 after 0/1, or kept the previous attempt's `error` line, and the fixture was unusable for replay. Recording now truncates and restarts when the opening prompt reappears, mirroring the replay side's rule and its caveat. Retain a consumed binding for the conversation that drove it. An extra user turn past the final recorded invocation found no next invocation and fell through to ordinary fake-model routing, so it was answered with a mock reply: the over-consumption guard never ran and the already-drained ledger still passed. Such a conversation is now recognized by its human turns opening with the fixture's recorded prompts, and stays bound so the stream raises the overrun. Continuing an in-progress binding still outranks restarting, which outranks retaining a consumed one, so a retry's fresh conversation rewinds rather than being read as an extra turn. Validate the fixture the recording actually wrote. Record mode honors `E2E_MODEL_FIXTURE_NAME`, but the spec always inspected the committed `deepseek-two-turn`; another name wrote elsewhere while the assertions read the pre-existing file, and because the prompts are fixed the stale answers could match and green a run that verified nothing it produced. * fix(e2e): make replay binding correct for tool and subagent fixtures Round two's retry and consumed-binding fixes both assumed one model invocation per user turn. A turn that calls a tool breaks that: the model is invoked again after the tool result under the same latest human message. Identify a retry by the conversation boundary, not the prompt. The recorder ran per invocation and truncated whenever the opening prompt reappeared, so a tool round trip looked like a retry and discarded the recorded tool-call invocation. Restart detection now sits in `installRecorder`, which runs once per `createRun`: a turn whose history holds no prior human message begins a conversation. Compare consumed bindings against user turns, not invocations. Several recorded invocations can share one prompt, so a one-to-one comparison could not recognize the originating conversation — invocations `[A, A, B]` against history `[A, B, C]` failed on both length and elements, and the extra turn fell through to the fake model with the drained ledger still passing. Fixtures now carry their collapsed turn sequence. Override the subagent model too. `graph.overrideModel` is not inherited by child executors, so a fixture recording a subagent call — record mode captures child invocations already — would leave the child on its configured provider: an underrun, and a real provider request in a lane that must stay keyless. Reject ambiguous prompt matches. Binding order followed filesystem enumeration, so a second fixture sharing a prompt could silently redirect a scenario to the wrong chunks and ledger; the spec's choice never reaches the server-side loop, so ambiguity fails instead of picking a winner. Fixture identity is the file name for the same reason — a recorded `meta.name` is descriptive, and trusting it let a copied fixture collapse onto another's registry key and ledger. Prove the recording is fresh. The spec removes the selected fixture before driving, so a run whose hook never installed the recorder fails instead of greening against a stale artifact whose answers still match these deterministic prompts. * fix(e2e): rewind a replay fixture at the conversation boundary Consecutive invocations can share a prompt — a tool call produces exactly that — so an attempt stopping mid-turn left the cursor on an invocation whose text still equalled the opening prompt. Matching the cursor first meant a retry's fresh conversation resumed after the tool call instead of rewinding, consuming the post-tool invocation and silently replaying a different script than was recorded. A conversation boundary now outranks an in-progress cursor: a fresh conversation whose prompt opens the fixture rewinds even when the cursor would have matched. Continuing still outranks restarting within a conversation, so a turn that calls a tool advances to its post-tool invocation rather than rewinding on its own repeated prompt. * fix(e2e): refuse cross-conversation binding and prove content streaming A fresh conversation could steal a partly consumed fixture's later turn. Only a conversation opening with the fixture's first prompt was treated as a boundary, so after `[A, B]` had consumed `A`, an unrelated new conversation whose first message was `B` matched the cursor, received the recorded second-turn response, and advanced the shared cursor without ever having driven `A`. A conversation start may now only rewind a partly consumed fixture, never continue it; continuation within a conversation is unaffected. The incrementality assertion counted empty frames. Providers emit empty initialization and usage-metadata chunks around the content deltas, so a total chunk count above one was satisfied by a single delta: the previous fixture's closing turn had four chunks and one content-bearing delta carrying the whole answer, and both modes stayed green without proving incremental assistant-content streaming at all. Fixtures now track content-bearing chunks separately, the closing prompt asks for prose rather than a number, and both modes require several content deltas on that turn. Re-recorded: the closing turn now carries 28 content deltas. * fix(e2e): scope record mode to the fixture spec `E2E_MODEL_FIXTURES=record` replaces the fake-model hook globally, so an unfiltered entry point such as `npm run e2e:mock` sent every spec under specs/mock to the paid real-provider endpoint, while each fresh conversation truncated and rewrote the one selected fixture — leaving an artifact from whichever scenario happened to run last. Record mode now matches only the fixture spec: an unfiltered recording run lists one test instead of 203. Replay mode is untouched and still collects the full suite. * 🪪 fix: Bind Replay Fixtures by Conversation, Not Prompt Text Prompt text was standing in for conversation identity, and three review rounds found the same class of defect underneath it: a tool call repeats a prompt across invocations, a retry repeats it across attempts, and a resumed run has neither prompt nor history because `createRun` is rebuilt with no messages while state comes from the checkpoint. Each fix in that space created the next gap. Thread the identity instead. `createRun` accepts a `conversationId` and passes it to the run hook, which the agents controller supplies at both call sites — the same value it already uses as the checkpointer's `thread_id`. The field is optional and the hook is env-gated, so nothing changes when the harness is not in use. Binding then collapses to ownership. A fixture is owned by the conversation that claimed it, and its cursor is authoritative wherever it stands: an extra turn reaches the over-consumption guard rather than falling through to the scripted fake model, and a resumed run keeps replaying with no prompt to match. A different conversation may claim the fixture only by opening it, which rewinds — what a Playwright retry looks like. Everything else is refused, so an unrelated conversation can no longer continue someone else's partly consumed script by repeating a later prompt. The prompt is still re-checked on every real turn; only a resume, which structurally carries no human message, is exempt. The previous text-and-history rules remain as a fallback when identity is absent. The recorder keys the same way: a new attempt is a new conversation, so a resume no longer truncates the fixture mid-turn and discards its tool-call invocation. Record summarization too. The summary provider runs on its own model with its own callback list, so a scenario crossing the context-pruning threshold recorded the agent's invocations but not the summariser's, leaving a fixture that could not reproduce the pruned context. * 🧾 fix: Harden Record Mode and Make the Rendered-Text Assertion Honest CI caught what local runs had not: the committed fixture was never replayed locally, because the record run overwrote it after the replay check rather than before. Re-recording and replaying in that order is what surfaced the rest of this. The DOM assertion compared raw recorded text against rendered markdown. The previous answer opened with `52.`, which Markdown renders as an ordered-list marker, so those characters never appear in the DOM and the match failed on all three CI attempts while replay itself was correct. The closing prompt now asks for prose beginning with a word, a leading enumerator is stripped before matching, and only a prose prefix is compared. Derived configs discarded the record-mode restriction. `config.redis.ts` and `config.mermaid.ts` spread this config and then replace `testMatch`, so `e2e:mock:redis` in record mode would still send its specs to the paid provider. A restriction expressed as an overridable value cannot hold, so record mode now refuses any config but the mock one. Superseded recording callbacks could write across a reset. A failed attempt with a provider call still in flight keeps its handler on the old graph; after the retry reset, that call would allocate an invocation from the new counter or append an `error` entry with a cleared mapping. Handlers now carry the recording generation they were installed for and ignore everything from an older one, and attachment dedupes against the current generation so a graph carried across a restart is not left with an inert handler. * 🚧 fix: Make Summarization an Explicit Boundary, Not a Half-Feature Recording summarization invocations without replaying them is worse than ignoring them. Replay routes the agent model and subagents only, so a recorded summarization entry takes a slot in the fixture sequence that replay never consumes, and the next primary call reads the summariser's chunks — a prompt mismatch or, worse, silently wrong content. The attachment was also aimed at the wrong shape: the SDK reads `summarizationConfig.parameters`, not `.parameters` nested under `.config`, so the previous attempt would have attached to nothing in a real run. Its test passed only because the test built the shape the code expected rather than the shape the SDK provides. Rather than ship a half-routed feature, recording now fails the moment summarization runs, naming the reason. Both shapes are guarded so the guard cannot miss the way the recorder did. Summarization fixtures need replay routing for that model before they can be supported. The derived-config guard added alongside it was itself broken: workers do not carry `--config`, and the argument lookup fell through to `process.argv[0]`, so every recording run aborted claiming the node binary was an unexpected config. The flag is now located explicitly and absence is treated as "not the process that parsed the CLI". * 🔒 fix: Close the -c Config Alias and Pin the Recorder's Fixture Name Playwright documents `-c` as an alias for `--config`, so record mode launched as `playwright test -c e2e/playwright.config.redis.ts` slipped past a guard that recognised only the long spelling. Both spellings and both `=` and space forms are now parsed. Accepting arbitrary fixture names also worked against the ambiguity check. This spec drives one fixed prompt pair, so recording under another name left two fixtures sharing those prompts; replay then refused to bind either and the keyless lane stopped working — a successful documented recording run could disable the suite it exists to serve. The spec now records only the fixture it owns and says so when asked for another. * 🔧 test: Record a Real Tool-Call Turn and Replay It Through the Tool Node The fixture format carried `tool_call_chunks` and the binding advanced through a turn's invocations, but nothing had recorded a real tool-calling conversation end to end — the path was covered only by hand-written synthetic fixtures, and it is the first one a new scenario would exercise. This records one: the provider calls the `remember_fact` MCP tool, the tool runs, and the model is invoked a second time with its result. That is the shape a single prompt cannot express — one user turn spanning several model invocations, all sharing one prompt — so it is what proves the turn-vs-invocation distinction the binding rules were built around. Replay drives the real tool node rather than replaying its output, so the tool executes again and the assertion checks its live result. Two fixtures now coexist, which the record path had to grow for: the config keeps an allowlist so an unknown name is still refused, record mode collects every replay spec, and each spec records only the fixture it owns and stands down for the others. MCP tools reach the model under a server-qualified name (`remember_fact_mcp_e2e-memory`); that qualification has changed before, so the assertions match the base name as a prefix rather than pinning the suffix. Verified: record 1 passed (15.6s, real API) then replay 1 passed (13.6s) against that fixture, ledger drained 2/2 invocations and 35/35 chunks; both replay specs together 2 passed; app-load, completion, chat and mcp-ephemeral 12 passed. |
||
|
|
67b7b441b2
|
🛂 feat: Filter Model-Bound Content by Source (#14425)
* feat: introduce optional content protection seam * feat: enforce source-aware content filters * feat: complete source-aware content enforcement * test: activate skill file-text fail-close fixtures * fix: harden source-aware content filters * fix: harden model-bound content filtering * fix: preserve legacy filters and generated files * fix: inspect shared scalar metadata * test: align mocks with current dev dependencies * feat: add persisted content filter safeguards * feat: complete source-aware content filter enforcement * fix: move resume content preflight into TypeScript * fix: close content inspection edge cases * fix: harden content protection boundaries * fix: complete content protection safeguards * test: align persisted memory filter coverage * fix: reconcile content protection with current dev * fix: reconcile content protection with latest dev * fix: close content protection review gaps * fix: enforce source-aware provider boundaries * fix: preserve legacy PII preflight semantics * test: stabilize stored branch preflight fixture * fix: defer agent writes until protected model admission * perf: harden source-aware model-bound filtering * fix: canonicalize provider lineage before validation * fix: satisfy model-bound callback type checks * perf: Bound content protection filtering work * fix: Bound submission array traversal * fix: Stabilize bounded content snapshots * fix: Scope model-bound traversal overflows * fix: Preserve scoped content inspection * fix: Accumulate aggregate traversal scopes * fix: centralize content policy boundaries * test: align deferred tool policy context * test: align controller policy mocks * style: normalize content protection imports * fix: close content policy review gaps * fix: narrow active skill policy config * fix: address content protection review boundaries * fix: retain exact provenance overflow sentinel * fix: preserve literal and scoped provenance updates * fix: narrow persisted edit provenance * fix: isolate exact overflow attribution * fix: centralize stored prompt protection * fix: fail closed on incomplete transcript evidence * fix: align canonical transcript routing * refactor: centralize content policy preflights * fix: isolate upload policy error typing * style: sort policy preflight imports * refactor: centralize content policy boundaries |
||
|
|
c5276fc63d
|
⏱️ feat: Run Scheduled Chats Through Durable Agent Triggers (#14939)
* feat: Scheduled Chats — agent-centric scheduled runs creating real conversations
Squash of the full review-hardened branch (PR 14540, supersedes 14373) onto
latest dev, preserving the exact verified tree. History prior to this commit
lived on the pre-squash branch; every invariant below survived 25 Codex review
rounds plus two external audit rounds (R26) with regression tests that fail
without their fixes.
Feature:
- Schedules CRUD + side-panel UI (cadence dialog, run cards, Run Now), roles/
permissions (SCHEDULES:USE), interface.schedules availability, per-user limits
and capacity slots, timezone-aware cadence with DST-conservative floors and
misfire grace.
- Engine: single-process claim/fire loop with leases, loopback POST dispatch
(signed schedule-fire JWT claims, per-occurrence idempotency key inside the
route's clientRequestId charset), overlap/balance/capacity/duplicate skip
policies with auto-disable streaks (too_many_failures, insufficient_balance),
reconciliation from retained terminal-job evidence, erasure sweep.
- Scheduled runs create real conversations through the resumable agents chat
path: HITL pauses surface on the card (requires_action), resumes re-apply the
fire boundary's admission policy (revision fence, enabled, global kill switch,
SCHEDULES:USE, availability) before continuing a billed generation.
Correctness invariants (the audit surface):
- Settlement discipline: every persistence-producing write happens-before a
run's terminal outcome write; Stop/complete/pause race through single-winner
terminal CAS claims (dev's TerminalJobClaim substrate) with retained,
completedAt-less evidence for scheduled fires plus an owner-intended outcome
stamp (scheduleOutcome) the reconciler prefers over re-derived success —
round-tripped through the Redis hash mapper.
- Swallowed generation failures (client error content parts) classify to
error/skipped_balance instead of success on both initial and resumed paths;
stale stamps are refreshed evidence-first when persistence plus the Mongo
outcome write both fail.
- Abort honesty: delivery judged by generation ownership and the CAS's actual
from-status; republication escalates to the transport's acknowledged variant;
the Stop route settles only genuinely paused runs.
- Account deletion: one-way barrier (deletionRequestedAt) with auth-cache
tombstone-before-stamp, boundary rechecks across all auth strategies, quiesce
of scheduled + interactive work with durable per-stream abort fences
(positive-evidence acknowledgement only), owner-side finalization markers for
post-terminal billed writes, deferred-deletion sweep (explicitly ensured
partial index) that completes cascades autonomously. Remote OpenAI-compatible/
Responses requests are documented as outside the quiesce and tracked in issue
14594.
- Store compatibility: finalization markers optional on the legacy IJobStore
contract with coherent degradation (registration fails -> synchronous title
fallback; count reads 0).
* fix: fence terminal response persistence from deletion; retain stale-pause evidence
Two blockers from the third external review round.
Terminal persistence visible to account deletion:
- The finalization marker was registered only when post-terminal TITLE work was
possible, but every persistence-owning terminal CAS opens the same window: the
claim drops the job out of the active set BEFORE the response save (and the
background user-message/convo saves), so a deletion quiesce landing there saw
neither an active job nor a marker and could cascade while the admitted request
could still recreate messages. Both controllers now register the marker before
every persistence-owning claim — the fresh-turn path and the HITL resume — and
release it only after their pending saves (and any post-terminal title) have
landed, on success and failure paths alike. The TTL bounds a crash.
- settleAbortFence no longer clears a complete/error fence while
`terminalPersistencePending` is set: terminal at the CAS is not settled while
the owner is still persisting. The job facade now surfaces the flag.
- The marker trio is REQUIRED by the runtime store contract (assertJobStoreV2
refuses a store without it at configure time, keeping the failure loud and
deterministic) while remaining optional on the legacy public IJobStore type for
source compatibility. The silent degrade path from the previous round is gone —
it was not deletion-safe.
Stale-pause recovery retains scheduled evidence:
- Three crash/timeout recovery paths — ApprovalLifecycle.failStalePausePersistence
and the InMemory/Redis stale-pause cleanups — unconditionally stamped
`completedAt`, putting a scheduled fire's failed-pause error terminal on the
short completed TTL. A Mongo outage longer than that TTL erased the evidence and
the reconciler recovered the run as `interrupted` instead of `error`. All three
now follow the controller-observed path from the previous round: scheduled jobs
omit `completedAt` (retained-evidence TTL) and stamp the error outcome.
The PR description now explicitly narrows the deletion guarantee for the remote
OpenAI-compatible/Responses paths (tracked in issue 14594).
* fix: deletion-fence marker protocol — generation-scoped, atomic, fail-closed, all terminal paths
One consolidated pass over the finalization-marker mechanism, per the fourth
external review round. The invariant it establishes: NO persistence-owning
terminal CAS runs without a durable, generation-qualified marker covering the
window it opens, and every consumer treats a pending terminal as unsettled.
- Generation-scoped markers. Entries were keyed (userId, streamId), so a
Stop-superseded generation finishing late could clear the marker its
replacement registered on the same conversation. Marker fields are now
qualified by the generation's createdAt; clears must present the same
identity, and an unqualified legacy clear cannot drop a qualified entry.
- Atomic Redis registration. HSET-then-EXPIRE loses the fresh marker when the
user's existing hash expires between the two commands (or the process dies
there); registration is now a single Lua script carrying both.
- Fail closed everywhere. Registration failure (after one retry) now REFUSES
the terminal CAS instead of proceeding uncovered: the completion claim throws
into the error path, the error path skips completeJob and leaves the job
ACTIVE — deletion-visible by itself, recovered by the stale-running reaper —
and abortJob returns a new retryable `fence_unavailable` failure the Stop
route answers with 503 and the deletion quiesce treats as an unacknowledged
stop (fence kept). The previous round's log-and-proceed is gone.
- Every terminal path enrolled. abortJob now owns its window (register before
the abort CAS, clear in its finally — the Stop route's checkpoint prune and
partial save run inside beforePublish, between CAS and publication); the
interactive and background generation-error paths register before their
completeJob; the resume controller's error finalization registers before its
completeJob. A lost or thrown claim releases the marker after pending saves
flush instead of holding the user's deletion behind the TTL.
- Every pending terminal unsettled. settleAbortFence defers on
terminalPersistencePending for ALL statuses — including `aborted`, whose
route-side persistence the previous guard missed.
Also: a direct Redis regression for scheduled stale-pause retention (the P2
test gap), and the PR description no longer claims to carry every commit.
* fix: lease-token lifecycle fences — same-generation isolation, admission fence, undelivered-Stop retention, legacy abort enrollment
Fifth external review round; four P1s, handled as the requested consolidated
lifecycle-fence pass.
- Lease tokens. Marker fields were (streamId, createdAt), shared by every
contender on the same generation — completion and Stop, or two racing Stops —
so a losing contender's cleanup erased the winner's still-live marker. Every
registrant now carries a unique lease token in the field and may only ever
clear its own lease; unqualified legacy clears cannot touch qualified entries.
- Admission fence. Authentication can pass before the deletion barrier goes up,
and the durable createJob is several async steps later — a deletion quiesce in
that window saw neither an active job nor a marker and could cascade before
the admitted request created its job. The controller now registers an
admission lease and THEN rereads the deletion barrier: the ordering guarantees
either this request observes the barrier (403, lease released, slot/claim
cleanup) or the quiesce observes the lease and defers. Held until createJob is
durable; released on every refusal and initialization-error path. Fail closed
when the lease itself cannot be registered (503 retryable).
- Undelivered-Stop retention. abortJob released its lease in a finally even when
delivery AND publication had provably failed — the job reads terminal
(invisible to active-set scans), a user Stop writes no durable abort fence,
and the remote owner keeps generating and will persist its abort-catch writes
whenever the signal finally lands. The lease is now retained in exactly that
case, and each resignal attempt heartbeats a fresh lease so the fence outlives
the TTL for as long as delivery is still being driven. The abort-winning
turn's own loser-side pending saves are additionally fenced in the controller
catch (best-effort — those writes are already in flight).
- Legacy abort enrollment. abortMiddleware (assistants abort route fallback for
non-assistants endpoints) awaited abortJob and then spent usage and saved the
stopped response AFTER the abort's lease was released. Both writes now run
inside `beforePublish`, between the abort CAS and publication, covered by the
same lease as every other abort.
Barrier tests, each verified to fail without its fix: same-generation lease
isolation (store), racing two-Stop loser cleanup (manager, stale-read forced
CAS race), undelivered-Stop lease retention, resignal heartbeat, admission
refusal with lease-before-reread ordering plus release-on-durable-create, and
legacy-abort persistence inside beforePublish.
* fix: heartbeat-backed owner-lifecycle leases close the settlement handoff races
Sixth external review round: the remaining P1 interleavings were one structural
problem — lease handoffs that were not atomic — resolved as the requested
consolidated lifecycle-lease pass.
- Quiesce reads leases BEFORE the active-job scan. The admission-lease -> durable
-job handoff is only atomic against a reader in the OPPOSITE order of the
writer: writers hold the lease strictly until the job is active-set visible,
so leases-first shows every interleaving either the lease or the job.
Jobs-first allowed a request to create its job after the scan and release its
lease before the count — hiding both, cascading, and letting the new
generation persist into a deleted account.
- The abort acknowledgement is fenced by the owner-lifecycle lease. Redis ACKed
the moment the owner's AbortController tripped; the stopping side released its
lease on that ACK while the owner's asynchronous abort-catch persistence was
still ahead. The transport now awaits a manager-installed pre-ACK hook that
registers a DETERMINISTIC owner lease (exactly one owner exists per
generation, and determinism is what lets the signal-time registrant and the
owner's catch-side release agree across processes) before the acknowledgement
is persisted or published; an owned same-replica abort bridges to the same
lease before tripping its local controller. Both generation-owner catches
(fresh turn, resume) release it once their writes land.
- A failed replacement handoff no longer orphans the predecessor. The atomic
replacement removes it from active storage, and an unconfirmed handoff
terminalizes the replacement too — leaving nothing a quiesce could discover
while the predecessor's provider may still be generating. Its owner lease is
now retained at the point the receipt fails delivery; the owner replica renews
it through the pre-ACK fence when the signal finally lands.
- Leases HEARTBEAT while held. The five-minute store TTL only bounds a crashed
holder; live persistence — a stalled save, a long deferred title — must never
outlive its own fence. holdUserFinalization registers and renews every minute
until released; the controllers' completion/error/admission leases all hold.
The undelivered-Stop retention moved to a deterministic `stop` lease that
every resignal attempt renews and the first successful one clears (no more
opaque leases accumulating to TTL), and a THROWN abort transition releases the
contender lease instead of leaking it.
- The user-document abort-fence mutations now invalidate the auth user-doc
cache, matching every other user-doc write.
Barrier tests, each verified to fail without its fix: quiesce lease-scan
ordering (plus the observed-lease defer), pre-ACK fence ordering at the
transport, owned-abort owner-lease bridging, replacement-handoff predecessor
retention, held-lease heartbeat past the TTL, and failed-then-successful
resignal reaping the retained stop lease.
* fix: one manager-owned owner-lease span across every abort delivery path
Seventh external review round; four lifecycle-fence gaps, closed by making the
owner-lifecycle lease a single manager-owned, heartbeat-held span.
- Fail-closed acknowledgements. The pre-ACK hook registered a one-shot lease and
the transport ACKed even when it failed; a same-replica owned abort likewise
swallowed registration failure. The hook now acquires a HELD owner lease
(heartbeat until the owner's catch releases it via releaseOwnerLease) and a
rejection SUPPRESSES the acknowledgement — the stopping side stays retryable
behind its retention lease, and every resignal re-drives the handler. The hook
also stopped gating on `job.createdAt === generationId`: during a replacement
handoff the store holds the replacement while the abort targets the
predecessor, and that gate silently skipped exactly the generation being
acknowledged (the store job is owner identity, never a generation gate). A
local owned abort acquires the same held lease before tripping its provider;
post-CAS the trip cannot be withheld, so acquisition failure downgrades
delivery and the retention handoff keeps the user fenced.
- Committed-but-lost-reply disambiguation. A thrown abort transition released
the contender lease as if nothing had happened, but a Lua CAS can commit and
lose its reply — an aborted job invisible to active-set scans whose provider
was never signalled, with no fence left. The throw path now re-reads the exact
generation: only a job still live under the caller's identity proves no
commit; committed or ambiguous outcomes hand the fence to the deterministic
stop lease (kept on the contender lease if even that fails) before rethrowing.
- Replacement handoff covered end to end. A LOCAL replacement abort acquires the
predecessor's held owner lease before the trip (failure reports the receipt
undelivered, engaging retention). Failed-handoff retention is no longer a
swallowed one-shot: it heartbeats with the durable acknowledgement proof as
its renewal predicate — acquisition failures keep retrying for as long as the
fence is needed, and the retainer stands down (without clearing the shared
field) once the remote owner ACKs and thereby holds its own lease.
- A LOCAL resignal delivery hands off to the owner lease BEFORE clearing the
retained stop lease, and keeps the stop lease when that handoff fails.
Barrier tests: hook rejection suppressing the ACK, commit-then-lost-reply
retention with its provably-uncommitted counterpart, local-resignal owner
handoff ordering, pre-ACK owner lease held past the store TTL until release
(and provably stopped after), and failed-handoff retention retrying on its
heartbeat — verified fail-before/pass-after by stashing the fixes.
* fix: finish scheduled chat lifecycle hardening
* fix: close scheduled chat review follow-ups
* fix: generation-fence abort recovery evidence
* test: wait for settled approval tool output
* test: preserve scheduled init reconciliation option
* refactor: rebuild scheduled chats on durable agent triggers
* test: reset MCP cache mock between cases
* test: isolate scheduler startup in server specs
* fix: harden scheduled run lifecycle
* fix: normalize schedule capacity conflicts
* test: type schedule collision fixture
* fix: re-fence scheduled resume and expiry
* fix: fence scheduled resume handoffs
* fix: release superseded manual schedule leases
* fix: release failed run-now claims
* fix: release superseded engine claims
* fix: repair schedule dialog interaction and rework its form
The agent picker was unusable: ControlCombobox portals its popover to the
body by default, which lands it outside the dialog's Radix focus trap. Clicks
passed through it, it could not be tabbed into, and the trap fighting Ariakit
for focus locked the page up on selection. The prop is documented for exactly
this case — pass `portal={false}` and give the dialog `overflow-visible`, as
ProjectButton already does. The time and day dropdowns defaulted the same way.
Alongside that:
- Extract the agent builder's instructions editor (special-variable menu plus
expand-to-fullscreen) into a controlled `VariableEditor` and use it for the
schedule prompt. Insertions now route through `onChange`, so react-hook-form
sees them — the schedule PATCH is built from `dirtyFields`, and a `setValue`
that skipped dirty tracking would have dropped an inserted variable silently.
- Replace the hand-rolled frequency buttons with the shared `Radio`. They marked
the selection with `bg-surface-hover` on an outline button whose hover is the
same token, so the selected option was indistinguishable from a hovered one;
`Radio` is also a real radiogroup rather than four `aria-pressed` toggles.
- Wrap the fields in a real `<form>` and associate the footer button by id, so
Enter submits. Group the frequency, day and time controls in fieldsets.
- Add placeholders for name and prompt, match the textarea fill to the other
fields, and label the hourly case as minutes past the hour.
- Widen the dialog to `md:max-w-3xl` and pair name with agent so the form fits
without scrolling on desktop.
- Move scheduled chats below skills and above prompts in the side nav.
The new dialog spec fails when the portal fix is reverted.
* test: cover scheduled and subagent deletion drains
* refactor: own the form-control appearance in the client primitives
Addresses the codex finding on ScheduleDialog: a feature-local `FIELD_CLASS`
restated the `Input` primitive's border, radius, height and background so it
could be pasted onto the schedule dropdowns, leaving those controls with no
connection to the primitive they were imitating.
Move that appearance into `packages/client/src/components/Field.ts` as the
single source `Input` and `Textarea` now compose, and give `Dropdown` and
`ControlCombobox` a `variant="field"` that applies it. The schedule dialog
passes the variant and carries no class strings of its own.
This also repairs a break the dev merge would otherwise have introduced: the
newer `Dropdown` splits `className` (wrapper) from `triggerClassName`, so the
old pasted classes would have landed on the wrapper and left the triggers
unstyled.
The semantic-token guard now watches the shared module and asserts each
primitive still composes it, which covers more than the two files it read
before.
* fix: keep an explicit schedules disable from becoming an opt-in
`use` is two things at once for a dual-purpose runtime interface field: a
permission bit, which DB overrides strip, and the runtime disable signal that
`getLimits` reads. Stripping it from `{ use: false, maxPerUser: 2 }` leaves an
object, and `getLimits` treats any object without `use: false` as enabled — so
an admin override written to stop scheduled billing for a role or user started
it instead.
Collapse an explicit disable to the boolean form before the strip, on both
paths that accept it: the `interface.schedules` field patch, which admitted the
object wholesale because bare runtime paths deliberately bypass the permission
gate, and the overrides merge, which reached the composite-field branch and
kept `maxPerUser`. Objects that only narrow limits are untouched, so a
principal can still be given a smaller cap.
Both regressions fail without the normalizer.
* fix(schedules): Wave A — null-balance CAS, atomic paused-card clear, clustered erasure sweep
Slice 1 (thread r3804518381): route the existing-null balance initialization
through a { user, tokenCredits: null } compare-and-set instead of a blind $set,
so a concurrent initializer/charge landing between the preflight read and the
write is never handed back its spent starting balance. On a CAS miss the
preflight re-reads the winner. The absent-record $setOnInsert path and the
credited-record refill-config sync are unchanged. Adds the initializeNullBalance
adapter (no upsert) and regression coverage for winner/miss/sync cases.
Slice 2 (thread r3804518388): updateScheduleById now drops a `requires_action`
lastRun projection atomically with the configRevision bump. Any pause present at
edit time was projected under the pre-edit revision and can never be replaced by
its own revision-fenced terminal outcome — a disabling edit would strand the
card on "Needs approval" forever. Implemented as classic-operator CAS branches
(DocumentDB rules out a conditional pipeline $unset), fenced on the card STILL
being the pause so a terminal outcome or newer occurrence that races in is
preserved. Terminal history survives untouched.
Slice 4 (thread r3803826204): expose initializeScheduleErasureSweep from the
schedule runtime facade and start it in every clustered (experimental) worker
after Mongo is up. It re-drives eraseScheduleIfDrained for soft-deleted rows so
a hidden prompt cannot outlive its drain when the delete/erase-on-settle
attempts miss. It arms nothing else and never infers owner death from a
process-local missing job (isTopologySafeToArm gates that).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq
* fix(schedules): Wave B.1 — reversible account-deletion schedule suspension
Thread r3804518383. Account-deletion quiesce marked every schedule `deleting`,
disabled it, and cleared nextRunAt destructively. When a later cascade step (or
the drain itself) failed, the controller cancelled the user-deletion fence —
restoring the user — but the schedules stayed `deleting` and were erased by the
sweep, silently losing all of a live user's scheduled prompts.
Replace the destructive marking with a REVERSIBLE, token-fenced suspension:
- suspendUserSchedulesForDeletion(userId, token) snapshots each schedule's prior
enabled/nextRunAt under a per-attempt token, then fences firing (disable, clear
nextRunAt, rotate claimToken). It never sets `deleting`, so a suspended row is
not erasure-eligible. Snapshotting reads then bulkWrites (a classic update
cannot copy field values under DocumentDB), fenced per row so an already-
suspended/soft-deleted/edited row is left alone; idempotent per token.
- restoreUserSchedulesFromDeletion(userId, token) reverses it, re-enabling and
re-arming only rows still carrying the exact attempt token and not independently
deleted — so an owner-deleted or newer-attempt-suspended schedule is never
resurrected.
- deleteUserController generates the attempt token, passes it to quiesce, and on
any failure that cancels the user-deletion fence restores the suspended rows. A
successful deletion hard-deletes them (and their snapshots) via the existing
cascade and never restores.
Adds a `deletionSuspension` embedded field (excluded from the wire schedule),
data-method regression tests (suspend/restore/fence/idempotency/no-resurrect),
and controller tests (restore on drain-false and post-quiesce cascade failure,
no restore on success).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq
* fix(schedules): Wave B.2 — wire the interactive Stop persistence protocol
Thread r3804255932. The durable Stop primitives (requestRunAbort 'stop',
getScheduleRunAbortState, markRunAbortPersisted) already existed but production
only used requestRunAbort(..., 'deletion'). An interactive Stop flipped the job
to `aborted` and then persisted its partial message + checkpoint inside
`beforePublish`, without ever stamping the schedule Stop or acknowledging it —
so reconciliation, the generation owner, or a concurrent schedule/account
deletion could terminalize the run and release its capacity (and erase data)
mid-write.
Wire the request -> persist -> acknowledge -> settle barrier through the
schedule runtime (the route never touches raw Mongo):
- Expose beginScheduledStop / acknowledgeScheduledStopPersistence on the service.
- The abort route stamps the Stop BEFORE signalling abortJob (a serialized
'in_progress' loser returns 409 STOP_IN_PROGRESS without a second abort),
acknowledges only after beforePublish persistence succeeds, and on a
persistence failure leaves the barrier unresolved so the run stays preserved
(client retries; stale-owner timeout is the bounded recovery). A failed abort
releases the stamp it placed so a replacement/retry is never blocked through
its predecessor.
- recordScheduleOutcome (the owner settlement path) now waits, bounded, for the
Stop acknowledgement before terminalizing; a resolved/non-stop/stale marker
proceeds immediately. The paused Stop settles only after its own ack.
Adds service-layer barrier tests and route-level ordering/persistence-failure/
in-progress tests; the data-layer serialization is already covered.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq
* fix(schedules): Wave C.1 — reconcile durable trigger delivery with the reservation
Threads r3803826192 (manual limiter) and r3804255924 (PII/moderation), plus the
independently-found long-Retry-After race. fireSchedule reserves a `started` run
and a global capacity slot BEFORE the durable trigger delivery reaches the chat
route, where an interactive limiter (manual Run Now), PII, or moderation can
reject it before any generation job exists — dead-lettering the delivery while
the run sat `started` until the 30-minute orphan sweep mislabeled it interrupted.
And a valid delivery deferred by Retry-After (up to 24h) could be orphan-settled
and have its capacity released, then fire anyway.
Translate durable delivery state into the schedule outcome:
- Store the deterministic trigger deliveryKey on the ScheduleRun reservation
(computed from the envelope BEFORE enqueue, so an ambiguous commit still has it).
- Add a getTriggerDelivery engine dep (wired to the merged trigger service's
getDelivery) that reads the durable delivery by key.
- Schedule reconciliation, for a jobless `started` run: staging/pending/leased →
admission is live, never orphan; dead → record `error` from the durable
lastError and release capacity promptly (no 30-minute wait), through the
ordinary outcome/auto-disable path; succeeded or no record → the existing
legacy orphan policy (interrupted only past the cutoff); a delivery lookup
failure defers rather than orphaning a possibly-live delivery. Limiter/PII/
moderation middleware writes no schedule state.
Adds reconcile state-mapping tests (dead/pending/leased/staging/succeeded/none/
lookup-failure) and a fire test that the reservation's deliveryKey equals the
enqueued delivery's idempotency key.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq
* fix(stream): Wave C.2 — durable retry/ack for terminal host lifecycle actions
Thread r3804518375. Approval expiry won the `requires_action → aborted` CAS and
then invoked the host hook best-effort: `runApprovalExpiredHandler` swallowed a
failure, and because later sweeps enumerate only `requires_action` jobs, the now-
aborted job was never offered again. In the clustered entrypoint (no schedule
reconciler) the ScheduleRun stayed `requires_action` and its retained job
persisted indefinitely.
Make the host lifecycle work durable rather than schedule-specific:
- Add a generic `terminalHostActionPending` marker, set ATOMICALLY in the same
terminal transition (ApprovalLifecycle.expireWithIdentity), only when a host
adapter is installed.
- Retain and index such jobs: both stores keep them out of terminal reaping and
expose getTerminalHostActionJobs(); Redis adds a set + extended (24h-bounded)
TTL, in-memory a bounded 24h retention so a permanently-failing hook cannot leak.
- The manager clears the marker only after the adapter acknowledges success,
fenced by generation identity (clearTerminalHostAction), so a replacement
generation can neither clear nor execute its predecessor's action.
- cleanup()/expireStaleApprovals() enumerates unacknowledged terminal host actions
across restarts and replicas and retries the idempotent hook; the relay only
re-invokes while the marker is unacknowledged, so a successful ack prevents
duplicate work. Store-won expiry marks it too, so a loser-replica relay still
crosses the hook.
- Terminal SSE notification continues regardless of host-hook outcome.
Covers in-memory behavior (retry after failure, restart/other-replica retry, ack
prevents duplicates, identity fence, terminal notification on failure, no marker
accumulation for non-scheduled jobs) and updates the Redis cluster-membership
contract test for the new index.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq
* style(schedules): satisfy import sorting in fire.ts and fire.spec.ts
CI "Static checks" failed on IMPORT_SORT for the two files Wave C.1 added imports
to (the AgentTriggerEnvelope type import and getAgentTriggerIdempotencyKey).
Applied scripts/sort-imports.mts to exactly those files — imports-only reordering,
no behavior change. Other files reported by a repo-wide check are pre-existing on
dev and deliberately left untouched so this PR is not widened.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq
* fix(schedules): repair the deletion CLI and order restore before the fence release
Addresses three findings from the fresh Codex review.
P1 — config/delete-user.js called methods.disableUserSchedulesForDeletion, which
Wave B removed in favor of suspendUserSchedulesForDeletion. The file is
@ts-nocheck and its spec mocked the removed name, so neither typecheck nor tests
caught it; the real CLI would throw a TypeError before deleting anything and then
only unwind the fence. The CLI now uses the tokenized protocol: it mints a
suspension token, suspends with it, and restores that exact attempt's rows in its
finally block when the deletion does not commit. Its spec mocks the real methods,
so the breakage can no longer hide.
P2 — both the HTTP controller and the CLI released the user-deletion fence BEFORE
restoring schedules. That fence is what refuses new schedule writes/claims, so the
gap let an owner PATCH edit a still-suspended row and have its enabled/next-run
state overwritten by the older snapshot, and let a second deletion attempt
re-suspend under a new token — making the first restore a no-op and stranding the
disabled snapshot permanently. Restore now runs first, while writes are still
fenced.
Hardening for the same defect class across a crash: suspendUserSchedulesForDeletion
now ADOPTS an existing suspension's snapshot when re-suspending a row abandoned by
an earlier attempt, instead of re-capturing the row's current (already-suspended)
state. Without this, an attempt that died before restoring would have its
successor snapshot "disabled, no next run" and permanently strand the schedule.
Tests: CLI restore-before-fence ordering and no-restore-on-success; the same
ordering assertion on both controller post-quiesce failure paths; a data-method
regression that a second attempt adopts the abandoned snapshot and restores the
original enabled/next-run state.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq
* fix(schedules): converge dead deliveries in topology-safe maintenance
Codex finding: the `dead` delivery mapping added in Wave C.1 lives only inside
startScheduleEngine's reconciler, but the clustered entrypoint arms no engine — it
runs erasure-only maintenance. A delivery queued before a restart into clustered
mode and then rejected before generation creation (interactive limiter, PII,
moderation) dead-letters while its ScheduleRun stays `started`, holding a global
capacity slot indefinitely for an ordinary non-deleting schedule.
Add a dead-delivery convergence pass to the erasure sweep, so every topology that
runs schedule maintenance settles it. The pass is POSITIVE-EVIDENCE-ONLY and is
therefore safe where absence-based reconciliation is not: a `dead` delivery is
durable shared state proving no generation owns the reservation. It settles only
when the job is confirmed absent or identity-mismatched (an identity-matched job
still owns the run), defers on an unknown job lookup, on an in-flight abort, and
on an in-flight resume hand-off, ignores legacy reservations with no deliveryKey,
and applies a short grace so an accepted delivery still creating its generation is
never settled mid-handoff. Auto-disable policy is deliberately left to the armed
engine; this path records the failure and frees the slot.
Deliberately does NOT touch api/server/experimental.js — the clustered entrypoint
already starts this sweep, so the convergence arrives through the existing
initializer and the shared-file footprint stays as-is.
Tests: settles a dead delivery as error under an explicitly UNSAFE topology,
leaves live deliveries alone, never settles under an identity-matched running
generation (delivery is not even consulted), defers an in-flight abort, and
ignores a reservation with no deliveryKey.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq
* fix(stream): refresh host-action retention on each retry attempt
Codex finding: unacknowledged terminal host-action evidence was capped at the
24h pause TTL, so a host dependency (Mongo) unreachable for longer than that let
the Redis key — and its pending marker — expire with no generation-fenced
acknowledgement, stranding the ScheduleRun where no reconciler is armed.
Measure retention from the LAST retry rather than from the terminal transition:
enumerating a pending host action IS the retry attempt, so both stores refresh
its retention as they hand it to the hook (Redis re-EXPIREs the job key; the
in-memory store stamps terminalHostActionRefreshedAt and bounds from it). Evidence
therefore survives as long as some replica is still actively retrying, while a
deployment that stops sweeping entirely still lets it age out — so this does not
reintroduce the unbounded leak the cap existed to prevent.
Test: after a failed hook, a later cleanup pass keeps the marker pending and moves
its retention basis forward.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq
* fix(schedules): defer settlement when the Stop barrier times out
Codex finding: waitForStopPersistence returned after its 5s poll budget even when
the Stop was still fresh and unacknowledged, and recordScheduleOutcome then went
straight on to terminalize the run — releasing its capacity, deletion, and erasure
barriers while beforePublish may still have been writing. Slow checkpoint cleanup
is indistinguishable from a dead route on that signal, so the timeout was being
treated as if the barrier had been satisfied.
The poll budget now means "undecided", not "clear". On timeout with a fresh,
unacknowledged Stop the barrier DEFERS: recordScheduleOutcome returns false
without recording, leaving the run active/preserved. Settlement then happens
either when the route acknowledges, or once the existing stale-owner cutoff
(ABORT_OWNER_PRESUMED_ALIVE_MS) authorizes a later attempt — which the loop
already treats as clear-to-settle. Callers with durable retry (the approval-expiry
host action, reconciliation) re-drive it, so a deferral converges rather than
stranding the run.
Test: a fresh Stop that never acknowledges within the budget reports not-settled
and records no outcome.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq
* fix(schedules): require definite delivery failure, extract its message, converge deferred Stops
Third Codex round. Three findings in code from this closeout, plus one pre-existing
P1 that is a one-line operator-safety fix.
P1 — dead-delivery settlement demanded too little evidence. `dead` does not prove a
request was rejected: the trigger host marks response timeouts and invalid success
responses `certainty: 'ambiguous'`, and the engine dead-letters those once retries
are exhausted. The erasure sweep treated every dead letter as positive evidence, so
an ambiguous one sitting over a generation a peer had accepted could terminalize the
run and release its capacity mid-flight. It now settles only on a DEFINITE rejection,
unless job absence is deployment-authoritative (safe topology), where the
confirmed-absent job is itself the evidence.
P1 — `lastError` is an `AgentTriggerDeliveryFailure` object, not a string. A
duplicated local interface declared it `string` (against CLAUDE.md's no-duplicate-
types rule), so both the sweep and the engine reconciler passed the object into the
String-typed run/schedule `error` fields; Mongoose would reject the cast, the per-row
catch would swallow it, and the run would keep its global capacity slot. The dep type
now reuses the canonical `AgentTriggerDeliveryFailure` and both call sites pass
`.message`. Re-typing immediately surfaced a stale test that had asserted a string.
P1 (pre-existing) — the base-config global stop is honored in `getLimits` via
`isRuntimeDisabled` rather than a literal `=== false`. The stop has two shapes, and
deepMerge turns base `{ use: false }` plus a principal override of `true` into
`{ use: true }`, so the literal check reported the feature enabled and Run Now
dispatched straight through fireSchedule, bypassing the operator's emergency stop.
Now the same predicate the engine gate already uses.
P2 — a Stop whose settlement DEFERRED past the poll budget had no convergence path
where no reconciler is armed. `acknowledgeScheduledStopPersistence` now optionally
re-drives the terminal outcome once the barrier clears; `recordRunOutcome` is
match-guarded and idempotent, so an owner that already settled makes it a no-op. The
abort route passes it for a running generation; a paused job still settles explicitly.
Tests: ambiguous dead letters refused under unsafe topology but settled when absence
is authoritative, definite rejections settled either way, the failure message carried
through, and the abort route's re-drive present for running / absent for paused.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZwZPDBgxy4mCWWg6zQSjq
* fix(schedules): converge terminal runs in clustered workers, retry suspension restore
Fourth Codex round on
|
||
|
|
f829aca9fb
|
🧩 fix: Align Tenant and MCP Configuration Resolution (#14904)
* fix: Align Tenant and MCP Configuration Resolution * fix: Preserve Operator-Owned MCP Entries * fix: Preserve Configuration Source Ownership * style: Normalize Middleware Import Order * fix: Preserve Process Server Precedence * test: Align Tenant-Aware E2E Setup |
||
|
|
1bccc2bc18
|
📡 fix: Refresh MCP Tools After List-Changed Notifications (#14686)
* fix(mcp): handle dynamic tool list changes Co-authored-by: Pascal Garber <pascal@artandcode.studio> * test(mcp): fix CI validation * fix(mcp): keep dynamic tool catalogs live * fix(mcp): harden dynamic catalog lifecycle * test(mcp): use typed startup connection * test(mcp): isolate dynamic e2e fixtures * fix(mcp): refresh tools after reconnect * fix(mcp): close dynamic catalog cache gaps * test(mcp): update OAuth connection mocks * fix(mcp): preserve app snapshot ownership * style(mcp): sort connection imports * fix(mcp): close review race conditions * fix(mcp): preserve cache ownership edges * fix(mcp): harden recovery lifecycle * fix(mcp): guard tool-less app refresh * fix(mcp): fence distributed cache races * fix(mcp): retire stale connection state * fix(mcp): keep tool snapshots authoritative * fix(mcp): fence stale app tool publications * style(mcp): sort repository test imports * test(mcp): mock empty startup publication * fix(mcp): preserve app publication generations * fix(mcp): harden publication recovery races * fix(mcp): address tool catalogs by runtime config * fix(mcp): load scoped catalogs for assistant writes * fix(mcp): harden catalog publication recovery * fix(mcp): serialize forced connection replacement * fix(mcp): serialize ordinary creation with replacements * fix(mcp): harden catalog fallback boundaries * fix(mcp): close lifecycle fencing gaps * fix(mcp): preserve catalog authority on failures * fix(mcp): compensate failed catalog mutations * fix(mcp): fence catalog refresh ordering * style(mcp): sort agent loader imports * fix(mcp): cancel stale connection creation * fix(mcp): fence catalog coordination * fix(mcp): close catalog race windows * fix(mcp): harden cross-pod catalog fencing * fix(mcp): close catalog lifecycle edges * style(mcp): sort assistant imports * fix(mcp): reject stale recovery authority * fix(mcp): restore static catalog on every startup * fix(mcp): order app catalog publications * style(mcp): sort catalog revision imports * fix(mcp): separate catalog allocation and commit fences --------- Co-authored-by: Pascal Garber <pascal@artandcode.studio> |
||
|
|
52b2ebf948
|
🧪 test: Run mock E2E against Redis in shards (#14551)
* 🧪 test: Run mock E2E against Redis in shards * 🧪 test: Isolate local Redis E2E data |
||
|
|
7b6900d556
|
🏷️ feat: Activity Groups With Fast-Model Headers (#14391)
* ✨ feat: Activity Groups with Fast-Model Labels Groups each contiguous block of reasoning + tool calls into a collapsible unit headed by a fast-model label (claude.ai-style hierarchy), off the critical path: a PostToolBatch hook claims a live content slot at the batch boundary (steering index-offset pattern), renders a deterministic counts phrase instantly, and swaps in the generated label ~1s later while the next model call streams. Labels are UI-only — stripped before the SDK formatter and skipped in the legacy formatter — and reach live clients via a dedicated on_activity_label SSE event (live/replay/pending paths). Grouping preserves legacy rendering byte-for-byte when no label part is present. Generation bridges to Run.generateActivityLabel() when the SDK ships it (session-grouped Langfuse tracing); falls back to a direct call today. Env-gated: ACTIVITY_LABELS_POC=true, ACTIVITY_LABEL_MODEL. * 🧷 fix: Address Codex and Copilot Review Findings for Activity Labels - Settle in-flight label fills (bounded 3s) before finalization on both the main and resume paths, so a label resolving during the final batch still reaches the durable log and saved message. - Overlay on_activity_label chunks in RedisJobStore content reconstruction (splice path last-wins per index; replay path chronological overwrite), matching steer handling. - Wire activity labels into the HITL resume createRun so post-resume batches keep claiming slots. - Guard against out-of-order publishes: fill() awaits the claim emit before emitting the resolved label, and the client applier ignores a stale pending placeholder once a resolved label is present. - Stamp the batch's groupId onto label parts so parallel-column runs place them inside their group instead of filtering them out. - Localize the counts fallback phrase (10 keys, singular/plural) through useLocalize across chat rendering and exports. - Type the hook with Providers/ClientOptions instead of stringly types; drop the unknown cast in the spec; add a dedicated rAF retry ref for label events with effect cleanup. * 🛡️ fix: Address Independent Review — Abort, Usage, Lane Context, Redis Test - Propagate the run abort signal into label generation (both wiring call sites; runtime combines host + dispatch signals with the timeout) so a user abort cancels in-flight label calls instead of paying to timeout. - Record label-call usage like titles: the SDK bridge aggregates via chainOptions callbacks, the fallback path via a per-generation callback factory; both feed recordCollectedUsage under context 'activity-label'. - Scope block-context capture: reasoning collection stops at the previous block's label part and filters by executingAgentId, so consecutive or parallel batches can no longer bleed another block's thinking into the payload; intent text still scans past labels (persists across batches). - Forward the effective charLimit to the SDK call so host and SDK prompts agree (SDK default aligned to 600 in agents#327). - Add a Redis integration test proving last-write-wins reconstruction of on_activity_label chunks per claimed index. - Rebased onto main: only the two activity commits replay (the nine steering commits belonged to the old base branch), zero conflicts, steering suites green. * 📐 refactor: Move Activity-Label Wiring to TypeScript, Address Codex Round 2 - [P1] Slot claiming, lane stamping, emit ordering, context capture, and settle tracking now live in packages/api (createActivityLabelWiring + captureActivityBlockContext); client.js is a thin closure wrapper. - Register the activity-label hook BEFORE the steer drain so a steer draining at the same batch boundary cannot flush the tool block and orphan the label outside its group. - Resolve request-based header placeholders in resolveActivityLabelLLM (titleConvo parity) so metadata-keyed proxies work on label calls. - Trim labels centrally before filling so whitespace-only output from either generation path keeps the deterministic counts fallback. * 🧭 fix: Codex Round 3 — Capture Order, Shared Strip, Token Estimator, Hide Filter - Capture block context BEFORE pushing the label part: the scan stops at ACTIVITY_LABEL parts, so post-push capture hit the just-inserted label and silently collected no reasoning excerpts (regression test added). - Share stripActivityLabelParts from packages/api and apply it in the Responses and OpenAI-compatible controllers, closing the replay leak for entry points still running SDKs without the formatter skip. - Exclude activity_label parts from the fallback response-token estimator (UI-only parts must not inflate no-usage provider billing). - Keep label parts explicitly under hide_sequential_outputs — they summarize exactly the outputs that mode hides. * 🔁 fix: Codex Round 4 — Resume Gap, Delta Flush, Agent-Scoped Intent, Token Counter - Synthesize on_activity_label events for labels claimed or filled in the snapshot→subscribe window (the publish is fire-and-forget, so Redis-mode reconnects missed them). Feature-gated so the default path adds no content re-read; the client applier already ignores duplicates. - Flush queued deltas before applying a label part, matching the pending- action and steer appliers — without it the handler read a stale message cache and syncStepMessage pushed a pre-delta copy back. - Skip another agent's tail text when resolving intent, so parallel runs cannot seed a label prompt with a sibling agent's narration. - Exclude activity_label parts from countFormattedMessageTokens (the agent-path counter), not just the legacy BaseClient one. * 🏗️ refactor: Codex Round 5 — Extract Label Host Logic, Report Usage, Icon Strip - Move provider/model resolution, usage-metadata mapping, and the settle loop into packages/api (activityLabels/host.ts); client.js keeps only thin delegations, per the repo's TypeScript-implementation convention. - Fold label usage into the response rollup with an 'activity-label' tag (subagent precedent) so metadata.usage and the live cost gauge account for it; tagged, so it stays out of PRIMARY usage/context pairing. - Narrow tool metadata once in ToolCallGroup so THINK parts in a labeled block no longer render phantom generic icons in the stacked strip. - Import the activity-label helpers by deep path in GenerationJobManager: the package barrel now reaches provider-config/cache modules that import back into the stream layer, and the cycle broke suite loading. Declined: resetting steerOffsetState before HITL resume — resume builds a FRESH AgentClient via initializeClient (initialize.js:978), so the offset is already zero; the seed wrapper alone accounts for pre-pause parts. * 🚦 fix: Codex Round 6 — Stream Label Usage, Close Late Fills - Emit an on_token_usage chunk for label calls (sink push alone left the live session gauge blind); retained in pendingSubagentEmits so job cleanup cannot race the persist, tagged 'activity-label' as before. - Close the label scope when settle times out: the wiring gates fill() on isClosed and the client fires a label-scoped AbortController, so a straggling generation can neither mutate a saved response nor emit into a job whose runtime is gone. The controller also chains to the run signal, so a user abort still cancels label work. * 🩹 fix: Repair CI — Package Typecheck and Module Mocks Local runs covered the client tsconfig and jest, but never packages/api's own tsconfig, so nine type errors in the extracted host module shipped. - Type host.ts against the real contracts: ServerRequest, EndpointDbMethods, AppConfig from @librechat/data-schemas, IUser for createSafeUser, and a MaybeAzureConfig view for the azure instance-name probe and configuration. - Widen resolveConfigHeaders' llmConfig to Partial<RunLLMConfig>: it only reads the three provider header carriers, so auxiliary generations with a bare ClientOptions can resolve headers without assembling a run config. Type-only widening; every existing caller still satisfies it. - Add stripActivityLabelParts to the @librechat/api mock in the OpenAI and Responses controller specs — those mocks enumerate exports, so a new import read as undefined and threw before the assertions ran. - Use the real activity-label helpers in the ToolCallGroup spec's ~/utils mock; stubbing them out would hide the header logic under test. * ⚙️ feat: Configure Activity Labels via librechat.yaml, Drop Env Vars Replaces the ACTIVITY_LABELS_POC / ACTIVITY_LABEL_MODEL env gate with per-endpoint settings, following the title options convention rather than a top-level block — each endpoint picks its own cheap label model. - Add activity, activityModel, activityEndpoint, activityPrompt, activityMaxPerRun, and activityCharLimit to the endpoint schema, and to the endpoints.all pick list (enumerated, so 'all:' would otherwise drop them silently). - resolveActivityConfig reads them with title-style precedence: endpoints.all > named endpoint > custom endpoint config. - Model precedence is now activityModel > titleModel > the agent's model. activityEndpoint runs labels on another endpoint's credentials, with titleConvo's fallback-on-unknown-name behavior. - Thread activityPrompt/MaxPerRun/CharLimit through the wiring into the hook and the SDK bridge; they were hardcoded defaults. - The resume gap-repair gate keyed on the env var; it now keys on the snapshot actually containing label parts, so deployments without the feature still perform no extra content read. - Document the fields in librechat.example.yaml; add host.spec.ts covering precedence, custom-endpoint fallback, and opt-out. * 📝 refactor: Rename Enable Flag to activityLabel, Document Schema Inheritance - Rename the boolean from `activity` to `activityLabel`, matching the titleConvo/titleModel shape: a verb-object toggle whose prefix matches its modifiers (activityModel, activityPrompt, ...). `activity: true` alone read ambiguously — it could mean tracking or logging activity. - Document the two endpoint-schema inheritance paths, which behave oppositely and are ~900 lines apart: * `endpoints.all` omits from baseEndpointSchema, so new options are inherited automatically — nothing to maintain. * `azureEndpointSchema` enumerates via .pick(), so a new option is silently unavailable on Azure endpoints until listed there. The activity block now carries a pointer to the Azure caveat. * 🔍 fix: Address Codex Findings on the Config Rework - Pass the matched custom-endpoint config into the label gate. Custom endpoints live in the `endpoints.custom` ARRAY, so without it every custom endpoint resolved as disabled — including the example this PR added to librechat.example.yaml. - Give label usage a unique `runId:seq`. Label usage is billed but never appended to `collectedUsage`, so its length was static: every label event reused the last primary usage's pair and collided with itself, and the client dedupes on exactly that. - Attach `cost` to label usage when `interface.contextCost` is on; aggregateEmittedUsage treats coverage as all-or-nothing, so an event without it suppressed the whole response's cost. - Honor `activityPrompt` on the direct fallback path, not just the SDK bridge — it previously always used the built-in instruction. - Seed the per-response label cap from labels already on the response so a HITL resume cannot mint a fresh quota after every approval. - Reconcile label gaps on resume via a durable per-job `activityLabels` flag instead of probing the snapshot: the FIRST label of a run can be claimed inside the snapshot->subscribe window, which the old signal missed. The flag is read from a job record already fetched there, so runs without the feature still add no content read. - Auto-collapse labeled single-tool groups; one-call batches are common in agent runs and rendering them expanded defeats the grouping. * 🎯 fix: Correct Label Usage Seq, Cross-Endpoint Pricing, Close Scopes - Give label usage a NEGATIVE seq namespace. The previous fix was wrong: seq is a position in `collectedUsage` (push, then emit with the new length), so sink-length + array-length still lands on a real position — primary emits 1, the label computes 2, the next primary also emits 2. Labels have no position at all (billed separately, never appended), so they now occupy a namespace positional sequences cannot reach. The client key is a string used for Set membership, so the sign is inert. - Price cross-endpoint labels with the LABEL endpoint's token config: resolveActivityLabelModel now returns the resolved endpointTokenConfig, and both the streamed cost and recordCollectedUsage use it instead of the agent endpoint's rates. - Make close state per-wiring rather than per-client. A HITL resume rebuilds the wiring, and resetting a shared flag re-opened closures from the pre-pause segment whose provider call ignored the abort; settle now closes every retained scope, past generations included. * 🎯 fix: Make the Activity Header Say Something the Cards Cannot The header read "ran 1 command" next to a card already labeled "Code" — it restated the UI beneath it instead of adding to it. Two causes, both about content rather than timing: - A deterministic tool-type tally was the primary display and also fed the prompt, so the best case was a tally and the worst case was a tally dressed as prose. Removed from the metadata, the prompt, the part type, and the client. - The instruction only ever reached the fallback path. The wiring passed a prompt only when was configured, so the preferred SDK path silently used the published package default. The wiring now always supplies one and the hook forwards it on both paths. The register is rewritten around what the cards cannot show: past-tense git-commit-subject, leading with the distinctive noun, outcome over attempt, tool names and counts and arguments explicitly forbidden. The batch entries are labeled as reference material so the model stops transcribing them. Claiming a slot no longer emits. The slot still reserves its index so streamed parts never collide, but with nothing to say there is nothing to render: until a description exists the block looks exactly as it does without the feature. * 🧹 fix: Drop the Localize Hook Left Unused by the Counts Removal * ✅ test: Add Activity-Label e2e Coverage with a Recording Label Server Activity labels are the one model call a mock run does not already fake: fake-model.js swaps the GRAPH model via overrideTestModel, while run.generateActivityLabel() calls the endpoint resolved client options over HTTP. The custom endpoints already point baseURL at 127.0.0.1:8889, so serving that port exercises the real path with no production seam. fake-label-server.js answers it in both JSON and SSE form, records each prompt, and can inject blank/error responses. Recording is what lets the spec assert the CONTRACT rather than the rendering: that this repo register and the tool OUTPUTS actually reach the model. That is the bug class that produced unusable labels before, and rendered text looks identical whether or not the instruction arrived. Labels get a dedicated endpoint (Mock Provider E). A labeled block auto-collapses even at one tool call, which hides the tool cards other specs assert on -- enabling this on a shared endpoint broke steering.spec.ts. Provider D is the unlabeled control. Request-count assertions are scoped to a per-test token: a 5xx label response is retried by the provider client, and a retry can land after the next test has reset the server. * 🩹 fix: Address Review Findings on Activity-Label Indexing and Pricing Replay index (P1). Reserving the slot only in server memory left no event for it, so a cross-instance replay rebuilt content as [tool, hole, later], compacted the hole away, and the fill for the reserved index landed on the following part and overwrote it. The claim now publishes the empty, pending part so the index is real for every consumer, and fill publishes even when generation returned nothing so the client cannot stay pending. It stays invisible: an empty label still DELIMITS its batch in groupSequentialToolCalls but is not attached as the header, so grouping does not re-shuffle when the text lands and the block renders exactly as it does with the feature off. Edited-response index (P1). Edit-and-resubmit replays the kept prefix and the server indexes only new content, so run steps offset by that prefix. Labels are claimed in the same space and now take the identical shift; without it a label could land inside the prefix and overwrite it. Redis flag. deserializeJob never read activityLabels back, so every Redis reload left it undefined and resume skipped label gap reconciliation. Executing agent. RunActivityLabelOptions.agentId selects the executing agent tracing metadata AND its tool-output redaction policy; omitting it let a handoff be redacted under the default agent configuration. Label pricing. An undefined endpointTokenConfig is meaningful for a built-in label endpoint (priced from the shared table), so the nullish fallback billed those labels at a custom primary rates. Inherit only when the label runs on the agent own endpoint. HITL usage sequence. runId is the response message id and the counter was instance-local, so a resume restarted at -1 and the client runId:seq deduper discarded the post-approval label usage. Seeded past the labels already on the response. Also distinguishes "cannot serve" (undefined) from "no label" (null) in the SDK bridge, so a missing run falls back to the direct call instead of filling the slot empty. Version gating already happens at wiring time via the sdkCapable prototype probe. * 🩹 fix: Keep Unfilled Activity Labels Invisible and Unmask Endpoint Settings Follow-up review round. Publishing the reservation on every batch made two latent rendering paths reachable on every run, and both are fixed here. Empty labels no longer change grouping. The previous pass still formed a tool-group for a textless label, which wrapped even a single tool call and pulled THINK parts inside it — and since a reservation is published the moment each batch ends, that applied during every generation and permanently after a blank or failed fill. An empty label now flushes the legacy way instead: it still delimits its batch, but the block re-splits exactly as it renders with the feature off. Parallel lanes no longer show a blank line. Lanes render raw parts, so an unfilled label had nothing to draw; empty ones are dropped. Making labels act as collapsible headers inside lanes is still a separate gap. Edited responses no longer offset on resume. The sync replaces initialResponse.content with the server's aggregatedContent, which already contains the kept prefix AND everything generated since — so its length is not the prefix length, and indices reconciled from that snapshot are already absolute. Offsetting again pushed the label past its slot onto a later part. The shift now applies only to a fresh edited submission. Activity settings resolve field by field. Selecting one config object whole meant any endpoints.all block — even one carrying nothing but headers — shadowed the named or custom endpoint and silently disabled activity labels everywhere. Global still wins per field. Adds groupToolCalls coverage for the invisible-while-empty contract, which is the part most likely to regress: it is normal state on every run, not an edge case. * 🔒 fix: Scope Detached Label Writes to Their Generation Epoch Epoch scoping (P1). Label generation is detached and can outlive the generation that started it. emitChunk only proves that SOME runtime is current, not that the caller belongs to it, so an aborted generation's fill(null) -- and its usage event -- could be attributed to whichever generation replaced it, landing an index from the abandoned response on top of the new one. Because an empty label renders nothing, that overwrote content silently. emitChunk now takes an optional jobCreatedAt and drops the event when the runtime epoch differs, mirroring the existing setGraph/setContentParts convention, and both label emitters pass it. An abort now CLOSES the label scope instead of only cancelling the call: the rejected generation still runs its catch and calls fill(null), which would otherwise emit into a stream the next generation may already own. Edited-response indexing (P1). The previous pass skipped the prefix offset on resume, which was the wrong half of the problem: a sync replaces initialResponse.content with the server's aggregatedContent, which is completion-local, so after a reconnect its length is not the kept-prefix length and the offset is wrong -- but it is wrong for run steps in exactly the same way. Tool cards and the label that heads them must share one index space; a label shifting differently from its tools lands on another part. The label path now uses the identical expression as useStepHandler, with no resume special-case. Correcting the post-resume prefix length belongs in calculateContentIndex, where it fixes both at once. titleModel masking. The activity settings were made per-field last pass, but the titleModel fallback a few lines below still selected an entire config object, so a partial endpoints.all (for example one carrying only headers) hid a named endpoint's titleModel and quietly fell the label back to the main agent model. Both now read through one shared per-field helper. Resume reconciliation no longer depends solely on markActivityLabels, which is best-effort yet had come to gate correctness: a lost flag write silently dropped a label. The snapshot is consulted as a fallback. The exported host type for generateLabel now admits undefined, which is the documented "cannot serve, fall back to the direct call" signal the hook keys on -- distinct from null, meaning it ran and produced nothing. * 🧷 fix: Keep Group Identity Stable and Memoize Label Endpoint Resolution Group remount. Tool-group identity was keyed on the first part in the block. An activity label absorbs the block's leading THINK part the moment its text lands, so the key flipped from tool:<id> to fallback:<scope>:<idx> mid-run, remounting the group and discarding whatever the user had expanded. The key now scans for the first tool call, which does not move when the block re-forms. Label endpoint resolution is memoized per response. It reads provider config and can hit the database for user keys, yet nothing it depends on changes between batches of one run — and it ran twice per batch, once for generation and once for usage accounting. The promise is cached rather than the value so concurrent batches share a single in-flight resolution, and a rejection is evicted so one transient credential failure cannot disable labels for the rest of the response. * 🎯 fix: Offset Edited Resubmissions by a Prefix Length That Survives Resume The server indexes only NEW content for an edited resubmission, so the client offsets incoming indices by the prefix it retained. That prefix was read as initialResponse.content.length, which is correct only until a resume: the sync replaces that array with the server's completion-local snapshot, whose length is unrelated to the prefix. After a reconnect every offset was therefore wrong -- run steps and activity labels alike -- and could write over content the edit kept. For a label the symptom is worse than a bad position: the fill misses its own reservation, so the pending placeholder is never resolved. The prefix length is now captured when the submission is built, while initialResponse.content still IS the retained prefix, and carried on the submission as editPrefixLength. calculateContentIndex takes that length instead of deriving it from an array that a resume may have replaced, so run steps and labels share one index space by construction rather than by both happening to read the same field. Note the prefix is the FULL original content with the edited part substituted in place (useChatFunctions clones latestMessage.content and mutates one entry) -- it is not a slice, so the length cannot be inferred from editedContent.index. Group identity no longer changes when a label fills. Tool-group keys were derived from the first part in the block; an activity label absorbs the leading THINK part when its text lands, flipping the key mid-run and remounting the group, which discarded the user's expansion state. The key now scans for the first tool call, which does not move. Label endpoint resolution is memoized per response. It reads provider config and can hit the database for user keys, yet ran twice per batch -- once to generate, once for usage accounting -- while nothing it depends on changes within a run. The promise is cached so concurrent batches share one in-flight resolution, and rejections are evicted so a transient credential failure cannot disable labels for the rest of the response. The resume gap passes for steers and activity labels now share a single lazy content read instead of each issuing its own. The label pass stays gated on the run flag with a snapshot fallback: reconciling unconditionally would also close the residual first-label window, but it would bill a read to every resume of every run, including deployments with the feature off -- which the steer pass deliberately avoids. That residual requires a lost flag write, which shares fate with the content writes the labels live in. * 💵 fix: Bill Cross-Endpoint Labels at Their Own Rates recordCollectedUsage never accepted an endpointTokenConfig, so the value the activity-label caller passed was dropped and the balance transaction was written at the primary agent's rates. Only the UI cost honored the label endpoint, so a custom primary pointing activityEndpoint at another endpoint showed one price and charged another. The parameter is now accepted, and an explicit config wins outright over per-agent resolution: that map is keyed by AGENT, so it cannot describe usage that ran on a different endpoint. Group identity is stable for id-less tool calls too. The previous pass anchored the key to the first tool call ID; where a supported tool call carries no id the fallback still used the block's first part index, which shifts when a filled label absorbs the leading THINK part. The fallback now anchors to the first TOOL entry's index, so only a block containing no tool call at all keys off parts[0]. markActivityLabels is retried rather than fire-and-forget. It gates resume gap reconciliation and is a SEPARATE write from the durable label append, so a single lost write silently drops a label the content itself recorded. The earlier "shared fate with content writes" reasoning was wrong. One retry at run setup costs nothing and removes the only realistic way the gate goes stale, without billing a content read to every resume. * 🧮 fix: Stop Offsetting Once SYNC Drops the Edited Prefix The edit offset was applied unconditionally, but whether it is correct depends on which branch SYNC took. SYNC either preserves the content already loaded for the response -- which still contains the retained prefix, so the offset is required -- or replaces it with the server's aggregatedContent, which is completion-local and indexed from zero, after which any offset writes past the end of a now shorter array. That is why the two previous attempts each fixed half of it: skipping the offset on resume was right for the replace branch, applying it unconditionally was right for the preserve branch, and neither holds on its own. The offset now tracks the actual state of the rendered content. For an activity label the replace branch was worse than a bad position: the fill landed past its own reservation, so the pending placeholder was never resolved and the block kept its generic header for the rest of the run. Applied to run steps as well, not just labels. useStepHandler reads the prefix from the same submission and had the same unconditional offset, so after a mid-session resume of an edited response tool cards were misplaced too. Normalizing at the dispatch boundary keeps both in ONE index space by construction: a label that shifted differently from the tools it heads would land on another part. Note the reload path was already coherent -- useResumeOnLoad rebuilds the submission without editedContent or editPrefixLength, giving no offset against server-supplied content -- so only the mid-session SYNC path was inconsistent. * 🧾 fix: Keep Label Accounting Out of the Primary Usage Slot Label usage no longer owns getStreamUsage(). recordCollectedUsage assigned its result to this.usage unconditionally, so when the primary provider reported no usage metadata but the label provider did, BaseClient took the label's output tokens as the assistant response's authoritative count. The later primary call returns early on an empty collectedUsage and never replaced it, so the wrong value stood, the text-based fallback was skipped, and the real generation went unbilled. Secondary usage is still billed but no longer writes that slot. Cross-endpoint pricing keys off an explicit discriminator rather than the presence of a value. A built-in label endpoint prices from the shared table, so an undefined endpointTokenConfig is its MEANINGFUL value -- reading that as "no override" fell back to the primary's custom rates and restored the exact mismatch the previous pass set out to fix. The caller already knows whether the label ran elsewhere and now says so. markActivityLabels rejects on failure instead of swallowing it. The flag gates resume gap reconciliation and the caller retries it, but the internal catch resolved successfully and made that retry unreachable -- so the two changes cancelled out and a transient write failure still left the flag absent. Late label accounting is suppressed with the same gate as the late fill. A straggler that outlived the settle timeout still ran its finally block, so it charged the balance and appended to usageEmitSink after the response had passed its usage flush and metadata snapshot: a cost the user pays but is never shown. The cleared-prefix state is scoped to one generation. It was set on a resume SYNC that replaced the response and then never reset, so a later edited resubmission in the same mounted hook dispatched run steps and labels with no offset against content that still held its retained prefix. Reconnects pass isResume and keep the state; a new generation clears it. * 🔑 fix: Key Prefix State to the Stream and Honor current_model for Labels The cleared-prefix reset keyed on isResume, which skips exactly the case it was added for: a submission whose POST succeeded server-side but lost its response is retried, comes back resumed: true, and subscribes in resume mode even though it is a NEW generation. A previous generation's cleared state then survived into it, and incoming run steps and labels applied no offset against content that still held its retained prefix. The state is now keyed to the stream id, which changes with the generation and stays put across reconnects of one. activityModel now honors current_model. The options are documented as title-shaped and the titleModel fallback already excludes the sentinel, but the higher-precedence activity override passed the literal string through to getOptions and the provider, so an endpoint following that convention failed every label instead of using the agent model. * 🎯 fix: Key Prefix State to the Generation and Resolve the Run Model The cleared-prefix state was keyed to the stream id, which never changes within a conversation: request.js sets streamId = conversationId, so once a reconnect cleared the state every later edited resubmission in that conversation dispatched run steps and labels with no offset and could overwrite the prefix it retained. It is now keyed to the response message id, the only per-generation identity available here -- minted per submission and carried through a resume unchanged. That is the third identity tried for this state. isResume missed the deduplicated-retry path (a lost response returns resumed: true for a new generation); the stream id is conversation-scoped. The response id is the boundary that actually matches a generation. current_model labels now resolve the model the run is really using. initializeAgent merges the request's endpointOption override into model_parameters and the run gives it precedence, so preferring the saved agent.model could send labels to a different, potentially unavailable or more expensive model than the conversation is on. * 🆔 fix: Key Prefix State to the Submission and Keep the Origin Title Model Editing an assistant response reuses that response's messageId as editedMessageId, and useChatFunctions carries it onto initialResponse.messageId -- so re-editing the same response produced two generations with the same key and the cleared-prefix state survived between them, leaving run steps and labels with no offset against content the edit retained. Keyed now to clientRequestId, the per-submission uuid, which is minted fresh per edit attempt and forwarded unchanged on retries. That is the fourth key this state has had, and each earlier one failed at a real boundary: isResume missed the deduplicated-retry path, the stream id is the conversation id, and the response message id is reused across edits of one response. clientRequestId is the identity that actually means "this submission". The titleModel fallback is read from the ORIGINATING endpoint again, matching how titleConvo captures its config before switching credentials. Reading it after an activityEndpoint switch meant an OpenAI endpoint configured with titleModel claude-haiku and activityEndpoint anthropic fell through to the OpenAI run model and sent that name to Anthropic, failing every label. The destination endpoint supplies credentials, not the model choice. * 🧷 fix: Close the Remaining Edit, Epoch, and Scope Gaps for Labels SYNC clears the edit prefix on the new-row branch too. When a resumed edited submission cannot match an existing assistant row, that branch builds the response straight from the server's completion-local aggregatedContent, so it holds no retained prefix -- but the reset lived only in the matched branch, leaving later steps and labels adding an offset to indices that were already absolute. Label usage is keyed per GENERATION. Editing one assistant response reuses its responseMessageId while each fresh generation restarts activityLabelUsageSeq, so a second edit re-emitted the same runId:seq and the client discarded the newer usage while its balance transaction was still written. The key now carries jobCreatedAt, the run's own epoch: stable across reconnects and HITL resumes, distinct between generations. The scope is revalidated at commit time. Checking once before the await let a scope that closed mid-flight still charge the balance after finalization, while the matching fill saw the closed scope and dropped the label -- billed but never surfaced, the exact outcome the guard exists to prevent. The titleModel fallback no longer reaches the destination endpoint. With activityEndpoint set and no titleModel on the originating endpoint, it picked up the destination's, so changing only the credential target silently changed the model and its cost. Precedence is activityModel, then the originating endpoint's titleModel, then the run model; the destination supplies credentials only. * ✂️ refactor: Confine the Edit-Prefix Offset to Activity Labels useStepHandler is now byte-identical to dev again. The resume-aware prefix offset was applied there too, which was more correct in principle -- the post-resume prefix length is genuinely wrong for run steps as well -- but it changed index math that EVERY run step flows through, for every user, including everyone who never enables activityLabel. That shared correction needed five revisions in two days (isResume, the stream id, the response message id, clientRequestId, and the SYNC new-row branch), each passing the full suite and each failing at a boundary only review found. Carrying it inside an opt-in feature put every user behind logic with that track record. It belongs in its own change, with tests that construct the edit-plus-resume states none of the current suites reach. The offset now applies only where the label handler places its part, so this PR cannot alter rendering for anyone with the feature off. The known consequence is recorded in the description: with activity labels ENABLED, an edited response that reconnects mid-generation can place its label and its tool cards in different index spaces. That is a bug for opt-in users rather than a regression for everyone, and it disappears once the shared fix lands. submission.editPrefixLength stays: the label path still needs a prefix length that survives a SYNC replacing initialResponse.content. * 🧾 fix: Commit Labels Before Billing and Keep Blank Slots Invisible Round-nine review (all P2, feature-scoped): - Billing ordering (client.js:409, runtime.ts): usage accounting ran BEFORE the slot commit on both generation paths, so the settlement deadline could expire during the balance write — charged, then the fill dropped as out-of-scope: billed, never shown. `slot.fill` now resolves a commit flag, generators register their accounting via `deferUsage`, and the hook runs it only after a committed fill. - Scope gates (client.js:757): the direct-fallback `collect` omitted `scopeOpen`; both paths now gate on the OWNING wiring's scope, so a pre-pause straggler cannot bill because the resumed generation's scope is still open. - Blank-label grouping (groupToolCalls.ts:81): a blank slot forced a flush, splitting adjacent single-call batches into standalone cards where the feature-off path merges them. Blank labels now only mark the claim boundary — structurally invisible, while a later filled label still cannot claim an earlier batch. - Stale fill indices (wiring.ts:301): the skill-card unshift and the hide-sequential filter reshape contentParts before the finalization settle, so an in-flight fill emitted its claim-time index against a shifted array. Both completion paths now settle label fills before any post-run content reshaping (the finally settle stays as the error-path net; the second call sees an empty pending list). - Bounded serialization (runtime.ts:238): `JSON.stringify` fully materialized unbounded tool results to keep 200/600 chars per entry. A budget-bounded serializer stops at the limit (which also bounds cyclic values) and preserves the exact truncate-with-ellipsis output. Tests: fill/bill ordering + suppression on dropped fills (runtime.spec), blank-slot merging and claim boundaries (groupToolCalls.test), bounded serialization equivalence and giant-output truncation (runtime.spec). * 🧮 fix: Keep Deferred Label Billing Inside the Settle Window Self-review follow-up to the billing reorder: deferring usage until after the commit moved it PAST the fill's resolution, so a settle keyed on fills alone could let finalization flush the usage sink and snapshot metadata while the label's billing was still in flight — the usage row would silently miss the message rollup even on the happy path. The hook now reports its whole detached task (generate → fill → deferred usage) via a `trackTask` option, wired to the same settle tracker as the fills, so finalization waits for billing exactly as it did when accounting preceded the fill. The task never rejects. Pinned in runtime.spec: the tracked task resolves only after usage collection. * 🧰 fix: Harden Label Resolution, Output Bounds, and Cache Billing Round-ten review (all P2, feature-scoped); the sixth finding is the documented edited+reconnect index-space limitation, answered on-thread as deliberately out of scope for this PR. - Rejected-LLM memoization (runtime.ts): the hook cached a rejected `resolveLLM()` promise permanently, failing every later batch and silently defeating the host resolver's own rejected-cache eviction. The memo now evicts on rejection so the next batch retries. - `current_model` precedence (host.ts): an explicit `activityModel: current_model` resolved to `undefined` and then lost to a configured `titleModel`. The sentinel now resolves straight to the run model; the title fallback applies only when `activityModel` is absent. - Output bounds (runtime.ts): label text was persisted verbatim; a model ignoring the 4–9-word instruction (or steered by injection in untrusted tool output) could emit thousands of tokens duplicated through SSE, the chunk log, persistence, and the UI. `normalizeLabelOutput` keeps the first non-empty line, collapses whitespace, and hard-caps at 200 chars on both generation paths. - Cache-token billing (host.ts, client.js): the usage mapper dropped cache fields, vanishing Anthropic cache tokens from billing and charging OpenAI cache reads at the full input rate. The mapper now normalizes Anthropic/OpenAI/LangChain cache shapes into `input_token_details`, and the emit + cost path carries them with the label endpoint's `provider` (additive-provider adjustment). - Usage-type union (runs.ts): `TTokenUsageEvent.usage_type` now includes the emitted `activity-label` literal; the lone consumer keys on `usage_type != null`, so this is type-level completion. Tests: sentinel/title/explicit model precedence and all three cache shapes (host.spec), transient-resolution retry and output normalization with truncation (runtime.spec), the new usage literal (runs.spec). * 🪗 fix: Let Settled Labels Collapse Void Tools and Keep the Tail Cursor Round-eleven review (all P2, client-side). Two fixed; the other two findings restate documented Known limitations (edited+reconnect run-step index space; parallel-lane collapsible headers), answered on-thread. - Void-tool auto-collapse (ToolCallGroup.tsx): `allCompleted` keyed solely on output truthiness, so a tool that legitimately returns an empty string kept its labeled group expanded forever. A settled, filled label is itself a completion proof — the PostToolBatch claim only happens after every output in the batch returned — so it now satisfies `allCompleted`; pending labels keep the group live. - Trailing-reservation cursor (ContentParts.tsx): a blank label reservation at the content tail renders nothing but still counted as the last part, stripping the streaming cursor and last-item affordances from the last VISIBLE part until the next delta. `lastContentIdx` now walks back past empty label slots. Tests: labeled void-tool group auto-collapses, pending-label group stays expanded (ToolCallGroup.test). * 💳 fix: Price Label Cache Correctly, Honor endpoints.agents, Cancel Every Retry Round-twelve review: four fixed here; the remaining P1 (move the client.js bridge into packages/api) is an architecture call answered on-thread for the maintainer. - Provider on billed entries (client.js, P1): round ten added cache details to label usage entries but not `provider`, and `splitUsage` treats an unknown provider as additive — re-adding cache_read and cache_creation on top of an input count that already contains them, double-charging Anthropic/OpenAI cached label calls while the streamed cost (which carried the provider) disagreed. Every mapped entry now carries the label endpoint's provider. - endpoints.agents honored (host.ts, client.js): `initializeAgent` rewrites `agent.endpoint` to the backing provider, so activity settings under the PUBLIC `agents` endpoint — valid config, inherited by `agentsEndpointSchema` — were silently ignored. Field resolution is now `all` > public endpoint > backing provider/custom, applied to both the enable gate and the model/titleModel resolution. - E2E_LABEL_PORT reaches the YAML (playwright.config.mock.ts): an overridden port moved the fake label server and its health check but not the generated config's hard-coded 8889 baseURLs, so readiness passed while every label request targeted the wrong port. The override is now substituted into the generated copy. - Every retry frame cancelled (useResumableSSE.ts): concurrent label retry chains (reservation + fill per slot) overwrote one rAF handle, so cleanup cancelled only the newest chain; the rest ran up to 120 frames past unmount and could apply a stale label to a replacement generation reusing the same response id. Outstanding frame ids now live in a Set that cleanup drains. Tests: public-endpoint gate/precedence/all-above-public (host.spec). * 🖱️ fix: Keep the Last-Part Cursor in Parallel Lanes Too Round-thirteen review (single P2): `ParallelContentRenderer` computed `lastContentIdx` from the unfiltered array, so a trailing blank label reservation — filtered out of every lane — left NO rendered part carrying the last-part cursor and running-subagent affordances until the label filled. The sequential renderer's walk-back is extracted into a shared `lastVisibleContentIdx` helper (utils/activityLabels) used by both `ContentParts` and `ParallelContentRenderer`, so the two index spaces cannot drift again. Behavior pinned in activityLabels.spec: trailing blank skipped, consecutive blanks skipped, filled label counts, label-free content unchanged. * 🧹 chore: Alias the Retry-Frame Set for the Effect Cleanup Lint Rule * 📏 fix: Let activityCharLimit Reach Tool Inputs Round-fifteen review: `activityCharLimit` is documented as the per-entry truncation for tool input AND output, but `buildPrompt` hard-coded inputs at 200 characters — so raising the setting could never surface a distinguishing path, query, or operation that appears past the first 200 characters of a long argument. Inputs now truncate at the configured limit alongside outputs; the 200-char constant remains only for the intent line (renamed INTENT_CHAR_LIMIT to match). Config fidelity pinned in runtime.spec: a 400-char argument survives a 450 limit and truncates under a 50 limit. The round's other finding is the fifth restatement of the documented edited+reconnect index-space limitation, answered on-thread with the prior four cross-references. * 🤝 fix: No Labels for Pure Handoff Batches Round-sixteen review: a PostToolBatch containing only `transfer_to_*` calls claimed a label slot, but transfer parts are never groupable — the client flushed the handoff card standalone and the label orphaned into a stray line after it, restating what the card already says. Two-sided fix: - Hook (runtime.ts): a batch whose every entry is a transfer call claims nothing — no slot, no model call, no `maxPerRun` consumption. Mixed batches still label (the header describes the real work). - Renderer (groupToolCalls.ts): an orphan label whose `tool_call_ids` are all transfer calls is dropped instead of rendered standalone, covering content persisted before the hook-side skip. The round's two P1s are repeats answered on-thread: the packages/api extraction (maintainer-decided follow-up, recorded in the description) and the sixth restatement of the edited+reconnect index limitation. Tests: transfer-only batch claims nothing, mixed batch still claims (runtime.spec); transfer-only orphan label dropped, real-batch orphan label still renders (groupToolCalls.test). * 🎛️ fix: Sanitize Label Client Options and Bound the Batch Prompt Round-seventeen review: two fixed; the other two findings repeat the maintainer-decided packages/api extraction (follow-up) and the edited+reconnect index limitation (seventh instance), answered on-thread. - Primary-option strip (host.ts): the label client copied the resolved `llmConfig` wholesale, so an endpoint whose defaults enable extended thinking or carry model-specific output caps forwarded them to the (often cheaper) label model — unsupported options failed every label, and supported thinking spent real tokens and the settlement window on a 4–9 word header. The copy now strips `omitTitleOptions` keys and the `modelKwargs` output caps exactly like the title path, restoring the Anthropic `clientOptions` carrier by reference so proxy `defaultHeaders` still reach label requests. - Batch prompt budget (runtime.ts): per-entry truncation left the batch dimension unbounded — hundreds of parallel calls could build a prompt past the fast model's window. The entries section now has a total budget (8k chars, scaling with `activityCharLimit` so a raised limit still fits several entries); entries past it are skipped without paying their serialization cost, and the list notes how many were omitted. The first entry always renders in full. Tests: option strip with header-carrier survival (host.spec); giant batch bounded with omission marker, small batch untouched (runtime.spec). * 🛡️ fix: Keep SSRF Guards on Label Calls, Skip Mixed Handoff Batches Round-eighteen review: four fixed; the fifth repeats the maintainer-decided packages/api extraction (eighth instance), answered on-thread. - SSRF-safe carrier (host.ts, P1): the sanitize step restored the Anthropic `clientOptions` carrier only when `defaultHeaders` existed, but for user-provided base URLs `getLLMConfig` stores the guarded Undici dispatcher and `redirect: 'error'` there — dropping it reopened DNS-rebinding/redirect paths on label calls to user-controlled URLs. The carrier (client CONSTRUCTION options, not generation params) is now restored whenever present, same reference. - Primary maxTokens (host.ts): top-level `maxTokens` is not in `omitTitleOptions` and survived the strip; the title path deletes it explicitly, and a cap sized for the primary model can be rejected by the substitute. Deleted on the copy. - Bounded keys (runtime.ts): the object branch materialized every key via `Object.keys` and quoted oversized keys in full before the budget check. Enumeration is now lazy (`for..in` + own-property guard) and keys slice to the budget before quoting, like string values. - Mixed handoff batches (runtime.ts, groupToolCalls.ts): the client flushes the block at the transfer card, so a mixed batch's label orphaned exactly like a pure one. The hook now skips ANY batch containing a transfer call, and the renderer drops orphan labels covering one (legacy content). Tests: carrier survival without headers by same reference, maxTokens strip (host.spec); mixed batch claims nothing (runtime.spec); mixed orphan dropped, real-batch orphan kept (groupToolCalls.test). * 🧢 fix: Cap Label Generation, Order the Flag Persist, Detach Settled Listeners Round-nineteen review: three fixed; the fourth is the ninth instance of the edited+reconnect index limitation, answered on-thread. - Generation cap (host.ts): stripping the primary output caps left label calls with NO cap at all — `normalizeLabelOutput` bounds what persists, not what the provider generates and bills, so a model ignoring the 4–9-word instruction (or steered by injected tool output) could emit its provider-default output per batch. The sanitize step now installs a 256-token label cap (per provider family: `maxOutputTokens` for Google-style wrappers, `maxTokens` otherwise), after the filter so the omit set cannot remove it. - Flag-persist ordering (client.js): the `markActivityLabels` write was fire-and-forget, so an immediate cross-replica reconnect could read the job between the write and the first claim, see neither flag nor snapshot label, and skip gap reconciliation. Label emission now awaits the (settled-on-failure) persist chain, making "a label event exists" imply "the flag is durable" — the race window is gone; only the documented double-write-failure residual remains. - Listener detach (client.js): each HITL approval cycle's wiring adds a `once` abort listener to the shared job signal that only an actual abort removes; settled segments now detach theirs in `settleActivityLabels`, so long multi-approval runs cannot accumulate dead closures toward the listener-limit warning. Tests: the primary cap is REPLACED by the 256-token label cap (host.spec). * 🎯 fix: Route the Label Cap Per Model Family Round-twenty review: the 256-token label cap set maxTokens unconditionally, but GPT-5+ rejects max_tokens (the OpenAI builder routes its cap into modelKwargs.max_completion_tokens / max_output_tokens) and o-series models reject it with no stable kwargs alternative — every label on those models would have failed. The cap now mirrors the builder: modelKwargs for GPT-5+ (responses-API aware), no cap for o-series (title parity; the 200-char persistence bound still applies), maxOutputTokens for Google, maxTokens otherwise. Pinned in host.spec for both reasoning families. The round's other finding is the tenth instance of the documented edited+reconnect index limitation, answered on-thread. * ⏱️ fix: Persist the Label Flag at Run Start, Not on the Emit Path Round-twenty-one review: two fixed; the other three repeat the maintainer-decided packages/api extraction, the edited+reconnect index limitation, and the parallel-lane header limitation — all answered on-thread with their standing decisions. - Flag ordering, corrected (client.js): sequencing label emission behind the flag persist (previous round) delayed the claim-time reservation while the shared index offset had ALREADY shifted subsequent SDK chunks — reopening the cross-instance hole-compaction overwrite the reservation emit exists to prevent. The reservation emits immediately again; instead, run start (processStream and resume alike) awaits the settled-on-failure persist chain, so the flag is durable before any batch can claim a label. Same guarantee, zero latency on the emit path. - Tail-label cursor (ContentParts.tsx): a filled label at the content tail is consumed into the group header rather than listed in `group.parts`, so the `isLast` check missed it and nothing held the streaming cursor until the next delta. The check now includes `labelPart.idx`. * 🔌 fix: Detach Label Abort Listeners Even Without Claims A segment with labels enabled can end without a single claim (text-only, or handoff batches, which skip labels); the early return in settleActivityLabels skipped the detach added for HITL listener accumulation. The detach now runs on both paths. * ⚖️ fix: Make the Commit Flag the Sole Billing Authority Round-twenty-three review: a committed fill racing a late scope close (user abort or settle timeout during the durable emit) stayed visible — the part is mutated and persisted before the close — yet the deferred accounting's scope gates then skipped the charge: a completed provider call escaping both the label charge and the primary abort accounting. The scope gates on the deferred-usage path are removed; the hook's commit flag is now the single billing authority in BOTH directions. A dropped fill never reaches the accounting callback (billed-never-shown stays impossible), and a committed fill bills regardless of when its scope closed (shown-never-billed now impossible too). The dead `scopeOpen` payload threading is removed with it; the `recordActivityLabelUsage` parameter survives, defaulting open, for callers that own no commit signal. The round's other finding is the twelfth instance of the documented edited+reconnect index limitation, answered on-thread. * 🧮 feat: Bill Labels by Estimate When Providers Omit Usage Maintainer decision: follow the title convention rather than leaving label calls unbilled when a provider returns no usage metadata. The hook now passes a LAZY estimate thunk with the deferred accounting on the success path — the EXACT prompt the direct path sent (or the locally built equivalent for the SDK path: same entries, context, instruction, truncation contract, and continuity headers) plus the final normalized label. `recordActivityLabelUsage` invokes it only when no collected entry carries a real token count, counts both texts with the shared o200k_base tokenizer, and feeds the synthesized entry through the SAME pipeline (provider-tagged, streamed event, cost, balance transaction). Real provider usage always wins when present. The failure path passes NO estimate: a throw before a response bills only real collected metadata, never a full phantom prompt. Tests: the estimate thunk carries the exact invoked prompt and final label; the failure path defers with no estimate (runtime.spec). * 💵 fix: Estimate From the Raw Completion, Not the Normalized Label The fallback estimate counted the normalized label (first line, 200-char cap) while the provider generated and would bill the raw output up to the 256-token generation cap — under-recording verbose replies. The estimate thunk now carries the raw pre-normalization text; the persisted label is unchanged. Pinned with a multi-line reply test. * 🧾 fix: Commit Label Text Only After the Durable Emit, Estimate the Real SDK Prompt Round review on the billing work: two fixed; the third is the fourteenth instance of the edited+reconnect index limitation, answered on-thread. - Copy-first fill (wiring.ts): the fill mutated the shared content part BEFORE its durable emit, so a failed emit left the label text on `contentParts` anyway — persistence could save and display a label no client ever received and billing (keyed on the commit flag) never charged. The new state is staged on a copy; the shared part mutates only after the emit succeeds, so content, delivery, and billing move together. - Real SDK prompt for estimates (client.js): the estimate thunk carried this module's locally built prompt, but the SDK path frames entries differently — the estimated input count was for a prompt never sent. Chain-start callbacks (handleLLMStart/handleChatModelStart) now capture the prompt the SDK actually rendered, and the deferred accounting substitutes it into the estimate when capture succeeded, falling back to the local approximation otherwise. |
||
|
|
4f5808d9ae
|
🧪 test: Reasoning-Stream Render Perf Benchmark via react-scan (#14494)
* 🧪 test: Reasoning-Stream Render Perf Benchmark via react-scan Adds a Playwright benchmark that streams one long, unsplit <think> block (18k chars — 4x the legacy SplitStreamHandler blockThreshold) plus 6k chars of markdown through the real mock-model agents pipeline, with react-scan injected to tally per-component renders. It verifies the legacy content-part splitting (removed in #10533) is not needed for rendering performance: - The whole reasoning section lands in ONE think part (a single Thoughts toggle) — nothing re-splits it anywhere in the pipeline. - rAF coalescing bounds the think box to ~1 render per 43 streamed chunks (122 renders / 5,290 chunks). - MarkdownBlock renders stay O(blocks + flushes) (153 renders / 2,092 text chunks), not O(blocks x tokens). - Long tasks during the 13.4s stream: one 96ms task; total render time 885ms. - Typing after the long transcript leaves transcript components quiet (<=2 renders across 40 keystrokes). Runs against the vite dev server (prod minification strips displayName assignments, which react-scan needs for naming). react-scan itself is not a repo dependency: install with `npm i --no-save react-scan` or point REACT_SCAN_PATH at its auto.global.js bundle. Also fixes the mock e2e stack for local runs: a developer .env with CHECK_BALANCE=true leaked through neutralizeCredentialEnv (not credential-shaped) and made every streaming mock spec fail with a token_balance violation, since the fresh e2e user has no balance record. vanillaOverrides now pins CHECK_BALANCE=false. * 🩹 fix: Address Codex Review — Payload Integrity, Frame Bounds, Proxy Port - Assert the full 18k-char reasoning payload survives the pipeline: expand the Thoughts toggle and compare rendered think text against the source (whitespace-normalized), instead of only counting toggles. - Derive render bounds from elapsed frames (60fps + headroom) rather than chunk counts, so the coalescing assertion stays meaningful regardless of how many chunks stream before resetPerf; apply the same bound to MarkdownBlock. - Tighten main-thread budgets: worst long task < 250ms and long-task total < 10% of stream wall time (baseline: one 51-96ms task per run). - Pass BACKEND_PORT derived from the configured E2E base URL to the vite dev server so its /api proxy follows a non-default app-server port. * 🧭 fix: Address Codex Round 2 — Typed Global, Drained Observer, Full-Payload Checks - Declare window.__PERF__ via global Window augmentation; drop the as-unknown-as double casts from both perf helpers. - Retain the longtask PerformanceObserver and drain takeRecords() before every snapshot/reset so stalls landing near the final render are counted. - Start the wall clock at the same instant as the tally reset so frame bounds and long-task percentages divide by exactly the measured interval. - Verify the complete markdown body: every generated section heading (exact-match), the exact list-item and table counts, and the generated code block — END_MARKER alone only proved the suffix rendered. - Require positive ThinkingContent/MarkdownBlock render counts so a renamed component or dropped instrumentation cannot void the upper bounds. - Cap cumulative render time at 25% of stream wall time to catch sustained sub-50ms work that never surfaces as a long task. * 🧷 fix: Address Codex Round 3 — Page Clock, Completion Wait, Exact Payload Checks - Measure the stream interval on the page's own clock: reset stamps the start, the snapshot evaluation reads the end, so bounds divide by exactly the tallied window including work between marker paint and snapshot. - Wait for the Stop generating button to hide before snapshotting, so generation finalization (usage chunk, terminal events, save re-render) is inside the measured interval. - Compare the rendered think text exactly (edges trimmed only) — internal paragraph breaks are user-visible under whitespace-pre-wrap and must survive verbatim. - Verify the markdown prose, not just structure: per-section doubled-sentence paragraph and both list-item texts, exact table count with cell values, and both generated code lines. - Derive the vite proxy port via getE2EServerAddress() so implicit ports in E2E_BASE_URL (default 80/443) agree between the app server and the proxy. - Pin react-scan@0.5.7 in the README — thresholds are calibrated against its instrumentation semantics. * 🪛 fix: Address Codex Round 4 — Pre-Send Reset, Count Every Code Block - Reset the tally immediately BEFORE triggering the send: with a 1ms chunk delay, the earliest deltas can render between the response headers resolving and a post-send evaluation, which the old order erased from the measurement. - Assert Math.floor(sectionCount / 3) occurrences of both generated code lines via code-element locators instead of .first(), so dropped later code blocks can no longer pass the payload check. * 🎛️ fix: Address Codex Round 5 — First-Render Clock, Expanded Box, Typing Budget - Stamp the wall clock at the FIRST render after each reset (inside onRender) so idle request-setup time between reset and stream start never pads the frame, long-task, or render-time denominators. - Seed showThinking=true so the reasoning box streams EXPANDED — the heavier live-layout path — and drop the post-hoc expand click. - Bound the typing phase itself: worst long task < 150ms and cumulative render time < 25% of the typed interval, so input lag without transcript re-renders still fails. - Derive the vite dev server host from getE2EServerAddress() alongside the port, so a non-localhost E2E base URL keeps the app server, listen host, and /api proxy in agreement. * 🧿 fix: Address Codex Round 6 — Stream-Anchored Clock, IPv6 Proxy, Rate-Free Bounds - Anchor the stream clock to the first ThinkingContent render — the payload opens with reasoning, so that is the first assistant-content paint — keeping composer renders and idle request setup out of the denominators. - Bracket IPv6 HOST values when building the vite /api proxy target in client/vite.config.ts; unbracketed ::1 produced an unparseable URL. - Add an absolute cumulative long-task budget (<300ms) to the typing phase so repeated sub-threshold stalls cannot evade the worst-case check or dilute the ratio via inflated elapsed time. - Add chunk-relative companion bounds (renders < chunks/4) for both ThinkingContent and MarkdownBlock, and hard-pin MOCK_LLM_CHUNK_DELAY_MS=1, so a slower stream can no longer loosen the coalescing assertions. |
||
|
|
49f4b659f6
|
🔐 fix: Honor Admin-Panel MCP Allowlist Overrides Without Restart (#13814)
* 🔐 fix: Honor Admin-Panel MCP Allowlist Overrides Without Restart MCPServersRegistry was built once at boot from getAppConfig({ baseOnly: true }), freezing allowedDomains/allowedAddresses to YAML. Admin-panel mcpSettings overrides were ignored by both inspection (addServer/ reinspectServer/updateServer/lazyInitConfigServer) and runtime connection enforcement (assertResolvedRuntimeConfigAllowed), so a domain allowed only via the panel failed inspection and never connected. Make the registry's effective allowlists mutable and refresh them from the merged admin-panel config: seed at boot, and re-apply on every config mutation via invalidateConfigCaches -> clearMcpConfigCache. Both inspection and connection paths read the same getters, so both honor overrides without a restart. Fail-safe: current allowlists are preserved when the merged read fails. * 🛡️ fix: Scope MCP allowlist refresh to global config, fail-safe on DB error Address Codex P1 review findings on the allowlist-refresh path: - Tenant-scoped config mutations no longer push one tenant's merged mcpSettings into the process-wide registry singleton (read by all MCP connection paths), which would leak allowlists across tenants. Only global (non-tenant) mutations refresh the registry; tenant mutations still evict the config-server cache. - The refresh read now uses strictOverrides:true so a transient DB error throws instead of silently returning YAML base config — preserving the last-known allowlists rather than overwriting them with fallback values. Adds the strictOverrides option to getAppConfig (default off, no behavior change for existing callers). * ♻️ refactor: Resolve MCP allowlists per-request (tenant-scoped) instead of a global singleton Supersedes the prior global-mutation approach. MCP allowlists live in mcpSettings, which is tenant/principal-scoped admin config, so a process-wide singleton value is the wrong model — it caused cross-tenant bleed and stale reads. Instead, inject a resolver (from the app layer, where the merged config lives) that the registry calls per inspection and per connection. It reads the ALS tenant context via getAppConfig and accepts the acting user so user/role-scoped overrides resolve; config-source inspection (no user) resolves at tenant scope. Falls back to the YAML base allowlists when no resolver is set or the lookup fails, so a transient error fails to the operator baseline rather than disabling the allowlist. Removes the now-unnecessary setAllowlists / boot-seed / invalidateConfigCaches refresh / getAppConfig.strictOverrides machinery. * 🔒 fix: Scope config-source cache by allowlist; resolve OAuth allowlists per-request Address Codex review of the per-request resolver: - Config-source cache key now folds in the resolved allowlists, not just the raw-config hash. Inspection results became allowlist-dependent, so without this a tenant whose allowlist rejects a URL could poison the shared key with an inspectionFailed stub for a tenant that allows it (and vice versa). The tenant-scoped allowlist is resolved once per ensureConfigServers pass and threaded through the cache key + inspection. - The two remaining request-time OAuth allowlist reads now use the merged config instead of the YAML base getters: the fallback OAuth-initiate path (routes/mcp.js) via resolveAllowlists, and OAuth revocation (UserController.maybeUninstallOAuthMCP) via the request's already-merged appConfig.mcpSettings. Without this, an OAuth endpoint allowed only by an admin-panel override was rejected while inspection/connection allowed it. * ✅ test: Update MCP OAuth registry/config mocks for per-request allowlists CI fix for the Finding-12 change. The OAuth-initiate route now calls registry.resolveAllowlists() and the revocation path reads the merged appConfig.mcpSettings, so the affected specs' mocks were asserting the old base-getter values: - routes/__tests__/mcp.spec.js: add resolveAllowlists to the registry mock. - UserController.mcpOAuth.spec.js: provide mcpSettings on the getAppConfig mock so revokeOAuthToken still receives the expected allowlists. * 🧪 test: e2e proof that admin-panel MCP allowlist override takes effect Adds a Playwright mock-harness spec for #13809. A URL-based MCP fixture (e2e-http, streamable-http SDK server) boots inspectionFailed because its origin is omitted from the YAML mcpSettings.allowedDomains; the spec adds that origin via an admin config override (PUT /api/admin/config/user/:id) and asserts the server reinitializes — exercising the real resolver path through the backend + DB. Before the fix, reinspection used the frozen YAML allowlist and the server stayed unreachable. - e2e/setup/fake-mcp-http-server.js: streamable-HTTP MCP fixture (health GET /). - e2e/playwright.config.mock.ts: boot the fixture as a second webServer. - e2e/config/librechat.e2e.yaml: mcpSettings.allowedDomains (excludes 127.0.0.1) + the e2e-http server. - e2e/specs/mock/mcp-allowlist-override.spec.ts: login → baseline reinit fails → apply override → reinit succeeds. |
||
|
|
6055ad0af2
|
🪃 fix: Restore Raw Spec Fallback for Enforced Presets (#13804)
Some checks are pending
Docker Dev Branch Images Build / build (Dockerfile, lc-dev, node) (push) Waiting to run
Docker Dev Branch Images Build / build (Dockerfile.multi, lc-dev-api, api-build) (push) Waiting to run
GitNexus Index / index (push) Waiting to run
GitNexus Index / post-index (push) Blocked by required conditions
Docker Dev Images Build / build (Dockerfile, librechat-dev, node) (push) Waiting to run
Docker Dev Images Build / build (Dockerfile.multi, librechat-dev-api, api-build) (push) Waiting to run
Sync Locize Translations & Create Translation PR / Sync Translation Keys with Locize (push) Waiting to run
Sync Locize Translations & Create Translation PR / Create Translation PR on Version Published (push) Blocked by required conditions
Sync Helm Chart Tags / Ignore non-main push (push) Waiting to run
Sync Helm Chart Tags / Sync chart tags (push) Waiting to run
* fix: rebuild enforced specs from preset * test: Add enforced model spec e2e coverage * test: Align enforced spec regression scope |
||
|
|
2c8d54e18c
|
🗂️ feat: Add Deployment Skill Directory (#13523)
Some checks are pending
Docker Dev Branch Images Build / build (Dockerfile, lc-dev, node) (push) Waiting to run
Docker Dev Branch Images Build / build (Dockerfile.multi, lc-dev-api, api-build) (push) Waiting to run
GitNexus Index / index (push) Waiting to run
GitNexus Index / post-index (push) Blocked by required conditions
* feat: Add deployment skill directory * chore: Address deployment skill review feedback * fix: Include deployment skill file metadata * test: Add deployment skills e2e smoke test |
||
|
|
a1bfa3b298
|
🎭 test: Run Mock E2E Suite Through createRun With In-Process Fake Model (#13508)
* 🎭 test: Run Mock E2E Suite Through createRun With In-Process Fake Model Replace the standalone HTTP mock LLM server with an in-process fake model injected into the real createRun -> Run.create pipeline via run.Graph.overrideTestModel, so the mock suite exercises the agents integration end-to-end without a live provider or a separate server. - Bump @librechat/agents to 3.2.2 for the FakeChatModel/createFakeStreamingLLM exports - Add an env-gated applyTestRunHook seam in packages/api createRun (no /api changes) - Add e2e/setup/fake-model.js to drive default replies + the skill-authoring tool-call flow - Drop the mock-llm webServer from playwright.config.mock.ts and set LIBRECHAT_TEST_RUN_HOOK * 🧹 test: Retire Standalone Mock LLM Server From E2E Recorder Migrate the `--profile=mock` recorder onto the same in-process fake model as the Playwright mock suite, then delete the now-unused HTTP mock server so the fake-LLM logic lives in a single place. - Point record.js mock profile at the fake model via LIBRECHAT_TEST_RUN_HOOK - Remove the mock-llm-server spawn/wait and MOCK_LLM_PORT plumbing from record.js - Delete e2e/setup/mock-llm-server.js (e2e/setup/fake-model.js is now the only source) - Update e2e/README.md to describe the in-process fake LLM * 🏷️ ci: Rename Playwright Mock E2E Check to Playwright E2E Tests |
||
|
|
d680763db3
|
🧭 ci: Use System Chrome for Mock E2E (#13481)
Some checks are pending
GitNexus Index / index (push) Waiting to run
GitNexus Index / post-index (push) Blocked by required conditions
Sync Locize Translations & Create Translation PR / Sync Translation Keys with Locize (push) Waiting to run
Sync Locize Translations & Create Translation PR / Create Translation PR on Version Published (push) Blocked by required conditions
Sync Helm Chart Tags / Ignore non-main push (push) Waiting to run
Sync Helm Chart Tags / Sync chart tags (push) Waiting to run
|
||
|
|
b45e4aeae5
|
🎭 feat: Add Credential-Free Playwright Smoke Suite with a Local Mock LLM (#13472)
* 🧪 feat: add e2e playwright tests * 🧪 feat: Add Playwright Recording Harness * test: fix mock playwright config * test: harden mock e2e environment * test: preserve mock dotenv secrets * test: harden mock isolation setup * ci: cache mock e2e builds * test: harden e2e cache and recorder checks * test: preserve data-provider exports in oauth route test * test: isolate mock auth logout state * test: allow isolated logout smoke setup * test: prepare logout smoke auth via api * test: isolate oauth route module mock --------- Co-authored-by: Danny Avila <danny@librechat.ai> |