mirror of
https://github.com/danny-avila/LibreChat.git
synced 2026-10-29 18:17:52 +00:00
7b6900d556
* ✨ feat: Activity Groups with Fast-Model Labels Groups each contiguous block of reasoning + tool calls into a collapsible unit headed by a fast-model label (claude.ai-style hierarchy), off the critical path: a PostToolBatch hook claims a live content slot at the batch boundary (steering index-offset pattern), renders a deterministic counts phrase instantly, and swaps in the generated label ~1s later while the next model call streams. Labels are UI-only — stripped before the SDK formatter and skipped in the legacy formatter — and reach live clients via a dedicated on_activity_label SSE event (live/replay/pending paths). Grouping preserves legacy rendering byte-for-byte when no label part is present. Generation bridges to Run.generateActivityLabel() when the SDK ships it (session-grouped Langfuse tracing); falls back to a direct call today. Env-gated: ACTIVITY_LABELS_POC=true, ACTIVITY_LABEL_MODEL. * 🧷 fix: Address Codex and Copilot Review Findings for Activity Labels - Settle in-flight label fills (bounded 3s) before finalization on both the main and resume paths, so a label resolving during the final batch still reaches the durable log and saved message. - Overlay on_activity_label chunks in RedisJobStore content reconstruction (splice path last-wins per index; replay path chronological overwrite), matching steer handling. - Wire activity labels into the HITL resume createRun so post-resume batches keep claiming slots. - Guard against out-of-order publishes: fill() awaits the claim emit before emitting the resolved label, and the client applier ignores a stale pending placeholder once a resolved label is present. - Stamp the batch's groupId onto label parts so parallel-column runs place them inside their group instead of filtering them out. - Localize the counts fallback phrase (10 keys, singular/plural) through useLocalize across chat rendering and exports. - Type the hook with Providers/ClientOptions instead of stringly types; drop the unknown cast in the spec; add a dedicated rAF retry ref for label events with effect cleanup. * 🛡️ fix: Address Independent Review — Abort, Usage, Lane Context, Redis Test - Propagate the run abort signal into label generation (both wiring call sites; runtime combines host + dispatch signals with the timeout) so a user abort cancels in-flight label calls instead of paying to timeout. - Record label-call usage like titles: the SDK bridge aggregates via chainOptions callbacks, the fallback path via a per-generation callback factory; both feed recordCollectedUsage under context 'activity-label'. - Scope block-context capture: reasoning collection stops at the previous block's label part and filters by executingAgentId, so consecutive or parallel batches can no longer bleed another block's thinking into the payload; intent text still scans past labels (persists across batches). - Forward the effective charLimit to the SDK call so host and SDK prompts agree (SDK default aligned to 600 in agents#327). - Add a Redis integration test proving last-write-wins reconstruction of on_activity_label chunks per claimed index. - Rebased onto main: only the two activity commits replay (the nine steering commits belonged to the old base branch), zero conflicts, steering suites green. * 📐 refactor: Move Activity-Label Wiring to TypeScript, Address Codex Round 2 - [P1] Slot claiming, lane stamping, emit ordering, context capture, and settle tracking now live in packages/api (createActivityLabelWiring + captureActivityBlockContext); client.js is a thin closure wrapper. - Register the activity-label hook BEFORE the steer drain so a steer draining at the same batch boundary cannot flush the tool block and orphan the label outside its group. - Resolve request-based header placeholders in resolveActivityLabelLLM (titleConvo parity) so metadata-keyed proxies work on label calls. - Trim labels centrally before filling so whitespace-only output from either generation path keeps the deterministic counts fallback. * 🧭 fix: Codex Round 3 — Capture Order, Shared Strip, Token Estimator, Hide Filter - Capture block context BEFORE pushing the label part: the scan stops at ACTIVITY_LABEL parts, so post-push capture hit the just-inserted label and silently collected no reasoning excerpts (regression test added). - Share stripActivityLabelParts from packages/api and apply it in the Responses and OpenAI-compatible controllers, closing the replay leak for entry points still running SDKs without the formatter skip. - Exclude activity_label parts from the fallback response-token estimator (UI-only parts must not inflate no-usage provider billing). - Keep label parts explicitly under hide_sequential_outputs — they summarize exactly the outputs that mode hides. * 🔁 fix: Codex Round 4 — Resume Gap, Delta Flush, Agent-Scoped Intent, Token Counter - Synthesize on_activity_label events for labels claimed or filled in the snapshot→subscribe window (the publish is fire-and-forget, so Redis-mode reconnects missed them). Feature-gated so the default path adds no content re-read; the client applier already ignores duplicates. - Flush queued deltas before applying a label part, matching the pending- action and steer appliers — without it the handler read a stale message cache and syncStepMessage pushed a pre-delta copy back. - Skip another agent's tail text when resolving intent, so parallel runs cannot seed a label prompt with a sibling agent's narration. - Exclude activity_label parts from countFormattedMessageTokens (the agent-path counter), not just the legacy BaseClient one. * 🏗️ refactor: Codex Round 5 — Extract Label Host Logic, Report Usage, Icon Strip - Move provider/model resolution, usage-metadata mapping, and the settle loop into packages/api (activityLabels/host.ts); client.js keeps only thin delegations, per the repo's TypeScript-implementation convention. - Fold label usage into the response rollup with an 'activity-label' tag (subagent precedent) so metadata.usage and the live cost gauge account for it; tagged, so it stays out of PRIMARY usage/context pairing. - Narrow tool metadata once in ToolCallGroup so THINK parts in a labeled block no longer render phantom generic icons in the stacked strip. - Import the activity-label helpers by deep path in GenerationJobManager: the package barrel now reaches provider-config/cache modules that import back into the stream layer, and the cycle broke suite loading. Declined: resetting steerOffsetState before HITL resume — resume builds a FRESH AgentClient via initializeClient (initialize.js:978), so the offset is already zero; the seed wrapper alone accounts for pre-pause parts. * 🚦 fix: Codex Round 6 — Stream Label Usage, Close Late Fills - Emit an on_token_usage chunk for label calls (sink push alone left the live session gauge blind); retained in pendingSubagentEmits so job cleanup cannot race the persist, tagged 'activity-label' as before. - Close the label scope when settle times out: the wiring gates fill() on isClosed and the client fires a label-scoped AbortController, so a straggling generation can neither mutate a saved response nor emit into a job whose runtime is gone. The controller also chains to the run signal, so a user abort still cancels label work. * 🩹 fix: Repair CI — Package Typecheck and Module Mocks Local runs covered the client tsconfig and jest, but never packages/api's own tsconfig, so nine type errors in the extracted host module shipped. - Type host.ts against the real contracts: ServerRequest, EndpointDbMethods, AppConfig from @librechat/data-schemas, IUser for createSafeUser, and a MaybeAzureConfig view for the azure instance-name probe and configuration. - Widen resolveConfigHeaders' llmConfig to Partial<RunLLMConfig>: it only reads the three provider header carriers, so auxiliary generations with a bare ClientOptions can resolve headers without assembling a run config. Type-only widening; every existing caller still satisfies it. - Add stripActivityLabelParts to the @librechat/api mock in the OpenAI and Responses controller specs — those mocks enumerate exports, so a new import read as undefined and threw before the assertions ran. - Use the real activity-label helpers in the ToolCallGroup spec's ~/utils mock; stubbing them out would hide the header logic under test. * ⚙️ feat: Configure Activity Labels via librechat.yaml, Drop Env Vars Replaces the ACTIVITY_LABELS_POC / ACTIVITY_LABEL_MODEL env gate with per-endpoint settings, following the title options convention rather than a top-level block — each endpoint picks its own cheap label model. - Add activity, activityModel, activityEndpoint, activityPrompt, activityMaxPerRun, and activityCharLimit to the endpoint schema, and to the endpoints.all pick list (enumerated, so 'all:' would otherwise drop them silently). - resolveActivityConfig reads them with title-style precedence: endpoints.all > named endpoint > custom endpoint config. - Model precedence is now activityModel > titleModel > the agent's model. activityEndpoint runs labels on another endpoint's credentials, with titleConvo's fallback-on-unknown-name behavior. - Thread activityPrompt/MaxPerRun/CharLimit through the wiring into the hook and the SDK bridge; they were hardcoded defaults. - The resume gap-repair gate keyed on the env var; it now keys on the snapshot actually containing label parts, so deployments without the feature still perform no extra content read. - Document the fields in librechat.example.yaml; add host.spec.ts covering precedence, custom-endpoint fallback, and opt-out. * 📝 refactor: Rename Enable Flag to activityLabel, Document Schema Inheritance - Rename the boolean from `activity` to `activityLabel`, matching the titleConvo/titleModel shape: a verb-object toggle whose prefix matches its modifiers (activityModel, activityPrompt, ...). `activity: true` alone read ambiguously — it could mean tracking or logging activity. - Document the two endpoint-schema inheritance paths, which behave oppositely and are ~900 lines apart: * `endpoints.all` omits from baseEndpointSchema, so new options are inherited automatically — nothing to maintain. * `azureEndpointSchema` enumerates via .pick(), so a new option is silently unavailable on Azure endpoints until listed there. The activity block now carries a pointer to the Azure caveat. * 🔍 fix: Address Codex Findings on the Config Rework - Pass the matched custom-endpoint config into the label gate. Custom endpoints live in the `endpoints.custom` ARRAY, so without it every custom endpoint resolved as disabled — including the example this PR added to librechat.example.yaml. - Give label usage a unique `runId:seq`. Label usage is billed but never appended to `collectedUsage`, so its length was static: every label event reused the last primary usage's pair and collided with itself, and the client dedupes on exactly that. - Attach `cost` to label usage when `interface.contextCost` is on; aggregateEmittedUsage treats coverage as all-or-nothing, so an event without it suppressed the whole response's cost. - Honor `activityPrompt` on the direct fallback path, not just the SDK bridge — it previously always used the built-in instruction. - Seed the per-response label cap from labels already on the response so a HITL resume cannot mint a fresh quota after every approval. - Reconcile label gaps on resume via a durable per-job `activityLabels` flag instead of probing the snapshot: the FIRST label of a run can be claimed inside the snapshot->subscribe window, which the old signal missed. The flag is read from a job record already fetched there, so runs without the feature still add no content read. - Auto-collapse labeled single-tool groups; one-call batches are common in agent runs and rendering them expanded defeats the grouping. * 🎯 fix: Correct Label Usage Seq, Cross-Endpoint Pricing, Close Scopes - Give label usage a NEGATIVE seq namespace. The previous fix was wrong: seq is a position in `collectedUsage` (push, then emit with the new length), so sink-length + array-length still lands on a real position — primary emits 1, the label computes 2, the next primary also emits 2. Labels have no position at all (billed separately, never appended), so they now occupy a namespace positional sequences cannot reach. The client key is a string used for Set membership, so the sign is inert. - Price cross-endpoint labels with the LABEL endpoint's token config: resolveActivityLabelModel now returns the resolved endpointTokenConfig, and both the streamed cost and recordCollectedUsage use it instead of the agent endpoint's rates. - Make close state per-wiring rather than per-client. A HITL resume rebuilds the wiring, and resetting a shared flag re-opened closures from the pre-pause segment whose provider call ignored the abort; settle now closes every retained scope, past generations included. * 🎯 fix: Make the Activity Header Say Something the Cards Cannot The header read "ran 1 command" next to a card already labeled "Code" — it restated the UI beneath it instead of adding to it. Two causes, both about content rather than timing: - A deterministic tool-type tally was the primary display and also fed the prompt, so the best case was a tally and the worst case was a tally dressed as prose. Removed from the metadata, the prompt, the part type, and the client. - The instruction only ever reached the fallback path. The wiring passed a prompt only when was configured, so the preferred SDK path silently used the published package default. The wiring now always supplies one and the hook forwards it on both paths. The register is rewritten around what the cards cannot show: past-tense git-commit-subject, leading with the distinctive noun, outcome over attempt, tool names and counts and arguments explicitly forbidden. The batch entries are labeled as reference material so the model stops transcribing them. Claiming a slot no longer emits. The slot still reserves its index so streamed parts never collide, but with nothing to say there is nothing to render: until a description exists the block looks exactly as it does without the feature. * 🧹 fix: Drop the Localize Hook Left Unused by the Counts Removal * ✅ test: Add Activity-Label e2e Coverage with a Recording Label Server Activity labels are the one model call a mock run does not already fake: fake-model.js swaps the GRAPH model via overrideTestModel, while run.generateActivityLabel() calls the endpoint resolved client options over HTTP. The custom endpoints already point baseURL at 127.0.0.1:8889, so serving that port exercises the real path with no production seam. fake-label-server.js answers it in both JSON and SSE form, records each prompt, and can inject blank/error responses. Recording is what lets the spec assert the CONTRACT rather than the rendering: that this repo register and the tool OUTPUTS actually reach the model. That is the bug class that produced unusable labels before, and rendered text looks identical whether or not the instruction arrived. Labels get a dedicated endpoint (Mock Provider E). A labeled block auto-collapses even at one tool call, which hides the tool cards other specs assert on -- enabling this on a shared endpoint broke steering.spec.ts. Provider D is the unlabeled control. Request-count assertions are scoped to a per-test token: a 5xx label response is retried by the provider client, and a retry can land after the next test has reset the server. * 🩹 fix: Address Review Findings on Activity-Label Indexing and Pricing Replay index (P1). Reserving the slot only in server memory left no event for it, so a cross-instance replay rebuilt content as [tool, hole, later], compacted the hole away, and the fill for the reserved index landed on the following part and overwrote it. The claim now publishes the empty, pending part so the index is real for every consumer, and fill publishes even when generation returned nothing so the client cannot stay pending. It stays invisible: an empty label still DELIMITS its batch in groupSequentialToolCalls but is not attached as the header, so grouping does not re-shuffle when the text lands and the block renders exactly as it does with the feature off. Edited-response index (P1). Edit-and-resubmit replays the kept prefix and the server indexes only new content, so run steps offset by that prefix. Labels are claimed in the same space and now take the identical shift; without it a label could land inside the prefix and overwrite it. Redis flag. deserializeJob never read activityLabels back, so every Redis reload left it undefined and resume skipped label gap reconciliation. Executing agent. RunActivityLabelOptions.agentId selects the executing agent tracing metadata AND its tool-output redaction policy; omitting it let a handoff be redacted under the default agent configuration. Label pricing. An undefined endpointTokenConfig is meaningful for a built-in label endpoint (priced from the shared table), so the nullish fallback billed those labels at a custom primary rates. Inherit only when the label runs on the agent own endpoint. HITL usage sequence. runId is the response message id and the counter was instance-local, so a resume restarted at -1 and the client runId:seq deduper discarded the post-approval label usage. Seeded past the labels already on the response. Also distinguishes "cannot serve" (undefined) from "no label" (null) in the SDK bridge, so a missing run falls back to the direct call instead of filling the slot empty. Version gating already happens at wiring time via the sdkCapable prototype probe. * 🩹 fix: Keep Unfilled Activity Labels Invisible and Unmask Endpoint Settings Follow-up review round. Publishing the reservation on every batch made two latent rendering paths reachable on every run, and both are fixed here. Empty labels no longer change grouping. The previous pass still formed a tool-group for a textless label, which wrapped even a single tool call and pulled THINK parts inside it — and since a reservation is published the moment each batch ends, that applied during every generation and permanently after a blank or failed fill. An empty label now flushes the legacy way instead: it still delimits its batch, but the block re-splits exactly as it renders with the feature off. Parallel lanes no longer show a blank line. Lanes render raw parts, so an unfilled label had nothing to draw; empty ones are dropped. Making labels act as collapsible headers inside lanes is still a separate gap. Edited responses no longer offset on resume. The sync replaces initialResponse.content with the server's aggregatedContent, which already contains the kept prefix AND everything generated since — so its length is not the prefix length, and indices reconciled from that snapshot are already absolute. Offsetting again pushed the label past its slot onto a later part. The shift now applies only to a fresh edited submission. Activity settings resolve field by field. Selecting one config object whole meant any endpoints.all block — even one carrying nothing but headers — shadowed the named or custom endpoint and silently disabled activity labels everywhere. Global still wins per field. Adds groupToolCalls coverage for the invisible-while-empty contract, which is the part most likely to regress: it is normal state on every run, not an edge case. * 🔒 fix: Scope Detached Label Writes to Their Generation Epoch Epoch scoping (P1). Label generation is detached and can outlive the generation that started it. emitChunk only proves that SOME runtime is current, not that the caller belongs to it, so an aborted generation's fill(null) -- and its usage event -- could be attributed to whichever generation replaced it, landing an index from the abandoned response on top of the new one. Because an empty label renders nothing, that overwrote content silently. emitChunk now takes an optional jobCreatedAt and drops the event when the runtime epoch differs, mirroring the existing setGraph/setContentParts convention, and both label emitters pass it. An abort now CLOSES the label scope instead of only cancelling the call: the rejected generation still runs its catch and calls fill(null), which would otherwise emit into a stream the next generation may already own. Edited-response indexing (P1). The previous pass skipped the prefix offset on resume, which was the wrong half of the problem: a sync replaces initialResponse.content with the server's aggregatedContent, which is completion-local, so after a reconnect its length is not the kept-prefix length and the offset is wrong -- but it is wrong for run steps in exactly the same way. Tool cards and the label that heads them must share one index space; a label shifting differently from its tools lands on another part. The label path now uses the identical expression as useStepHandler, with no resume special-case. Correcting the post-resume prefix length belongs in calculateContentIndex, where it fixes both at once. titleModel masking. The activity settings were made per-field last pass, but the titleModel fallback a few lines below still selected an entire config object, so a partial endpoints.all (for example one carrying only headers) hid a named endpoint's titleModel and quietly fell the label back to the main agent model. Both now read through one shared per-field helper. Resume reconciliation no longer depends solely on markActivityLabels, which is best-effort yet had come to gate correctness: a lost flag write silently dropped a label. The snapshot is consulted as a fallback. The exported host type for generateLabel now admits undefined, which is the documented "cannot serve, fall back to the direct call" signal the hook keys on -- distinct from null, meaning it ran and produced nothing. * 🧷 fix: Keep Group Identity Stable and Memoize Label Endpoint Resolution Group remount. Tool-group identity was keyed on the first part in the block. An activity label absorbs the block's leading THINK part the moment its text lands, so the key flipped from tool:<id> to fallback:<scope>:<idx> mid-run, remounting the group and discarding whatever the user had expanded. The key now scans for the first tool call, which does not move when the block re-forms. Label endpoint resolution is memoized per response. It reads provider config and can hit the database for user keys, yet nothing it depends on changes between batches of one run — and it ran twice per batch, once for generation and once for usage accounting. The promise is cached rather than the value so concurrent batches share a single in-flight resolution, and a rejection is evicted so one transient credential failure cannot disable labels for the rest of the response. * 🎯 fix: Offset Edited Resubmissions by a Prefix Length That Survives Resume The server indexes only NEW content for an edited resubmission, so the client offsets incoming indices by the prefix it retained. That prefix was read as initialResponse.content.length, which is correct only until a resume: the sync replaces that array with the server's completion-local snapshot, whose length is unrelated to the prefix. After a reconnect every offset was therefore wrong -- run steps and activity labels alike -- and could write over content the edit kept. For a label the symptom is worse than a bad position: the fill misses its own reservation, so the pending placeholder is never resolved. The prefix length is now captured when the submission is built, while initialResponse.content still IS the retained prefix, and carried on the submission as editPrefixLength. calculateContentIndex takes that length instead of deriving it from an array that a resume may have replaced, so run steps and labels share one index space by construction rather than by both happening to read the same field. Note the prefix is the FULL original content with the edited part substituted in place (useChatFunctions clones latestMessage.content and mutates one entry) -- it is not a slice, so the length cannot be inferred from editedContent.index. Group identity no longer changes when a label fills. Tool-group keys were derived from the first part in the block; an activity label absorbs the leading THINK part when its text lands, flipping the key mid-run and remounting the group, which discarded the user's expansion state. The key now scans for the first tool call, which does not move. Label endpoint resolution is memoized per response. It reads provider config and can hit the database for user keys, yet ran twice per batch -- once to generate, once for usage accounting -- while nothing it depends on changes within a run. The promise is cached so concurrent batches share one in-flight resolution, and rejections are evicted so a transient credential failure cannot disable labels for the rest of the response. The resume gap passes for steers and activity labels now share a single lazy content read instead of each issuing its own. The label pass stays gated on the run flag with a snapshot fallback: reconciling unconditionally would also close the residual first-label window, but it would bill a read to every resume of every run, including deployments with the feature off -- which the steer pass deliberately avoids. That residual requires a lost flag write, which shares fate with the content writes the labels live in. * 💵 fix: Bill Cross-Endpoint Labels at Their Own Rates recordCollectedUsage never accepted an endpointTokenConfig, so the value the activity-label caller passed was dropped and the balance transaction was written at the primary agent's rates. Only the UI cost honored the label endpoint, so a custom primary pointing activityEndpoint at another endpoint showed one price and charged another. The parameter is now accepted, and an explicit config wins outright over per-agent resolution: that map is keyed by AGENT, so it cannot describe usage that ran on a different endpoint. Group identity is stable for id-less tool calls too. The previous pass anchored the key to the first tool call ID; where a supported tool call carries no id the fallback still used the block's first part index, which shifts when a filled label absorbs the leading THINK part. The fallback now anchors to the first TOOL entry's index, so only a block containing no tool call at all keys off parts[0]. markActivityLabels is retried rather than fire-and-forget. It gates resume gap reconciliation and is a SEPARATE write from the durable label append, so a single lost write silently drops a label the content itself recorded. The earlier "shared fate with content writes" reasoning was wrong. One retry at run setup costs nothing and removes the only realistic way the gate goes stale, without billing a content read to every resume. * 🧮 fix: Stop Offsetting Once SYNC Drops the Edited Prefix The edit offset was applied unconditionally, but whether it is correct depends on which branch SYNC took. SYNC either preserves the content already loaded for the response -- which still contains the retained prefix, so the offset is required -- or replaces it with the server's aggregatedContent, which is completion-local and indexed from zero, after which any offset writes past the end of a now shorter array. That is why the two previous attempts each fixed half of it: skipping the offset on resume was right for the replace branch, applying it unconditionally was right for the preserve branch, and neither holds on its own. The offset now tracks the actual state of the rendered content. For an activity label the replace branch was worse than a bad position: the fill landed past its own reservation, so the pending placeholder was never resolved and the block kept its generic header for the rest of the run. Applied to run steps as well, not just labels. useStepHandler reads the prefix from the same submission and had the same unconditional offset, so after a mid-session resume of an edited response tool cards were misplaced too. Normalizing at the dispatch boundary keeps both in ONE index space by construction: a label that shifted differently from the tools it heads would land on another part. Note the reload path was already coherent -- useResumeOnLoad rebuilds the submission without editedContent or editPrefixLength, giving no offset against server-supplied content -- so only the mid-session SYNC path was inconsistent. * 🧾 fix: Keep Label Accounting Out of the Primary Usage Slot Label usage no longer owns getStreamUsage(). recordCollectedUsage assigned its result to this.usage unconditionally, so when the primary provider reported no usage metadata but the label provider did, BaseClient took the label's output tokens as the assistant response's authoritative count. The later primary call returns early on an empty collectedUsage and never replaced it, so the wrong value stood, the text-based fallback was skipped, and the real generation went unbilled. Secondary usage is still billed but no longer writes that slot. Cross-endpoint pricing keys off an explicit discriminator rather than the presence of a value. A built-in label endpoint prices from the shared table, so an undefined endpointTokenConfig is its MEANINGFUL value -- reading that as "no override" fell back to the primary's custom rates and restored the exact mismatch the previous pass set out to fix. The caller already knows whether the label ran elsewhere and now says so. markActivityLabels rejects on failure instead of swallowing it. The flag gates resume gap reconciliation and the caller retries it, but the internal catch resolved successfully and made that retry unreachable -- so the two changes cancelled out and a transient write failure still left the flag absent. Late label accounting is suppressed with the same gate as the late fill. A straggler that outlived the settle timeout still ran its finally block, so it charged the balance and appended to usageEmitSink after the response had passed its usage flush and metadata snapshot: a cost the user pays but is never shown. The cleared-prefix state is scoped to one generation. It was set on a resume SYNC that replaced the response and then never reset, so a later edited resubmission in the same mounted hook dispatched run steps and labels with no offset against content that still held its retained prefix. Reconnects pass isResume and keep the state; a new generation clears it. * 🔑 fix: Key Prefix State to the Stream and Honor current_model for Labels The cleared-prefix reset keyed on isResume, which skips exactly the case it was added for: a submission whose POST succeeded server-side but lost its response is retried, comes back resumed: true, and subscribes in resume mode even though it is a NEW generation. A previous generation's cleared state then survived into it, and incoming run steps and labels applied no offset against content that still held its retained prefix. The state is now keyed to the stream id, which changes with the generation and stays put across reconnects of one. activityModel now honors current_model. The options are documented as title-shaped and the titleModel fallback already excludes the sentinel, but the higher-precedence activity override passed the literal string through to getOptions and the provider, so an endpoint following that convention failed every label instead of using the agent model. * 🎯 fix: Key Prefix State to the Generation and Resolve the Run Model The cleared-prefix state was keyed to the stream id, which never changes within a conversation: request.js sets streamId = conversationId, so once a reconnect cleared the state every later edited resubmission in that conversation dispatched run steps and labels with no offset and could overwrite the prefix it retained. It is now keyed to the response message id, the only per-generation identity available here -- minted per submission and carried through a resume unchanged. That is the third identity tried for this state. isResume missed the deduplicated-retry path (a lost response returns resumed: true for a new generation); the stream id is conversation-scoped. The response id is the boundary that actually matches a generation. current_model labels now resolve the model the run is really using. initializeAgent merges the request's endpointOption override into model_parameters and the run gives it precedence, so preferring the saved agent.model could send labels to a different, potentially unavailable or more expensive model than the conversation is on. * 🆔 fix: Key Prefix State to the Submission and Keep the Origin Title Model Editing an assistant response reuses that response's messageId as editedMessageId, and useChatFunctions carries it onto initialResponse.messageId -- so re-editing the same response produced two generations with the same key and the cleared-prefix state survived between them, leaving run steps and labels with no offset against content the edit retained. Keyed now to clientRequestId, the per-submission uuid, which is minted fresh per edit attempt and forwarded unchanged on retries. That is the fourth key this state has had, and each earlier one failed at a real boundary: isResume missed the deduplicated-retry path, the stream id is the conversation id, and the response message id is reused across edits of one response. clientRequestId is the identity that actually means "this submission". The titleModel fallback is read from the ORIGINATING endpoint again, matching how titleConvo captures its config before switching credentials. Reading it after an activityEndpoint switch meant an OpenAI endpoint configured with titleModel claude-haiku and activityEndpoint anthropic fell through to the OpenAI run model and sent that name to Anthropic, failing every label. The destination endpoint supplies credentials, not the model choice. * 🧷 fix: Close the Remaining Edit, Epoch, and Scope Gaps for Labels SYNC clears the edit prefix on the new-row branch too. When a resumed edited submission cannot match an existing assistant row, that branch builds the response straight from the server's completion-local aggregatedContent, so it holds no retained prefix -- but the reset lived only in the matched branch, leaving later steps and labels adding an offset to indices that were already absolute. Label usage is keyed per GENERATION. Editing one assistant response reuses its responseMessageId while each fresh generation restarts activityLabelUsageSeq, so a second edit re-emitted the same runId:seq and the client discarded the newer usage while its balance transaction was still written. The key now carries jobCreatedAt, the run's own epoch: stable across reconnects and HITL resumes, distinct between generations. The scope is revalidated at commit time. Checking once before the await let a scope that closed mid-flight still charge the balance after finalization, while the matching fill saw the closed scope and dropped the label -- billed but never surfaced, the exact outcome the guard exists to prevent. The titleModel fallback no longer reaches the destination endpoint. With activityEndpoint set and no titleModel on the originating endpoint, it picked up the destination's, so changing only the credential target silently changed the model and its cost. Precedence is activityModel, then the originating endpoint's titleModel, then the run model; the destination supplies credentials only. * ✂️ refactor: Confine the Edit-Prefix Offset to Activity Labels useStepHandler is now byte-identical to dev again. The resume-aware prefix offset was applied there too, which was more correct in principle -- the post-resume prefix length is genuinely wrong for run steps as well -- but it changed index math that EVERY run step flows through, for every user, including everyone who never enables activityLabel. That shared correction needed five revisions in two days (isResume, the stream id, the response message id, clientRequestId, and the SYNC new-row branch), each passing the full suite and each failing at a boundary only review found. Carrying it inside an opt-in feature put every user behind logic with that track record. It belongs in its own change, with tests that construct the edit-plus-resume states none of the current suites reach. The offset now applies only where the label handler places its part, so this PR cannot alter rendering for anyone with the feature off. The known consequence is recorded in the description: with activity labels ENABLED, an edited response that reconnects mid-generation can place its label and its tool cards in different index spaces. That is a bug for opt-in users rather than a regression for everyone, and it disappears once the shared fix lands. submission.editPrefixLength stays: the label path still needs a prefix length that survives a SYNC replacing initialResponse.content. * 🧾 fix: Commit Labels Before Billing and Keep Blank Slots Invisible Round-nine review (all P2, feature-scoped): - Billing ordering (client.js:409, runtime.ts): usage accounting ran BEFORE the slot commit on both generation paths, so the settlement deadline could expire during the balance write — charged, then the fill dropped as out-of-scope: billed, never shown. `slot.fill` now resolves a commit flag, generators register their accounting via `deferUsage`, and the hook runs it only after a committed fill. - Scope gates (client.js:757): the direct-fallback `collect` omitted `scopeOpen`; both paths now gate on the OWNING wiring's scope, so a pre-pause straggler cannot bill because the resumed generation's scope is still open. - Blank-label grouping (groupToolCalls.ts:81): a blank slot forced a flush, splitting adjacent single-call batches into standalone cards where the feature-off path merges them. Blank labels now only mark the claim boundary — structurally invisible, while a later filled label still cannot claim an earlier batch. - Stale fill indices (wiring.ts:301): the skill-card unshift and the hide-sequential filter reshape contentParts before the finalization settle, so an in-flight fill emitted its claim-time index against a shifted array. Both completion paths now settle label fills before any post-run content reshaping (the finally settle stays as the error-path net; the second call sees an empty pending list). - Bounded serialization (runtime.ts:238): `JSON.stringify` fully materialized unbounded tool results to keep 200/600 chars per entry. A budget-bounded serializer stops at the limit (which also bounds cyclic values) and preserves the exact truncate-with-ellipsis output. Tests: fill/bill ordering + suppression on dropped fills (runtime.spec), blank-slot merging and claim boundaries (groupToolCalls.test), bounded serialization equivalence and giant-output truncation (runtime.spec). * 🧮 fix: Keep Deferred Label Billing Inside the Settle Window Self-review follow-up to the billing reorder: deferring usage until after the commit moved it PAST the fill's resolution, so a settle keyed on fills alone could let finalization flush the usage sink and snapshot metadata while the label's billing was still in flight — the usage row would silently miss the message rollup even on the happy path. The hook now reports its whole detached task (generate → fill → deferred usage) via a `trackTask` option, wired to the same settle tracker as the fills, so finalization waits for billing exactly as it did when accounting preceded the fill. The task never rejects. Pinned in runtime.spec: the tracked task resolves only after usage collection. * 🧰 fix: Harden Label Resolution, Output Bounds, and Cache Billing Round-ten review (all P2, feature-scoped); the sixth finding is the documented edited+reconnect index-space limitation, answered on-thread as deliberately out of scope for this PR. - Rejected-LLM memoization (runtime.ts): the hook cached a rejected `resolveLLM()` promise permanently, failing every later batch and silently defeating the host resolver's own rejected-cache eviction. The memo now evicts on rejection so the next batch retries. - `current_model` precedence (host.ts): an explicit `activityModel: current_model` resolved to `undefined` and then lost to a configured `titleModel`. The sentinel now resolves straight to the run model; the title fallback applies only when `activityModel` is absent. - Output bounds (runtime.ts): label text was persisted verbatim; a model ignoring the 4–9-word instruction (or steered by injection in untrusted tool output) could emit thousands of tokens duplicated through SSE, the chunk log, persistence, and the UI. `normalizeLabelOutput` keeps the first non-empty line, collapses whitespace, and hard-caps at 200 chars on both generation paths. - Cache-token billing (host.ts, client.js): the usage mapper dropped cache fields, vanishing Anthropic cache tokens from billing and charging OpenAI cache reads at the full input rate. The mapper now normalizes Anthropic/OpenAI/LangChain cache shapes into `input_token_details`, and the emit + cost path carries them with the label endpoint's `provider` (additive-provider adjustment). - Usage-type union (runs.ts): `TTokenUsageEvent.usage_type` now includes the emitted `activity-label` literal; the lone consumer keys on `usage_type != null`, so this is type-level completion. Tests: sentinel/title/explicit model precedence and all three cache shapes (host.spec), transient-resolution retry and output normalization with truncation (runtime.spec), the new usage literal (runs.spec). * 🪗 fix: Let Settled Labels Collapse Void Tools and Keep the Tail Cursor Round-eleven review (all P2, client-side). Two fixed; the other two findings restate documented Known limitations (edited+reconnect run-step index space; parallel-lane collapsible headers), answered on-thread. - Void-tool auto-collapse (ToolCallGroup.tsx): `allCompleted` keyed solely on output truthiness, so a tool that legitimately returns an empty string kept its labeled group expanded forever. A settled, filled label is itself a completion proof — the PostToolBatch claim only happens after every output in the batch returned — so it now satisfies `allCompleted`; pending labels keep the group live. - Trailing-reservation cursor (ContentParts.tsx): a blank label reservation at the content tail renders nothing but still counted as the last part, stripping the streaming cursor and last-item affordances from the last VISIBLE part until the next delta. `lastContentIdx` now walks back past empty label slots. Tests: labeled void-tool group auto-collapses, pending-label group stays expanded (ToolCallGroup.test). * 💳 fix: Price Label Cache Correctly, Honor endpoints.agents, Cancel Every Retry Round-twelve review: four fixed here; the remaining P1 (move the client.js bridge into packages/api) is an architecture call answered on-thread for the maintainer. - Provider on billed entries (client.js, P1): round ten added cache details to label usage entries but not `provider`, and `splitUsage` treats an unknown provider as additive — re-adding cache_read and cache_creation on top of an input count that already contains them, double-charging Anthropic/OpenAI cached label calls while the streamed cost (which carried the provider) disagreed. Every mapped entry now carries the label endpoint's provider. - endpoints.agents honored (host.ts, client.js): `initializeAgent` rewrites `agent.endpoint` to the backing provider, so activity settings under the PUBLIC `agents` endpoint — valid config, inherited by `agentsEndpointSchema` — were silently ignored. Field resolution is now `all` > public endpoint > backing provider/custom, applied to both the enable gate and the model/titleModel resolution. - E2E_LABEL_PORT reaches the YAML (playwright.config.mock.ts): an overridden port moved the fake label server and its health check but not the generated config's hard-coded 8889 baseURLs, so readiness passed while every label request targeted the wrong port. The override is now substituted into the generated copy. - Every retry frame cancelled (useResumableSSE.ts): concurrent label retry chains (reservation + fill per slot) overwrote one rAF handle, so cleanup cancelled only the newest chain; the rest ran up to 120 frames past unmount and could apply a stale label to a replacement generation reusing the same response id. Outstanding frame ids now live in a Set that cleanup drains. Tests: public-endpoint gate/precedence/all-above-public (host.spec). * 🖱️ fix: Keep the Last-Part Cursor in Parallel Lanes Too Round-thirteen review (single P2): `ParallelContentRenderer` computed `lastContentIdx` from the unfiltered array, so a trailing blank label reservation — filtered out of every lane — left NO rendered part carrying the last-part cursor and running-subagent affordances until the label filled. The sequential renderer's walk-back is extracted into a shared `lastVisibleContentIdx` helper (utils/activityLabels) used by both `ContentParts` and `ParallelContentRenderer`, so the two index spaces cannot drift again. Behavior pinned in activityLabels.spec: trailing blank skipped, consecutive blanks skipped, filled label counts, label-free content unchanged. * 🧹 chore: Alias the Retry-Frame Set for the Effect Cleanup Lint Rule * 📏 fix: Let activityCharLimit Reach Tool Inputs Round-fifteen review: `activityCharLimit` is documented as the per-entry truncation for tool input AND output, but `buildPrompt` hard-coded inputs at 200 characters — so raising the setting could never surface a distinguishing path, query, or operation that appears past the first 200 characters of a long argument. Inputs now truncate at the configured limit alongside outputs; the 200-char constant remains only for the intent line (renamed INTENT_CHAR_LIMIT to match). Config fidelity pinned in runtime.spec: a 400-char argument survives a 450 limit and truncates under a 50 limit. The round's other finding is the fifth restatement of the documented edited+reconnect index-space limitation, answered on-thread with the prior four cross-references. * 🤝 fix: No Labels for Pure Handoff Batches Round-sixteen review: a PostToolBatch containing only `transfer_to_*` calls claimed a label slot, but transfer parts are never groupable — the client flushed the handoff card standalone and the label orphaned into a stray line after it, restating what the card already says. Two-sided fix: - Hook (runtime.ts): a batch whose every entry is a transfer call claims nothing — no slot, no model call, no `maxPerRun` consumption. Mixed batches still label (the header describes the real work). - Renderer (groupToolCalls.ts): an orphan label whose `tool_call_ids` are all transfer calls is dropped instead of rendered standalone, covering content persisted before the hook-side skip. The round's two P1s are repeats answered on-thread: the packages/api extraction (maintainer-decided follow-up, recorded in the description) and the sixth restatement of the edited+reconnect index limitation. Tests: transfer-only batch claims nothing, mixed batch still claims (runtime.spec); transfer-only orphan label dropped, real-batch orphan label still renders (groupToolCalls.test). * 🎛️ fix: Sanitize Label Client Options and Bound the Batch Prompt Round-seventeen review: two fixed; the other two findings repeat the maintainer-decided packages/api extraction (follow-up) and the edited+reconnect index limitation (seventh instance), answered on-thread. - Primary-option strip (host.ts): the label client copied the resolved `llmConfig` wholesale, so an endpoint whose defaults enable extended thinking or carry model-specific output caps forwarded them to the (often cheaper) label model — unsupported options failed every label, and supported thinking spent real tokens and the settlement window on a 4–9 word header. The copy now strips `omitTitleOptions` keys and the `modelKwargs` output caps exactly like the title path, restoring the Anthropic `clientOptions` carrier by reference so proxy `defaultHeaders` still reach label requests. - Batch prompt budget (runtime.ts): per-entry truncation left the batch dimension unbounded — hundreds of parallel calls could build a prompt past the fast model's window. The entries section now has a total budget (8k chars, scaling with `activityCharLimit` so a raised limit still fits several entries); entries past it are skipped without paying their serialization cost, and the list notes how many were omitted. The first entry always renders in full. Tests: option strip with header-carrier survival (host.spec); giant batch bounded with omission marker, small batch untouched (runtime.spec). * 🛡️ fix: Keep SSRF Guards on Label Calls, Skip Mixed Handoff Batches Round-eighteen review: four fixed; the fifth repeats the maintainer-decided packages/api extraction (eighth instance), answered on-thread. - SSRF-safe carrier (host.ts, P1): the sanitize step restored the Anthropic `clientOptions` carrier only when `defaultHeaders` existed, but for user-provided base URLs `getLLMConfig` stores the guarded Undici dispatcher and `redirect: 'error'` there — dropping it reopened DNS-rebinding/redirect paths on label calls to user-controlled URLs. The carrier (client CONSTRUCTION options, not generation params) is now restored whenever present, same reference. - Primary maxTokens (host.ts): top-level `maxTokens` is not in `omitTitleOptions` and survived the strip; the title path deletes it explicitly, and a cap sized for the primary model can be rejected by the substitute. Deleted on the copy. - Bounded keys (runtime.ts): the object branch materialized every key via `Object.keys` and quoted oversized keys in full before the budget check. Enumeration is now lazy (`for..in` + own-property guard) and keys slice to the budget before quoting, like string values. - Mixed handoff batches (runtime.ts, groupToolCalls.ts): the client flushes the block at the transfer card, so a mixed batch's label orphaned exactly like a pure one. The hook now skips ANY batch containing a transfer call, and the renderer drops orphan labels covering one (legacy content). Tests: carrier survival without headers by same reference, maxTokens strip (host.spec); mixed batch claims nothing (runtime.spec); mixed orphan dropped, real-batch orphan kept (groupToolCalls.test). * 🧢 fix: Cap Label Generation, Order the Flag Persist, Detach Settled Listeners Round-nineteen review: three fixed; the fourth is the ninth instance of the edited+reconnect index limitation, answered on-thread. - Generation cap (host.ts): stripping the primary output caps left label calls with NO cap at all — `normalizeLabelOutput` bounds what persists, not what the provider generates and bills, so a model ignoring the 4–9-word instruction (or steered by injected tool output) could emit its provider-default output per batch. The sanitize step now installs a 256-token label cap (per provider family: `maxOutputTokens` for Google-style wrappers, `maxTokens` otherwise), after the filter so the omit set cannot remove it. - Flag-persist ordering (client.js): the `markActivityLabels` write was fire-and-forget, so an immediate cross-replica reconnect could read the job between the write and the first claim, see neither flag nor snapshot label, and skip gap reconciliation. Label emission now awaits the (settled-on-failure) persist chain, making "a label event exists" imply "the flag is durable" — the race window is gone; only the documented double-write-failure residual remains. - Listener detach (client.js): each HITL approval cycle's wiring adds a `once` abort listener to the shared job signal that only an actual abort removes; settled segments now detach theirs in `settleActivityLabels`, so long multi-approval runs cannot accumulate dead closures toward the listener-limit warning. Tests: the primary cap is REPLACED by the 256-token label cap (host.spec). * 🎯 fix: Route the Label Cap Per Model Family Round-twenty review: the 256-token label cap set maxTokens unconditionally, but GPT-5+ rejects max_tokens (the OpenAI builder routes its cap into modelKwargs.max_completion_tokens / max_output_tokens) and o-series models reject it with no stable kwargs alternative — every label on those models would have failed. The cap now mirrors the builder: modelKwargs for GPT-5+ (responses-API aware), no cap for o-series (title parity; the 200-char persistence bound still applies), maxOutputTokens for Google, maxTokens otherwise. Pinned in host.spec for both reasoning families. The round's other finding is the tenth instance of the documented edited+reconnect index limitation, answered on-thread. * ⏱️ fix: Persist the Label Flag at Run Start, Not on the Emit Path Round-twenty-one review: two fixed; the other three repeat the maintainer-decided packages/api extraction, the edited+reconnect index limitation, and the parallel-lane header limitation — all answered on-thread with their standing decisions. - Flag ordering, corrected (client.js): sequencing label emission behind the flag persist (previous round) delayed the claim-time reservation while the shared index offset had ALREADY shifted subsequent SDK chunks — reopening the cross-instance hole-compaction overwrite the reservation emit exists to prevent. The reservation emits immediately again; instead, run start (processStream and resume alike) awaits the settled-on-failure persist chain, so the flag is durable before any batch can claim a label. Same guarantee, zero latency on the emit path. - Tail-label cursor (ContentParts.tsx): a filled label at the content tail is consumed into the group header rather than listed in `group.parts`, so the `isLast` check missed it and nothing held the streaming cursor until the next delta. The check now includes `labelPart.idx`. * 🔌 fix: Detach Label Abort Listeners Even Without Claims A segment with labels enabled can end without a single claim (text-only, or handoff batches, which skip labels); the early return in settleActivityLabels skipped the detach added for HITL listener accumulation. The detach now runs on both paths. * ⚖️ fix: Make the Commit Flag the Sole Billing Authority Round-twenty-three review: a committed fill racing a late scope close (user abort or settle timeout during the durable emit) stayed visible — the part is mutated and persisted before the close — yet the deferred accounting's scope gates then skipped the charge: a completed provider call escaping both the label charge and the primary abort accounting. The scope gates on the deferred-usage path are removed; the hook's commit flag is now the single billing authority in BOTH directions. A dropped fill never reaches the accounting callback (billed-never-shown stays impossible), and a committed fill bills regardless of when its scope closed (shown-never-billed now impossible too). The dead `scopeOpen` payload threading is removed with it; the `recordActivityLabelUsage` parameter survives, defaulting open, for callers that own no commit signal. The round's other finding is the twelfth instance of the documented edited+reconnect index limitation, answered on-thread. * 🧮 feat: Bill Labels by Estimate When Providers Omit Usage Maintainer decision: follow the title convention rather than leaving label calls unbilled when a provider returns no usage metadata. The hook now passes a LAZY estimate thunk with the deferred accounting on the success path — the EXACT prompt the direct path sent (or the locally built equivalent for the SDK path: same entries, context, instruction, truncation contract, and continuity headers) plus the final normalized label. `recordActivityLabelUsage` invokes it only when no collected entry carries a real token count, counts both texts with the shared o200k_base tokenizer, and feeds the synthesized entry through the SAME pipeline (provider-tagged, streamed event, cost, balance transaction). Real provider usage always wins when present. The failure path passes NO estimate: a throw before a response bills only real collected metadata, never a full phantom prompt. Tests: the estimate thunk carries the exact invoked prompt and final label; the failure path defers with no estimate (runtime.spec). * 💵 fix: Estimate From the Raw Completion, Not the Normalized Label The fallback estimate counted the normalized label (first line, 200-char cap) while the provider generated and would bill the raw output up to the 256-token generation cap — under-recording verbose replies. The estimate thunk now carries the raw pre-normalization text; the persisted label is unchanged. Pinned with a multi-line reply test. * 🧾 fix: Commit Label Text Only After the Durable Emit, Estimate the Real SDK Prompt Round review on the billing work: two fixed; the third is the fourteenth instance of the edited+reconnect index limitation, answered on-thread. - Copy-first fill (wiring.ts): the fill mutated the shared content part BEFORE its durable emit, so a failed emit left the label text on `contentParts` anyway — persistence could save and display a label no client ever received and billing (keyed on the commit flag) never charged. The new state is staged on a copy; the shared part mutates only after the emit succeeds, so content, delivery, and billing move together. - Real SDK prompt for estimates (client.js): the estimate thunk carried this module's locally built prompt, but the SDK path frames entries differently — the estimated input count was for a prompt never sent. Chain-start callbacks (handleLLMStart/handleChatModelStart) now capture the prompt the SDK actually rendered, and the deferred accounting substitutes it into the estimate when capture succeeded, falling back to the local approximation otherwise.
3314 lines
133 KiB
JavaScript
3314 lines
133 KiB
JavaScript
require('events').EventEmitter.defaultMaxListeners = 100;
|
|
const { logger } = require('@librechat/data-schemas');
|
|
const { getBufferString, HumanMessage } = require('@librechat/agents/langchain/messages');
|
|
const {
|
|
createRun,
|
|
isEnabled,
|
|
checkAccess,
|
|
buildToolSet,
|
|
logToolError,
|
|
sanitizeTitle,
|
|
payloadParser,
|
|
createSafeUser,
|
|
initializeAgent,
|
|
resolveConfigHeaders,
|
|
countTokens,
|
|
getBalanceConfig,
|
|
omitTitleOptions,
|
|
getProviderConfig,
|
|
memoryInstructions,
|
|
createTokenCounter,
|
|
applyContextToAgent,
|
|
isMemoryAgentEnabled,
|
|
recordCollectedUsage,
|
|
sendEvent,
|
|
computeUsageCostUSD,
|
|
aggregateEmittedUsage,
|
|
resolveAgentTokenConfig,
|
|
buildPersistedContextUsage,
|
|
computeSummaryUsedTokens,
|
|
priorRunOutputTokens,
|
|
createSubagentUsageSink,
|
|
anyAgentReplaysReasoningContent,
|
|
GenerationJobManager,
|
|
getTransactionsConfig,
|
|
resolveRecursionLimit,
|
|
buildPendingAction,
|
|
toClientPendingAction,
|
|
computeAgentRequestFingerprint,
|
|
extractDiscoveredToolsFromHistory,
|
|
captureResumeModelParameters,
|
|
pickResumeContext,
|
|
getApprovalTtlMs,
|
|
isHITLEnabled,
|
|
deleteAgentCheckpoint,
|
|
agentRequestsAskUserQuestion,
|
|
attachAskUserQuestionArgs,
|
|
hydrateResumeRunSteps,
|
|
createContentIndexOffsetHandlers,
|
|
createSteerIndexOffsetHandlers,
|
|
createSteerDrainHook,
|
|
isSteeringSupported,
|
|
buildSteerMedia,
|
|
stampSteerPartMedia,
|
|
createActivityLabelWiring,
|
|
resolveActivityConfig,
|
|
getCustomEndpointConfig,
|
|
mapCollectedMetadataToUsage,
|
|
resolveActivityLabelModel,
|
|
settlePendingLabelFills,
|
|
stripActivityLabelParts,
|
|
getRequestMemories,
|
|
getMemoryAgentId,
|
|
createMemoryProcessor,
|
|
agentHasInlineMemoryTools,
|
|
loadAgent: loadAgentFn,
|
|
createMultiAgentMapper,
|
|
filterMalformedContentParts,
|
|
countFormattedMessageTokens,
|
|
prependFileContext,
|
|
prependQuotes,
|
|
hydrateMissingIndexTokenCounts,
|
|
injectSkillPrimes,
|
|
collectFreshSkillPrimeNames,
|
|
isSkillPrimeMessage,
|
|
collectFileIds,
|
|
processTextWithTokenLimit,
|
|
buildAgentScopedContext,
|
|
buildSkillPrimeContentParts,
|
|
buildInitialToolSessions,
|
|
hasUrlContextTool,
|
|
hasYouTubeVideoParts,
|
|
appendYouTubeVideoParts,
|
|
resolveGoogleVideoError,
|
|
resolveYouTubeInjectionConfig,
|
|
decrementPendingRequest,
|
|
maybePrewarmCodeSandbox,
|
|
} = require('@librechat/api');
|
|
const {
|
|
Run,
|
|
Callback,
|
|
Providers,
|
|
TitleMethod,
|
|
formatMessage,
|
|
formatAgentMessages,
|
|
createMetadataAggregator,
|
|
} = require('@librechat/agents');
|
|
const {
|
|
Constants,
|
|
SteerEvents,
|
|
ActivityLabelEvents,
|
|
UsageEvents,
|
|
Permissions,
|
|
VisionModes,
|
|
ContentTypes,
|
|
ApprovalEvents,
|
|
EModelEndpoint,
|
|
PermissionTypes,
|
|
AgentCapabilities,
|
|
isAgentsEndpoint,
|
|
isEphemeralAgentId,
|
|
removeNullishValues,
|
|
DEFAULT_MEMORY_MAX_INPUT_TOKENS,
|
|
} = require('librechat-data-provider');
|
|
const { filterFilesByAgentAccess } = require('~/server/services/Files/permissions');
|
|
const { encodeAndFormat } = require('~/server/services/Files/images/encode');
|
|
const { createContextHandlers } = require('~/app/clients/prompts');
|
|
const { resolveConfigServers } = require('~/server/services/MCP');
|
|
const { getMCPServerTools } = require('~/server/services/Config');
|
|
const BaseClient = require('~/app/clients/BaseClient');
|
|
const { getMCPManager } = require('~/config');
|
|
const db = require('~/models');
|
|
|
|
const loadAgent = (params) => loadAgentFn(params, { getAgent: db.getAgent, getMCPServerTools });
|
|
|
|
const MEMORY_INPUT_CHARS_PER_TOKEN = 8;
|
|
|
|
class AgentClient extends BaseClient {
|
|
constructor(options = {}) {
|
|
super(null, options);
|
|
/** The current client class
|
|
* @type {string} */
|
|
this.clientName = EModelEndpoint.agents;
|
|
|
|
/** @deprecated @type {true} - Is a Chat Completion Request */
|
|
this.isChatCompletion = true;
|
|
/** @type {number | undefined} */
|
|
this.jobCreatedAt = options.jobCreatedAt;
|
|
|
|
/** @type {AgentRun} */
|
|
this.run;
|
|
|
|
/** Resolves with the agent run once `chatCompletion` initializes it (or
|
|
* `null` if initialization fails), letting immediate-mode title generation
|
|
* await the run instead of throwing when fired before the run exists.
|
|
* @type {Promise<AgentRun | null> | null} */
|
|
this._runReady = null;
|
|
/** @type {((run: AgentRun | null) => void) | null} */
|
|
this._resolveRun = null;
|
|
|
|
const {
|
|
agentConfigs,
|
|
contentParts,
|
|
stepMap,
|
|
collectedUsage,
|
|
collectedThoughtSignatures,
|
|
artifactPromises,
|
|
maxContextTokens,
|
|
subagentAggregatorsByToolCallId,
|
|
contextUsageSink,
|
|
usageEmitSink,
|
|
toolInputValidationErrors,
|
|
...clientOptions
|
|
} = options;
|
|
|
|
this.agentConfigs = agentConfigs;
|
|
this.maxContextTokens = maxContextTokens;
|
|
/** Latest visible context snapshot for this response, captured live by the
|
|
* ON_CONTEXT_USAGE handler; persisted on `metadata.contextUsage`.
|
|
* @type {{ latest: import('librechat-data-provider').TContextUsageEvent | null } | undefined} */
|
|
this.contextUsageSink = contextUsageSink;
|
|
/** Every emitted `on_token_usage` payload for this response (primary,
|
|
* summarization, sequential, and subagent); aggregated into the rollup
|
|
* persisted on `metadata.usage`.
|
|
* @type {Array<import('librechat-data-provider').TTokenUsageEvent> | undefined} */
|
|
this.usageEmitSink = usageEmitSink;
|
|
/** Schema-validation exceptions keyed by tool-call ID. The completion
|
|
* handler consumes these to distinguish execution failures from tool
|
|
* output that merely contains similar text.
|
|
* @type {Map<string, import('@librechat/api').ToolInputValidationError> | undefined} */
|
|
this.toolInputValidationErrors = toolInputValidationErrors;
|
|
/** @type {MessageContentComplex[]} */
|
|
this.contentParts = contentParts;
|
|
/** Original run-step identity used by the content aggregator to attach
|
|
* completion events to their rendered content indices.
|
|
* @type {Map<string, import('@librechat/agents').RunStep | undefined> | undefined} */
|
|
this.stepMap = stepMap;
|
|
/** @type {Array<UsageMetadata>} */
|
|
this.collectedUsage = collectedUsage;
|
|
/** Vertex Gemini 3 thought signatures captured during the run, keyed by
|
|
* `tool_call_id`. Persisted on `responseMessage.metadata.thoughtSignatures`
|
|
* and restored as `additional_kwargs.signatures` on subsequent turns to
|
|
* keep tool round-trips valid across DB reconstruction.
|
|
* @type {Record<string, string> | undefined} */
|
|
this.collectedThoughtSignatures = collectedThoughtSignatures;
|
|
/** @type {ArtifactPromises} */
|
|
this.artifactPromises = artifactPromises;
|
|
/** Per-request map of `createContentAggregator` instances keyed by
|
|
* the parent's `tool_call_id`. `ON_SUBAGENT_UPDATE` events stream
|
|
* into each aggregator as they arrive; `finalizeSubagentContent`
|
|
* harvests `contentParts` onto the matching `subagent` tool_call
|
|
* so the child's full activity survives a page refresh. */
|
|
this.subagentAggregatorsByToolCallId = subagentAggregatorsByToolCallId ?? new Map();
|
|
/** In-flight `on_token_usage` emits from subagent child runs. The sink
|
|
* fires the emitter without awaiting, so chatCompletion's finally flushes
|
|
* these before returning — otherwise job cleanup can race the persist.
|
|
* @type {Promise<void>[]} */
|
|
this.pendingSubagentEmits = [];
|
|
/** @type {AgentClientOptions} */
|
|
this.options = Object.assign({ endpoint: options.endpoint }, clientOptions);
|
|
/** @type {string} */
|
|
this.model = this.options.agent.model_parameters.model;
|
|
/** The key for the usage object's input tokens
|
|
* @type {string} */
|
|
this.inputTokensKey = 'input_tokens';
|
|
/** The key for the usage object's output tokens
|
|
* @type {string} */
|
|
this.outputTokensKey = 'output_tokens';
|
|
/** @type {UsageMetadata} */
|
|
this.usage;
|
|
/** @type {Record<string, number>} */
|
|
this.indexTokenCountMap = {};
|
|
/** @type {Array<Record<string, unknown>> | null} */
|
|
this.memoryPayload = null;
|
|
/** Mutable content-index shift shared with the steer offset handlers.
|
|
* Incremented each time a steer part is spliced into `contentParts`, so
|
|
* SDK-emitted indices that arrive after an injection land past it.
|
|
* @type {import('@librechat/api').SteerOffsetState} */
|
|
this.steerOffsetState = { offset: 0 };
|
|
/** @type {(messages: BaseMessage[]) => Promise<void>} */
|
|
this.processMemory;
|
|
}
|
|
|
|
/**
|
|
* Returns the aggregated content parts for the current run.
|
|
* @returns {MessageContentComplex[]} */
|
|
getContentParts() {
|
|
return this.contentParts;
|
|
}
|
|
|
|
/**
|
|
* Harvest the `contentParts` from each per-subagent `createContentAggregator`
|
|
* instance and attach them onto the matching parent `subagent` tool_call
|
|
* as `subagent_content`. Runs once per message save (from
|
|
* `sendCompletion`'s `finally`) so the child's full reasoning / tool
|
|
* calls / final text survive a page refresh — the client-side Recoil
|
|
* atom is session-only. Aggregators keyed by a tool_call_id that never
|
|
* appeared in `contentParts` are discarded (no home to attach to).
|
|
*/
|
|
finalizeSubagentContent() {
|
|
const buffer = this.subagentAggregatorsByToolCallId;
|
|
if (!buffer || buffer.size === 0 || !Array.isArray(this.contentParts)) {
|
|
return;
|
|
}
|
|
for (const part of this.contentParts) {
|
|
if (part?.type !== ContentTypes.TOOL_CALL) continue;
|
|
const toolCall = part[ContentTypes.TOOL_CALL];
|
|
if (!toolCall || toolCall.name !== Constants.SUBAGENT || !toolCall.id) continue;
|
|
const aggregator = buffer.get(toolCall.id);
|
|
if (!aggregator) continue;
|
|
try {
|
|
/** `createContentAggregator` returns a sparse array (undefined
|
|
* slots for indices that never received content). Strip those
|
|
* so the persisted shape is a clean `TMessageContentParts[]`. */
|
|
const parts = Array.isArray(aggregator.contentParts)
|
|
? aggregator.contentParts.filter((p) => p != null)
|
|
: [];
|
|
if (parts.length > 0) {
|
|
toolCall.subagent_content = parts;
|
|
}
|
|
} catch (err) {
|
|
logger.warn(
|
|
`[AgentClient] Failed to attach subagent content for tool_call ${toolCall.id}: ${err?.message ?? err}`,
|
|
);
|
|
}
|
|
}
|
|
buffer.clear();
|
|
}
|
|
|
|
/**
|
|
* Apply one drained steer to host state: append the steer content part at
|
|
* the live content index, bump the shared index offset so subsequent SDK
|
|
* step indices land past it, and emit `on_steer_applied` so the live client
|
|
* replaces its pending chip with the inline part (the emitted chunk also
|
|
* reaches the Redis chunk log for reconnect reconstruction).
|
|
*
|
|
* Runs BEFORE the drain hook's media encode so an abort during the encode
|
|
* cannot lose the steer. File refs persist from the queue item (sanitized at
|
|
* enqueue); replay/token accounting re-fetch owner-scoped and re-encode per
|
|
* turn (stampSteerPartMedia), so unauthorized ids drop out there.
|
|
*
|
|
* @param {string} streamId
|
|
* @param {import('@librechat/api').SteerQueueItem} item
|
|
*/
|
|
async applySteerPart(streamId, item) {
|
|
const index = this.contentParts.length;
|
|
const part = {
|
|
type: ContentTypes.STEER,
|
|
[ContentTypes.STEER]: item.text,
|
|
steerId: item.steerId,
|
|
createdAt: item.createdAt,
|
|
...(item.files?.length && { files: item.files }),
|
|
};
|
|
this.contentParts.push(part);
|
|
this.steerOffsetState.offset += 1;
|
|
// durable: the chunk-log XADD is this event's recovery record — it must
|
|
// commit before the publish or a cross-replica reconnect that missed the
|
|
// pub/sub delivery reconstructs content without the steer part.
|
|
await GenerationJobManager.emitChunk(
|
|
streamId,
|
|
{
|
|
event: SteerEvents.ON_STEER_APPLIED,
|
|
data: {
|
|
steerId: item.steerId,
|
|
index,
|
|
part,
|
|
responseMessageId: this.responseMessageId,
|
|
conversationId: this.conversationId,
|
|
},
|
|
},
|
|
{ durable: true, expectedCreatedAt: this.jobCreatedAt },
|
|
);
|
|
}
|
|
|
|
/**
|
|
* The `steering` fragment for `createRun`: the run-scoped PostToolBatch
|
|
* drain hook, or `undefined` when there is no resumable job surface or the
|
|
* installed SDK cannot inject hook messages (draining would drop them).
|
|
*
|
|
* @param {string | undefined} streamId
|
|
*/
|
|
buildSteerWiring(streamId) {
|
|
if (!streamId || !isSteeringSupported()) {
|
|
return undefined;
|
|
}
|
|
return {
|
|
hook: createSteerDrainHook({
|
|
streamId,
|
|
jobCreatedAt: this.jobCreatedAt,
|
|
applySteer: (item) => this.applySteerPart(streamId, item),
|
|
buildMedia: (item) =>
|
|
buildSteerMedia({
|
|
client: this,
|
|
user: this.options.req?.user,
|
|
item,
|
|
getFiles: db.getFiles,
|
|
}),
|
|
}),
|
|
};
|
|
}
|
|
|
|
setOptions(_options) {}
|
|
|
|
/**
|
|
* Resolve provider + client options for the
|
|
* tool-batch summary model. Same resolution path as titleConvo minus the
|
|
* title-specific branches. Model precedence: the endpoint's
|
|
* `activityModel` > its `titleModel` > the agent's own model, on the
|
|
* endpoint named by `activityEndpoint` (default: the agent's).
|
|
*/
|
|
async resolveActivityLabelLLM() {
|
|
/** Memoized per response: resolution reads provider config and can hit the
|
|
* database for user keys, and nothing it depends on changes between
|
|
* batches of the same run — so re-resolving on every batch (twice, with
|
|
* usage accounting) is repeated credential work for an identical result.
|
|
* The promise is cached rather than the value so concurrent batches share
|
|
* one in-flight resolution. */
|
|
this.activityLabelLLMPromise =
|
|
this.activityLabelLLMPromise ??
|
|
resolveActivityLabelModel({
|
|
req: this.options.req,
|
|
agent: this.options.agent,
|
|
/** Same public-endpoint-first field resolution as the wiring gate. */
|
|
publicEndpoint: this.options.endpoint,
|
|
ids: {
|
|
messageId: this.responseMessageId,
|
|
conversationId: this.conversationId,
|
|
parentMessageId: this.parentMessageId,
|
|
},
|
|
db: { getUserKey: db.getUserKey, getUserKeyValues: db.getUserKeyValues },
|
|
}).catch((error) => {
|
|
/** Never cache a rejection: a transient credential read failure would
|
|
* otherwise disable labels for the rest of the response. */
|
|
this.activityLabelLLMPromise = null;
|
|
throw error;
|
|
});
|
|
return this.activityLabelLLMPromise;
|
|
}
|
|
|
|
/**
|
|
* Bills the label call and folds its usage into the response rollup with
|
|
* an `activity-label` tag (subagent precedent) so `metadata.usage` and the
|
|
* live cost gauge reflect it. Tagged, so it is not a PRIMARY usage event
|
|
* and cannot disturb the context-snapshot pairing in buildResponseMetadata.
|
|
*/
|
|
async recordActivityLabelUsage(
|
|
collectedMetadata,
|
|
model,
|
|
endpointTokenConfig,
|
|
sameEndpoint,
|
|
/** Optional suppression gate, defaulting open. The hook-driven paths
|
|
* deliberately pass nothing: they invoke accounting ONLY for a
|
|
* COMMITTED fill, and a committed (visible) label must bill even when
|
|
* its scope closed during the durable emit — the commit flag, not the
|
|
* scope, is the billing authority. */
|
|
scopeOpen = () => true,
|
|
/** The LABEL endpoint's provider — cost math needs it to know whether
|
|
* cache tokens are folded into `input_tokens` (additive providers like
|
|
* Bedrock keep them separate). */
|
|
provider = undefined,
|
|
/** Lazy `() => ({ promptText, completionText })` fallback. When the
|
|
* provider omits usage metadata entirely, labels bill by ESTIMATE —
|
|
* the title convention — from locally counted text rather than going
|
|
* unbilled. Invoked only when no entry carries a real token count. */
|
|
estimate = undefined,
|
|
) {
|
|
const appConfig = this.options.req?.config;
|
|
/** Provider ON EVERY ENTRY, not just the streamed event: `splitUsage`
|
|
* keys additive-vs-subset cache math on `usage.provider`, and an
|
|
* unknown provider takes the additive branch — for Anthropic/OpenAI
|
|
* (cache already inside `input_tokens`) that re-adds cache_read and
|
|
* cache_creation on top, double-charging the balance while the
|
|
* streamed cost (which carries the provider) disagrees. */
|
|
let collectedUsage = mapCollectedMetadataToUsage(collectedMetadata).map((usage) =>
|
|
provider != null ? { ...usage, provider } : usage,
|
|
);
|
|
const hasRealUsage = collectedUsage.some(
|
|
(usage) => usage.input_tokens != null || usage.output_tokens != null,
|
|
);
|
|
if (!hasRealUsage && typeof estimate === 'function') {
|
|
try {
|
|
const { promptText = '', completionText = '' } = estimate() ?? {};
|
|
const [input_tokens, output_tokens] = await Promise.all([
|
|
countTokens(promptText),
|
|
countTokens(completionText),
|
|
]);
|
|
collectedUsage = [
|
|
provider != null
|
|
? { input_tokens, output_tokens, provider }
|
|
: { input_tokens, output_tokens },
|
|
];
|
|
} catch (err) {
|
|
logger.warn(
|
|
`[AgentClient] Failed to estimate activity-label usage: ${err?.message ?? err}`,
|
|
);
|
|
}
|
|
}
|
|
if (
|
|
collectedUsage.length === 0 ||
|
|
!collectedUsage.some((usage) => usage.input_tokens != null || usage.output_tokens != null)
|
|
) {
|
|
return;
|
|
}
|
|
if (!scopeOpen()) {
|
|
return;
|
|
}
|
|
const streamId = this.options.req?._resumableStreamId || null;
|
|
const includeCost = this.options.req?.config?.interfaceConfig?.contextCost === true;
|
|
/** Cross-endpoint labels (`activityEndpoint`) price with THEIR endpoint's
|
|
* rates. `undefined` is a MEANINGFUL result for a built-in label endpoint
|
|
* (built-ins price from the shared table, not a per-endpoint map), so it
|
|
* must not fall through to the agent's custom rates — a custom primary
|
|
* pointing `activityEndpoint` at a built-in would bill the label at its
|
|
* own rates. Only inherit when the label actually runs on the agent's
|
|
* endpoint. */
|
|
const labelTokenConfig = sameEndpoint
|
|
? (endpointTokenConfig ?? this.options.endpointTokenConfig)
|
|
: endpointTokenConfig;
|
|
for (const usage of collectedUsage) {
|
|
/** `seq` is normally a position in `collectedUsage` (each emitter
|
|
* pushes, then emits with the new length). Label usage is billed
|
|
* separately and never appended there, so it has no position: any
|
|
* positive value eventually collides with a real one, and the client
|
|
* dedupes on `runId:seq`. Labels therefore occupy a NEGATIVE seq
|
|
* namespace that positional sequences can never reach. The key is
|
|
* only used for Set membership, so the sign is inert. */
|
|
this.activityLabelUsageSeq = (this.activityLabelUsageSeq ?? 0) + 1;
|
|
const data = {
|
|
input_tokens: usage.input_tokens,
|
|
output_tokens: usage.output_tokens,
|
|
/** Cache tokens ride along (subagent-event shape) so display and
|
|
* aggregation price cached label calls at cache rates. */
|
|
...(usage.input_token_details != null && {
|
|
input_token_details: usage.input_token_details,
|
|
}),
|
|
...(provider != null && { provider }),
|
|
model,
|
|
usage_type: 'activity-label',
|
|
/**
|
|
* Scoped to the GENERATION, not just the response. Editing one
|
|
* assistant response reuses its `responseMessageId` while each fresh
|
|
* server generation restarts `activityLabelUsageSeq`, so a second
|
|
* edit re-emitted `<responseId>:-1` and the client — which dedupes on
|
|
* exactly `runId:seq` — discarded the newer usage even though its
|
|
* balance transaction was still written. `jobCreatedAt` is the run's
|
|
* own epoch: stable across reconnects and HITL resumes of one
|
|
* generation, distinct between generations.
|
|
*/
|
|
runId:
|
|
this.jobCreatedAt != null
|
|
? `${this.responseMessageId}:${this.jobCreatedAt}`
|
|
: this.responseMessageId,
|
|
seq: -this.activityLabelUsageSeq,
|
|
/** Cost coverage is all-or-nothing in `aggregateEmittedUsage`: an
|
|
* event without `cost` suppresses the whole response's cost when
|
|
* `interface.contextCost` is on. */
|
|
cost: includeCost
|
|
? computeUsageCostUSD(
|
|
{ ...usage, model, provider },
|
|
{ getMultiplier: db.getMultiplier, getCacheMultiplier: db.getCacheMultiplier },
|
|
labelTokenConfig,
|
|
)
|
|
: undefined,
|
|
};
|
|
/** Fold into the response rollup synchronously, then stream it like
|
|
* primary/subagent usage so the live session gauge stays honest.
|
|
* Retained and flushed with the subagent emits so job cleanup cannot
|
|
* race the persist. */
|
|
this.usageEmitSink?.push(data);
|
|
if (streamId) {
|
|
const emit = GenerationJobManager.emitChunk(
|
|
streamId,
|
|
{
|
|
event: UsageEvents.ON_TOKEN_USAGE,
|
|
data,
|
|
},
|
|
/** Same epoch scoping as the label event: this usage is recorded
|
|
* from a detached generation and must not bill against whichever
|
|
* generation replaced it. */
|
|
{ expectedCreatedAt: this.jobCreatedAt },
|
|
).catch((err) => {
|
|
logger.warn(`[AgentClient] Failed to emit activity-label usage: ${err?.message ?? err}`);
|
|
});
|
|
this.pendingSubagentEmits.push(emit);
|
|
}
|
|
}
|
|
await this.recordCollectedUsage({
|
|
collectedUsage,
|
|
context: 'activity-label',
|
|
model,
|
|
endpointTokenConfig: labelTokenConfig,
|
|
/** The label ran elsewhere, so its config governs even when undefined. */
|
|
crossEndpoint: sameEndpoint === false,
|
|
balance: getBalanceConfig(appConfig),
|
|
transactions: getTransactionsConfig(appConfig),
|
|
messageId: this.responseMessageId,
|
|
/** Billed, but NOT the response's stream usage — see the parameter. */
|
|
updateStreamUsage: false,
|
|
}).catch((err) => {
|
|
logger.error(
|
|
'[api/server/controllers/agents/client.js #recordActivityLabelUsage] Error recording usage',
|
|
err,
|
|
);
|
|
});
|
|
}
|
|
|
|
/**
|
|
* Bridges label generation to the SDK's `run.generateActivityLabel()` so
|
|
* the fast-model call is Langfuse-traced under the conversation's session
|
|
* (thread_id) with its own tags — never as an orphan trace. Returns null
|
|
* when the label could not be generated.
|
|
*/
|
|
async generateActivityLabelViaRun({
|
|
entries,
|
|
context,
|
|
previousLabels,
|
|
traceSeed,
|
|
signal,
|
|
charLimit,
|
|
prompt,
|
|
executingAgentId,
|
|
deferUsage,
|
|
}) {
|
|
/** Version gating happens at wiring time via the `sdkCapable` prototype
|
|
* probe, so this only catches a run that is missing or not yet built.
|
|
* Resolve `undefined` (not `null`) so the hook reads it as "this path
|
|
* cannot serve the request" and falls back to the direct model call;
|
|
* `null` would mean "ran, produced no label" and would leave the slot
|
|
* permanently empty. */
|
|
if (typeof this.run?.generateActivityLabel !== 'function') {
|
|
return undefined;
|
|
}
|
|
const { provider, clientOptions, endpointTokenConfig, sameEndpoint } =
|
|
await this.resolveActivityLabelLLM();
|
|
const { handleLLMEnd, collected: collectedMetadata } = createMetadataAggregator();
|
|
/**
|
|
* NO scope gate here: the hook invokes this ONLY for a COMMITTED fill,
|
|
* and the commit flag is the single billing authority. A scope that
|
|
* closes while the fill's durable emit is in flight does not un-commit
|
|
* the label — it is persisted and visible — so gating on the scope here
|
|
* turned that race into a completed provider call escaping both the
|
|
* label charge and the primary abort accounting. The reverse direction
|
|
* (billed but never shown) is enforced by the commit gate itself: a
|
|
* dropped fill never reaches this callback.
|
|
*/
|
|
/**
|
|
* The PROMPT THE SDK ACTUALLY SENT, captured at chain start. The hook's
|
|
* estimate thunk carries this module's locally built prompt — same
|
|
* entries and instruction but different framing — so estimated billing
|
|
* on this path would count a prompt that was never sent. When capture
|
|
* succeeded, it replaces the thunk's promptText.
|
|
*/
|
|
let sdkPromptText;
|
|
const capturePrompt = {
|
|
handleLLMStart: (_llm, prompts) => {
|
|
sdkPromptText = Array.isArray(prompts) ? prompts.join('\n') : undefined;
|
|
},
|
|
handleChatModelStart: (_llm, messages) => {
|
|
try {
|
|
sdkPromptText = (messages ?? [])
|
|
.flat()
|
|
.map((message) =>
|
|
typeof message?.content === 'string'
|
|
? message.content
|
|
: JSON.stringify(message?.content ?? ''),
|
|
)
|
|
.join('\n');
|
|
} catch {
|
|
/** Estimation falls back to the local approximation. */
|
|
}
|
|
},
|
|
};
|
|
const recordUsage = async (estimate) => {
|
|
const refined =
|
|
typeof estimate === 'function'
|
|
? () => {
|
|
const base = estimate() ?? {};
|
|
return sdkPromptText != null && sdkPromptText.length > 0
|
|
? { ...base, promptText: sdkPromptText }
|
|
: base;
|
|
}
|
|
: estimate;
|
|
await this.recordActivityLabelUsage(
|
|
collectedMetadata,
|
|
clientOptions.model,
|
|
endpointTokenConfig,
|
|
sameEndpoint,
|
|
undefined,
|
|
provider,
|
|
refined,
|
|
);
|
|
};
|
|
/**
|
|
* Accounting is DEFERRED to the hook, which runs it only after the slot
|
|
* commit settles. Awaiting it here (pre-fill) let the settlement window
|
|
* expire during the balance write: the charge landed, then the fill was
|
|
* dropped as out-of-scope — billed, never shown. Registered before the
|
|
* call so a mid-call throw still bills whatever metadata the provider
|
|
* returned, exactly like the old `finally` did.
|
|
*/
|
|
let usageDeferred = false;
|
|
if (typeof deferUsage === 'function') {
|
|
usageDeferred = true;
|
|
deferUsage(recordUsage);
|
|
}
|
|
try {
|
|
const { label } = await this.run.generateActivityLabel({
|
|
provider,
|
|
clientOptions,
|
|
entries: entries.map(({ toolName, toolInput, toolOutput, error, status }) => ({
|
|
toolName,
|
|
toolInput,
|
|
toolOutput,
|
|
error,
|
|
status,
|
|
})),
|
|
thinkingExcerpts: context.thinkingExcerpts,
|
|
lastAssistantText: context.lastAssistantText,
|
|
...(previousLabels != null && { previousLabels }),
|
|
traceSeed,
|
|
charLimit,
|
|
/** Selects the EXECUTING agent's Langfuse metadata and, more
|
|
* importantly, its tool-output redaction policy. Omitting it lets a
|
|
* handoff's activity be traced and redacted under the default
|
|
* agent's configuration, bypassing a stricter per-agent policy. */
|
|
...(executingAgentId != null && { agentId: executingAgentId }),
|
|
/** The wiring always supplies one (the yaml `activityPrompt` when
|
|
* set, else this repo's instruction). Falling through to the SDK's
|
|
* built-in prompt would silently use a different register. */
|
|
...((prompt ?? this.activityLabelPrompt) != null && {
|
|
prompt: prompt ?? this.activityLabelPrompt,
|
|
}),
|
|
chainOptions: {
|
|
signal,
|
|
callbacks: [{ handleLLMEnd, ...capturePrompt }],
|
|
configurable: {
|
|
thread_id: this.conversationId,
|
|
user_id: this.user ?? this.options.req?.user?.id,
|
|
},
|
|
},
|
|
});
|
|
return label ?? null;
|
|
} finally {
|
|
/** Safety net for a caller that did not defer (none in-tree): the old
|
|
* inline accounting, still scope-gated. */
|
|
if (!usageDeferred) {
|
|
await recordUsage();
|
|
}
|
|
}
|
|
}
|
|
|
|
/** Bounded settle for in-flight label fills before finalization. On
|
|
* timeout the label scope is closed and its abort controller fired, so a
|
|
* straggler cannot mutate the saved response or emit into a dead job. */
|
|
async settleActivityLabels(timeoutMs = 3000) {
|
|
/** Detached even when nothing settled: the wiring attaches its abort
|
|
* listener at BUILD time, and a segment can end without a single claim
|
|
* (text-only, or handoff batches, which skip labels) — the early
|
|
* return below would otherwise leave that listener accumulating across
|
|
* HITL approval cycles on the shared job signal. Idempotent. */
|
|
const detachScopeListeners = () => {
|
|
for (const scope of this.activityLabelScopes ?? []) {
|
|
scope.detach?.();
|
|
}
|
|
};
|
|
const pending = this.pendingActivityLabelFills;
|
|
if (!pending || pending.length === 0) {
|
|
detachScopeListeners();
|
|
return;
|
|
}
|
|
this.pendingActivityLabelFills = [];
|
|
await settlePendingLabelFills(pending, timeoutMs, () => {
|
|
/** Close EVERY generation's scope: a pre-pause wiring's straggler must
|
|
* stay closed even though a resume built a newer one. */
|
|
for (const scope of this.activityLabelScopes ?? []) {
|
|
scope.closed = true;
|
|
scope.abort.abort();
|
|
}
|
|
});
|
|
detachScopeListeners();
|
|
}
|
|
|
|
/**
|
|
* Activity-label wiring. At each batch boundary the hook synchronously
|
|
* claims a live content slot (steering's index-offset pattern: push
|
|
* placeholder with deterministic counts, bump the shared offset so
|
|
* subsequent SDK indices land past it) and fills it when the fast-model
|
|
* label resolves. Both states reach the live client via the dedicated
|
|
* `on_activity_label` event; failures leave the counts-only part.
|
|
* @param {string | undefined} streamId
|
|
*/
|
|
buildActivityLabelWiring(streamId, abortSignal) {
|
|
if (!streamId) {
|
|
return undefined;
|
|
}
|
|
/** Per-endpoint opt-in via `activityLabel: true` in librechat.yaml,
|
|
* resolved the same way the title options are (endpoints.all > named
|
|
* endpoint > custom endpoint config). Custom endpoints live in the
|
|
* `endpoints.custom` ARRAY, so their settings are only visible through
|
|
* the matched entry — without it every custom endpoint reads as
|
|
* disabled. */
|
|
const agentEndpoint = this.options.agent?.endpoint ?? '';
|
|
const appConfigForActivity = this.options.req?.config;
|
|
let customEndpointConfig;
|
|
try {
|
|
customEndpointConfig = getCustomEndpointConfig({
|
|
endpoint: agentEndpoint,
|
|
appConfig: appConfigForActivity,
|
|
});
|
|
} catch {
|
|
customEndpointConfig = undefined;
|
|
}
|
|
const activityConfig = resolveActivityConfig(
|
|
appConfigForActivity,
|
|
agentEndpoint,
|
|
customEndpointConfig,
|
|
/** The PUBLIC endpoint (`agents`): `initializeAgent` rewrites
|
|
* `agent.endpoint` to the backing provider, so without this an
|
|
* admin's `endpoints.agents.activityLabel: true` reads the
|
|
* provider's block instead and the feature stays off. */
|
|
this.options.endpoint,
|
|
);
|
|
if (!activityConfig.enabled) {
|
|
return undefined;
|
|
}
|
|
this.activityLabelPrompt = activityConfig.prompt;
|
|
/**
|
|
* Mark the job so a resume can reconcile label gaps without probing
|
|
* content. Retried rather than fire-and-forget: this flag GATES that
|
|
* reconciliation, and it is a separate write from the durable label
|
|
* append — so a single lost write silently drops a label that the label
|
|
* content itself recorded perfectly well. One retry costs nothing at run
|
|
* setup and removes the only realistic way the gate goes stale.
|
|
*/
|
|
/** Retained (not fire-and-forget): the RUN START awaits this persist
|
|
* (chatCompletion/resumeCompletion, before processStream/resume), so
|
|
* the flag is durable before any batch can claim a label — closing the
|
|
* immediate-reconnect race WITHOUT delaying the claim-time reservation
|
|
* emit, whose ordering against shifted SDK indices is load-bearing.
|
|
* The chain settles on failure (warned retry), so a lost write can
|
|
* never wedge run startup. */
|
|
this.activityLabelsMarkedPromise = GenerationJobManager.markActivityLabels(streamId).catch(() =>
|
|
GenerationJobManager.markActivityLabels(streamId).catch(() => {
|
|
logger.warn(
|
|
`[AgentClient] Could not flag activity labels for ${streamId}; a label resolving during a resume gap may not be reconciled.`,
|
|
);
|
|
}),
|
|
);
|
|
/** SDK support probe (steering-style): the Run method and the formatter
|
|
* replay skip ship together, so method presence is the capability. */
|
|
const sdkCapable = typeof Run?.prototype?.generateActivityLabel === 'function';
|
|
/** Label-scoped abort: fired when settle times out so a straggling
|
|
* generation stops burning provider time for a finalized response.
|
|
* Chained to the run signal so a user abort still cancels labels. */
|
|
/** Close state is PER WIRING, not per client: a HITL resume rebuilds the
|
|
* wiring, and resetting a shared instance flag would re-open closures
|
|
* from the pre-pause segment whose provider call ignored the abort.
|
|
* Scopes are retained so settle closes every generation, past included. */
|
|
const labelScope = { closed: false, abort: new AbortController() };
|
|
this.activityLabelScopes = this.activityLabelScopes ?? [];
|
|
this.activityLabelScopes.push(labelScope);
|
|
/** Seed the usage sequence past the labels already on this response.
|
|
* `runId` is the response message id, so a HITL resume — which builds a
|
|
* NEW client for the SAME response — would otherwise restart at -1 and
|
|
* the client's `runId:seq` deduper would discard the post-approval
|
|
* label's usage as already counted. Each label generation is a single
|
|
* non-streaming invoke, so one existing label part == one consumed seq. */
|
|
this.activityLabelUsageSeq =
|
|
this.activityLabelUsageSeq ??
|
|
(this.contentParts ?? []).filter((part) => part?.type === ContentTypes.ACTIVITY_LABEL).length;
|
|
this.activityLabelAbort = labelScope.abort;
|
|
/** An abort CLOSES the scope, not just cancels the call. The rejected
|
|
* generation still runs its catch and calls `fill(null)`; with the scope
|
|
* merely aborted that fill would emit — and by then the next generation
|
|
* may already own the stream, so the event would land an index from the
|
|
* abandoned response onto the new one. */
|
|
const closeOnAbort = () => {
|
|
labelScope.closed = true;
|
|
labelScope.abort.abort();
|
|
};
|
|
if (abortSignal != null) {
|
|
if (abortSignal.aborted) {
|
|
closeOnAbort();
|
|
} else {
|
|
abortSignal.addEventListener('abort', closeOnAbort, { once: true });
|
|
/** Detached once this segment settles: HITL runs rebuild a wiring
|
|
* per approval cycle on the SAME job signal, and `once` only
|
|
* removes the listener if an abort actually fires — long
|
|
* multi-approval runs would otherwise accumulate obsolete
|
|
* closures toward the listener-limit warning. */
|
|
labelScope.detach = () => abortSignal.removeEventListener('abort', closeOnAbort);
|
|
}
|
|
}
|
|
/** Thin wrapper: slot claiming, lane stamping, emit ordering, and settle
|
|
* tracking live in `createActivityLabelWiring` (packages/api, TS). */
|
|
return createActivityLabelWiring({
|
|
maxPerRun: activityConfig.maxPerRun,
|
|
charLimit: activityConfig.charLimit,
|
|
prompt: activityConfig.prompt,
|
|
abortSignal: labelScope.abort.signal,
|
|
isClosed: () => labelScope.closed,
|
|
getContentParts: () => this.contentParts,
|
|
bumpIndexOffset: () => {
|
|
this.steerOffsetState.offset += 1;
|
|
},
|
|
/** Emits IMMEDIATELY — never sequenced behind the flag persist. The
|
|
* claim has already bumped the shared index offset, so delaying the
|
|
* reservation while shifted SDK chunks persist would let a
|
|
* cross-instance reconnect reconstruct a hole, compact it, and have
|
|
* the late label event overwrite the part that moved into its index.
|
|
* Flag ordering is guaranteed upstream instead: run start awaits the
|
|
* persist, so the flag is durable before any batch can claim. */
|
|
emitLabelEvent: (index, part) =>
|
|
GenerationJobManager.emitChunk(
|
|
streamId,
|
|
{
|
|
event: ActivityLabelEvents.ON_ACTIVITY_LABEL,
|
|
data: {
|
|
index,
|
|
part,
|
|
responseMessageId: this.responseMessageId,
|
|
conversationId: this.conversationId,
|
|
},
|
|
},
|
|
/** Label generation is detached and can outlive its generation, so
|
|
* the emit is scoped to the epoch that claimed the index. Without
|
|
* it a straggler from a replaced generation lands its old index on
|
|
* the new response — invisibly, since an empty label renders
|
|
* nothing — overwriting whatever occupies that slot. */
|
|
{ durable: true, expectedCreatedAt: this.jobCreatedAt },
|
|
),
|
|
trackPendingFill: (fillDone) => {
|
|
this.pendingActivityLabelFills = this.pendingActivityLabelFills ?? [];
|
|
this.pendingActivityLabelFills.push(fillDone);
|
|
},
|
|
resolveLLM: () => this.resolveActivityLabelLLM(),
|
|
/** Per-generation usage accounting for the direct fallback path;
|
|
* the SDK bridge records its own via chainOptions callbacks. */
|
|
getInvokeCallbacks: () => {
|
|
const { handleLLMEnd, collected } = createMetadataAggregator();
|
|
return {
|
|
callbacks: [{ handleLLMEnd }],
|
|
collect: async (estimate) => {
|
|
const { provider, clientOptions, endpointTokenConfig, sameEndpoint } =
|
|
await this.resolveActivityLabelLLM();
|
|
await this.recordActivityLabelUsage(
|
|
collected,
|
|
clientOptions.model,
|
|
endpointTokenConfig,
|
|
sameEndpoint,
|
|
/** No scope gate — the hook invokes collect ONLY for a
|
|
* COMMITTED fill (the billing authority), and a scope that
|
|
* closes during the fill's durable emit must not let a
|
|
* visible label escape its charge. Dropped fills never
|
|
* reach this callback. */
|
|
undefined,
|
|
provider,
|
|
estimate,
|
|
);
|
|
},
|
|
};
|
|
},
|
|
...(sdkCapable && {
|
|
generateLabel: (payload) => this.generateActivityLabelViaRun(payload),
|
|
}),
|
|
});
|
|
}
|
|
|
|
/**
|
|
* `AgentClient` is not opinionated about vision requests, so we don't do anything here
|
|
* @param {MongoFile[]} attachments
|
|
*/
|
|
checkVisionRequest() {}
|
|
|
|
getSaveOptions() {
|
|
let runOptions = {};
|
|
try {
|
|
runOptions = payloadParser(this.options) ?? {};
|
|
} catch (error) {
|
|
logger.error(
|
|
'[api/server/controllers/agents/client.js #getSaveOptions] Error parsing options',
|
|
error,
|
|
);
|
|
}
|
|
|
|
return removeNullishValues(
|
|
Object.assign(
|
|
{
|
|
spec: this.options.spec,
|
|
iconURL: this.options.iconURL,
|
|
chatProjectId: this.options.chatProjectId,
|
|
endpoint: this.options.endpoint,
|
|
agent_id: this.options.agent.id,
|
|
modelLabel: this.options.modelLabel,
|
|
resendFiles: this.options.resendFiles,
|
|
imageDetail: this.options.imageDetail,
|
|
maxContextTokens: this.maxContextTokens,
|
|
},
|
|
// TODO: PARSE OPTIONS BY PROVIDER, MAY CONTAIN SENSITIVE DATA
|
|
runOptions,
|
|
),
|
|
);
|
|
}
|
|
|
|
/**
|
|
* Returns build message options. For AgentClient, agent-specific instructions
|
|
* are retrieved directly from agent objects in buildMessages, so this returns empty.
|
|
* @returns {Object} Empty options object
|
|
*/
|
|
getBuildMessagesOptions() {
|
|
return {};
|
|
}
|
|
|
|
/**
|
|
*
|
|
* @param {TMessage} message
|
|
* @param {Array<MongoFile>} attachments
|
|
* @returns {Promise<Array<Partial<MongoFile>>>}
|
|
*/
|
|
async addImageURLs(message, attachments) {
|
|
const { files, image_urls } = await encodeAndFormat(
|
|
this.options.req,
|
|
attachments,
|
|
{
|
|
provider: this.options.agent.provider,
|
|
endpoint: this.options.endpoint,
|
|
},
|
|
VisionModes.agents,
|
|
);
|
|
message.image_urls = image_urls.length ? image_urls : undefined;
|
|
return files;
|
|
}
|
|
|
|
async buildMessages(messages, parentMessageId, _buildOptions, opts) {
|
|
/** Always pass mapMethod; getMessagesForConversation applies it only to messages with addedConvo flag */
|
|
const orderedMessages = this.constructor.getMessagesForConversation({
|
|
messages,
|
|
parentMessageId,
|
|
summary: this.shouldSummarize,
|
|
mapMethod: createMultiAgentMapper(this.options.agent, this.agentConfigs),
|
|
mapCondition: (message) => message.addedConvo === true,
|
|
});
|
|
|
|
let payload;
|
|
/** @type {number | undefined} */
|
|
let promptTokens;
|
|
|
|
/** Normalize instruction fields before applying per-run context. */
|
|
const normalizeInstructions = (agent) => {
|
|
agent.instructions = agent.instructions?.trim() || undefined;
|
|
agent.additional_instructions = agent.additional_instructions?.trim() || undefined;
|
|
return agent;
|
|
};
|
|
|
|
/** Collect all agents for unified processing while preserving stable/dynamic instruction fields. */
|
|
const allAgents = [
|
|
{ agent: normalizeInstructions(this.options.agent), agentId: this.options.agent.id },
|
|
...(this.agentConfigs?.size > 0
|
|
? Array.from(this.agentConfigs.entries()).map(([agentId, agent]) => ({
|
|
agent: normalizeInstructions(agent),
|
|
agentId,
|
|
}))
|
|
: []),
|
|
];
|
|
|
|
/**
|
|
* Memory authorization/loading and MCP config resolution do not depend on
|
|
* attachment hydration or prompt formatting. Start them before that work,
|
|
* but keep the existing context-application barrier below.
|
|
*
|
|
* Attach a rejection observer immediately because these operations may
|
|
* settle while request attachments are still being prepared. Awaiting the
|
|
* original promise later still propagates either error.
|
|
*/
|
|
const earlySharedContextPromise = Promise.all([
|
|
this.useMemory(),
|
|
resolveConfigServers(this.options.req),
|
|
]);
|
|
void earlySharedContextPromise.catch(() => {});
|
|
|
|
const sharedRunAttachmentIds = new Set();
|
|
/** @type {ReturnType<typeof buildAgentScopedContext>} */
|
|
let agentScopedContextPromise;
|
|
const startAgentScopedContext = () => {
|
|
const contextPromise = buildAgentScopedContext({
|
|
agentIds: allAgents.map(({ agentId }) => agentId),
|
|
attachmentsByAgentId: this.options.agentContextAttachmentsByAgentId,
|
|
sharedRunAttachmentIds,
|
|
req: this.options.req,
|
|
tokenCountFn: (text) => countTokens(text),
|
|
});
|
|
void contextPromise.catch(() => {});
|
|
return contextPromise;
|
|
};
|
|
|
|
if (this.options.attachments) {
|
|
const attachments = await this.options.attachments;
|
|
const latestMessage = orderedMessages[orderedMessages.length - 1];
|
|
|
|
for (const fileId of collectFileIds(attachments)) {
|
|
sharedRunAttachmentIds.add(fileId);
|
|
}
|
|
|
|
/** Agent-scoped extraction only depends on the shared attachment IDs. */
|
|
agentScopedContextPromise = startAgentScopedContext();
|
|
|
|
if (this.message_file_map) {
|
|
this.message_file_map[latestMessage.messageId] = attachments;
|
|
} else {
|
|
this.message_file_map = {
|
|
[latestMessage.messageId]: attachments,
|
|
};
|
|
}
|
|
|
|
const [, files] = await Promise.all([
|
|
this.addFileContextToMessage(latestMessage, attachments),
|
|
this.processAttachments(latestMessage, attachments),
|
|
]);
|
|
|
|
this.options.attachments = files;
|
|
} else {
|
|
agentScopedContextPromise = startAgentScopedContext();
|
|
}
|
|
|
|
/** Note: Bedrock uses legacy RAG API handling */
|
|
if (this.message_file_map && !isAgentsEndpoint(this.options.endpoint)) {
|
|
this.contextHandlers = createContextHandlers(
|
|
this.options.req,
|
|
orderedMessages[orderedMessages.length - 1].text,
|
|
);
|
|
}
|
|
|
|
/** @type {Record<number, number>} */
|
|
const indexTokenCountMap = {};
|
|
/** @type {Record<string, number>} */
|
|
const tokenCountMap = {};
|
|
const memoryPayload = [];
|
|
let hasFileContext = false;
|
|
let promptTokenTotal = 0;
|
|
const encoding = this.getEncoding();
|
|
const formattedMessages = orderedMessages.map((message, i) => {
|
|
const formattedMessage = formatMessage({
|
|
message,
|
|
userName: this.options?.name,
|
|
assistantName: this.options?.modelLabel,
|
|
});
|
|
const memoryFormattedMessage = formatMessage({
|
|
message,
|
|
userName: this.options?.name,
|
|
assistantName: this.options?.modelLabel,
|
|
});
|
|
|
|
/**
|
|
* Bind file context to the message it belongs to. Historical attachments
|
|
* are resent inline, so the current turn's text attachment must be inline
|
|
* too instead of living only in the dynamic system tail.
|
|
*/
|
|
if (message.fileContext) {
|
|
hasFileContext = true;
|
|
prependFileContext(formattedMessage, message.fileContext);
|
|
}
|
|
|
|
/**
|
|
* Durably re-merge quoted excerpts into every user turn that carries them
|
|
* (current and historical) so the model receives the referenced context on
|
|
* every prompt and the token count matches what was persisted. Applied to
|
|
* the memory copy too so the canonical per-message count includes them.
|
|
*/
|
|
if (Array.isArray(message.quotes) && message.quotes.length > 0) {
|
|
prependQuotes(formattedMessage, message.quotes);
|
|
prependQuotes(memoryFormattedMessage, message.quotes);
|
|
}
|
|
|
|
memoryPayload.push(memoryFormattedMessage);
|
|
|
|
const dbTokenCount = Number(orderedMessages[i].tokenCount);
|
|
const hasDbTokenCount = Number.isFinite(dbTokenCount) && dbTokenCount > 0;
|
|
/**
|
|
* Force a recount when the message carries quotes: a plain text-only
|
|
* "Save" edit recomputes `tokenCount` from `text` alone while leaving
|
|
* `message.quotes` persisted, so the stored count would undercount the
|
|
* quote block this turn prepends. Recounting from the quote-merged memory
|
|
* copy keeps context accounting accurate (and self-heals stale counts).
|
|
*/
|
|
const needsCanonicalTokenCount =
|
|
!hasDbTokenCount ||
|
|
(this.isVisionModel && (message.image_urls || message.files)) ||
|
|
(Array.isArray(message.quotes) && message.quotes.length > 0);
|
|
|
|
let canonicalTokenCount = hasDbTokenCount ? dbTokenCount : 0;
|
|
if (needsCanonicalTokenCount) {
|
|
canonicalTokenCount = countFormattedMessageTokens(memoryFormattedMessage, encoding);
|
|
}
|
|
|
|
const promptMessageTokenCount = message.fileContext
|
|
? countFormattedMessageTokens(formattedMessage, encoding)
|
|
: canonicalTokenCount;
|
|
|
|
/* If message has files, calculate image token cost */
|
|
if (this.message_file_map && this.message_file_map[message.messageId]) {
|
|
const attachments = this.message_file_map[message.messageId];
|
|
for (const file of attachments) {
|
|
if (file.embedded) {
|
|
this.contextHandlers?.processFile(file);
|
|
continue;
|
|
}
|
|
if (file.metadata?.codeEnvRef) {
|
|
continue;
|
|
}
|
|
}
|
|
}
|
|
|
|
const normalizedCanonicalTokenCount =
|
|
Number.isFinite(canonicalTokenCount) && canonicalTokenCount > 0 ? canonicalTokenCount : 0;
|
|
const normalizedPromptTokenCount =
|
|
Number.isFinite(promptMessageTokenCount) && promptMessageTokenCount > 0
|
|
? promptMessageTokenCount
|
|
: 0;
|
|
|
|
orderedMessages[i].tokenCount = normalizedCanonicalTokenCount;
|
|
indexTokenCountMap[i] = normalizedPromptTokenCount;
|
|
promptTokenTotal += normalizedPromptTokenCount;
|
|
|
|
if (message.messageId) {
|
|
tokenCountMap[message.messageId] = normalizedCanonicalTokenCount;
|
|
}
|
|
|
|
if (isEnabled(process.env.AGENT_DEBUG_LOGGING)) {
|
|
const role = message.isCreatedByUser ? 'user' : 'assistant';
|
|
const hasSummary =
|
|
Array.isArray(message.content) && message.content.some((p) => p && p.type === 'summary');
|
|
const suffix = hasSummary ? '[S]' : '';
|
|
const id = (message.messageId ?? message.id ?? '').slice(-8);
|
|
const recalced = needsCanonicalTokenCount ? normalizedCanonicalTokenCount : null;
|
|
const promptRecalced = message.fileContext ? normalizedPromptTokenCount : null;
|
|
logger.debug(
|
|
`[AgentClient] msg[${i}] ${role}${suffix} id=…${id} db=${dbTokenCount} needsRecount=${needsCanonicalTokenCount} recalced=${recalced} promptRecalced=${promptRecalced} tokens=${normalizedPromptTokenCount}`,
|
|
);
|
|
}
|
|
|
|
return formattedMessage;
|
|
});
|
|
|
|
/**
|
|
* Native YouTube -> video understanding: when Google `url_context` is enabled
|
|
* (resolved to the native `urlContext` provider tool), inject any YouTube URLs
|
|
* from the latest user turn as Gemini `fileData` video parts. The URL Context
|
|
* tool cannot read YouTube, so this routes those links through the video path
|
|
* while other URLs still flow through `urlContext`. Done after token counting
|
|
* (video tokens are reported by the provider) and only on the LLM payload, so
|
|
* the memory copy and persisted message are untouched.
|
|
*/
|
|
const latestOrdered = orderedMessages[orderedMessages.length - 1];
|
|
const provider = this.options.agent?.provider;
|
|
if (
|
|
latestOrdered?.isCreatedByUser === true &&
|
|
(provider === Providers.GOOGLE || provider === Providers.VERTEXAI) &&
|
|
hasUrlContextTool(this.options.agent?.tools)
|
|
) {
|
|
const latestFormatted = formattedMessages[formattedMessages.length - 1];
|
|
/** Use the resolved run model (model_parameters override) rather than the saved base model. */
|
|
const resolvedModel =
|
|
this.options.agent?.model_parameters?.model ?? this.options.agent?.model;
|
|
const { max, mimeType } = resolveYouTubeInjectionConfig({
|
|
provider,
|
|
model: resolvedModel,
|
|
});
|
|
latestFormatted.content = appendYouTubeVideoParts({
|
|
enabled: true,
|
|
text: latestOrdered.text,
|
|
content: latestFormatted.content,
|
|
max,
|
|
mimeType,
|
|
});
|
|
/** Google rejects an unusable video with a generic `INVALID_ARGUMENT` that names no cause,
|
|
* so `#sendCompletion` can only attribute one by knowing this turn carried a video. */
|
|
this.injectedYouTubeVideo = hasYouTubeVideoParts(latestFormatted.content);
|
|
}
|
|
|
|
payload = formattedMessages;
|
|
if (this.options.resendFiles) {
|
|
/** Persisted steer parts of past turns replay with their attachments:
|
|
* one batched owner-scoped fetch, re-encoded per turn and stamped as a
|
|
* transient `media` array (same resend semantics as message files).
|
|
* The stamp lands after the loop above finalized its counts, so the
|
|
* re-encoded media (minus the text part the steer part already counted)
|
|
* is folded into the budget here — large steered attachments must
|
|
* shrink the window like any other resent media. */
|
|
const stamped = await stampSteerPartMedia({
|
|
client: this,
|
|
user: this.options.req?.user,
|
|
payload,
|
|
// addPreviousAttachments already fetched steer-part refs in its single
|
|
// per-turn historical-files query — no second round trip.
|
|
docsById: this.authorizedHistoricalFiles,
|
|
getFiles: db.getFiles,
|
|
});
|
|
for (const { index, media, steerText } of stamped) {
|
|
/** Count the FULL stamped content and subtract only the steer body
|
|
* (already counted inside the assistant message): extracted file
|
|
* context prepended into the text part must hit the budget too, or
|
|
* large steered documents bypass pruning. */
|
|
const fullTokens = countFormattedMessageTokens({ role: 'user', content: media }, encoding);
|
|
const bodyTokens = steerText
|
|
? countFormattedMessageTokens(
|
|
{ role: 'user', content: [{ type: ContentTypes.TEXT, text: steerText }] },
|
|
encoding,
|
|
)
|
|
: 0;
|
|
const mediaTokens = Math.max(0, (fullTokens ?? 0) - (bodyTokens ?? 0));
|
|
if (Number.isFinite(mediaTokens) && mediaTokens > 0) {
|
|
indexTokenCountMap[index] = (indexTokenCountMap[index] ?? 0) + mediaTokens;
|
|
promptTokenTotal += mediaTokens;
|
|
}
|
|
}
|
|
}
|
|
this.memoryPayload = hasFileContext ? memoryPayload : null;
|
|
messages = orderedMessages;
|
|
promptTokens = promptTokenTotal;
|
|
|
|
/**
|
|
* Build shared run context - applies to ALL agents in the run.
|
|
* Request attachment file context is already bound inline to the latest
|
|
* user message above; only side-channel context belongs here.
|
|
* Memory context is handled separately and applied per-agent based on config.
|
|
*/
|
|
const sharedRunContextParts = [];
|
|
const [augmentedPrompt, [memories, configServers], agentScopedContext] = await Promise.all([
|
|
this.contextHandlers?.createContext(),
|
|
earlySharedContextPromise,
|
|
agentScopedContextPromise,
|
|
]);
|
|
|
|
/** Augmented prompt from RAG/context handlers */
|
|
this.augmentedPrompt = augmentedPrompt;
|
|
if (this.augmentedPrompt) {
|
|
sharedRunContextParts.push(this.augmentedPrompt);
|
|
}
|
|
|
|
/** Memory context (user preferences/memories). Keyed context (with memory
|
|
* keys + token metadata) is reserved for agents that can call
|
|
* `delete_memory`; everyone else gets the unkeyed values only. */
|
|
/** Partition the loaded memories belong to (the primary agent's). */
|
|
const loadedMemoryAgentId = getMemoryAgentId(this.options.agent);
|
|
const buildMemoryContext = (text) =>
|
|
text ? `${memoryInstructions}\n\n# Existing memory about the user:\n${text}` : undefined;
|
|
/** Resolves formatted memories for an agent's own partition. A defined
|
|
* `memories` means the run-level gates (permission, opt-out, config)
|
|
* passed; agents on other partitions fetch through the request-scoped
|
|
* cache so repeated partitions share one query. */
|
|
const getAgentPartitionMemories = async (agent) => {
|
|
if (!memories) {
|
|
return undefined;
|
|
}
|
|
const agentPartition = getMemoryAgentId(agent);
|
|
if (agentPartition === loadedMemoryAgentId) {
|
|
return memories;
|
|
}
|
|
try {
|
|
return await getRequestMemories({
|
|
req: this.options.req,
|
|
userId: this.options.req.user.id + '',
|
|
agentId: agentPartition,
|
|
getFormattedMemories: db.getFormattedMemories,
|
|
});
|
|
} catch (error) {
|
|
logger.error('[AgentClient] Error loading partition memories', error);
|
|
return undefined;
|
|
}
|
|
};
|
|
|
|
const sharedRunContext = sharedRunContextParts.join('\n\n');
|
|
const memoryAgentEnabled = isMemoryAgentEnabled(this.options.req.config?.memory);
|
|
|
|
/** Preserve prompt token counts for graph formatting and pruning. */
|
|
this.indexTokenCountMap = indexTokenCountMap;
|
|
|
|
/** Extract contextMeta from the parent response (second-to-last in ordered chain;
|
|
* last is the current user message). Seeds the pruner's calibration EMA for this run. */
|
|
const parentResponse =
|
|
orderedMessages.length >= 2 ? orderedMessages[orderedMessages.length - 2] : undefined;
|
|
if (parentResponse?.contextMeta && !parentResponse.isCreatedByUser) {
|
|
this.contextMeta = parentResponse.contextMeta;
|
|
}
|
|
|
|
const result = {
|
|
prompt: payload,
|
|
tokenCountMap,
|
|
promptTokens,
|
|
messages,
|
|
};
|
|
|
|
if (promptTokens >= 0 && typeof opts?.getReqData === 'function') {
|
|
opts.getReqData({ promptTokens });
|
|
}
|
|
|
|
/**
|
|
* Apply context to all agents.
|
|
* Stable agent/MCP instructions stay on `instructions`; shared runtime context
|
|
* is appended to `additional_instructions` as the dynamic system tail.
|
|
*
|
|
* NOTE: This intentionally mutates agent objects in place. The agentConfigs Map
|
|
* holds references to config objects that will be passed to the graph runtime.
|
|
*/
|
|
const ephemeralAgent = this.options.req.body.ephemeralAgent;
|
|
const mcpManager = getMCPManager();
|
|
|
|
await Promise.all(
|
|
allAgents.map(async ({ agent, agentId }) => {
|
|
const agentRunContextParts = [sharedRunContext];
|
|
const agentHasMemory = agentHasInlineMemoryTools(agent);
|
|
if (agentId === this.options.agent.id || memoryAgentEnabled || agentHasMemory) {
|
|
const partitionMemories = await getAgentPartitionMemories(agent);
|
|
const agentMemoryContext = buildMemoryContext(
|
|
agentHasMemory ? partitionMemories?.withKeys : partitionMemories?.withoutKeys,
|
|
);
|
|
if (agentMemoryContext) {
|
|
agentRunContextParts.push(agentMemoryContext);
|
|
}
|
|
}
|
|
const scopedContext = agentScopedContext.get(agentId);
|
|
if (scopedContext) {
|
|
agentRunContextParts.push(scopedContext);
|
|
}
|
|
|
|
return applyContextToAgent({
|
|
agent,
|
|
agentId,
|
|
logger,
|
|
mcpManager,
|
|
configServers,
|
|
sharedRunContext: agentRunContextParts.filter(Boolean).join('\n\n'),
|
|
ephemeralAgent: agentId === this.options.agent.id ? ephemeralAgent : undefined,
|
|
});
|
|
}),
|
|
);
|
|
|
|
return result;
|
|
}
|
|
|
|
/**
|
|
* Creates a promise that resolves with the memory promise result or undefined after a timeout
|
|
* @param {Promise<(TAttachment | null)[] | undefined>} memoryPromise - The memory promise to await
|
|
* @param {number} timeoutMs - Timeout in milliseconds (default: 3000)
|
|
* @returns {Promise<(TAttachment | null)[] | undefined>}
|
|
*/
|
|
async awaitMemoryWithTimeout(memoryPromise, timeoutMs = 3000) {
|
|
if (!memoryPromise) {
|
|
return;
|
|
}
|
|
|
|
try {
|
|
const timeoutPromise = new Promise((_, reject) =>
|
|
setTimeout(() => reject(new Error('Memory processing timeout')), timeoutMs),
|
|
);
|
|
|
|
const attachments = await Promise.race([memoryPromise, timeoutPromise]);
|
|
return attachments;
|
|
} catch (error) {
|
|
if (error.message === 'Memory processing timeout') {
|
|
logger.warn('[AgentClient] Memory processing timed out after 3 seconds');
|
|
} else {
|
|
logger.error('[AgentClient] Error processing memory:', error);
|
|
}
|
|
return;
|
|
}
|
|
}
|
|
|
|
/**
|
|
* @returns {Promise<{ withKeys?: string; withoutKeys?: string } | undefined>}
|
|
*/
|
|
async useMemory() {
|
|
const user = this.options.req.user;
|
|
if (user.personalization?.memories === false) {
|
|
return;
|
|
}
|
|
const hasAccess = await checkAccess({
|
|
user,
|
|
permissionType: PermissionTypes.MEMORIES,
|
|
permissions: [Permissions.USE],
|
|
getRoleByName: db.getRoleByName,
|
|
});
|
|
|
|
if (!hasAccess) {
|
|
logger.debug(
|
|
`[api/server/controllers/agents/client.js #useMemory] User ${user.id} does not have USE permission for memories`,
|
|
);
|
|
return;
|
|
}
|
|
const appConfig = this.options.req.config;
|
|
const memoryConfig = appConfig.memory;
|
|
if (!memoryConfig || memoryConfig.disabled === true) {
|
|
return;
|
|
}
|
|
|
|
const userId = this.options.req.user.id + '';
|
|
/** Memory partition of the primary agent; undefined = shared personal pool */
|
|
const memoryAgentId = getMemoryAgentId(this.options.agent);
|
|
this.processMemory = undefined;
|
|
|
|
if (!isMemoryAgentEnabled(memoryConfig)) {
|
|
try {
|
|
const { withKeys, withoutKeys } = await getRequestMemories({
|
|
req: this.options.req,
|
|
userId,
|
|
agentId: memoryAgentId,
|
|
getFormattedMemories: db.getFormattedMemories,
|
|
});
|
|
return { withKeys, withoutKeys };
|
|
} catch (error) {
|
|
logger.error(
|
|
'[api/server/controllers/agents/client.js #useMemory] Error loading memories',
|
|
error,
|
|
);
|
|
return;
|
|
}
|
|
}
|
|
|
|
/** @type {Agent} */
|
|
let prelimAgent;
|
|
const allowedProviders = new Set(
|
|
appConfig?.endpoints?.[EModelEndpoint.agents]?.allowedProviders,
|
|
);
|
|
try {
|
|
if (memoryConfig.agent?.id != null && memoryConfig.agent.id !== this.options.agent.id) {
|
|
prelimAgent = await loadAgent({
|
|
req: this.options.req,
|
|
agent_id: memoryConfig.agent.id,
|
|
endpoint: EModelEndpoint.agents,
|
|
});
|
|
} else if (memoryConfig.agent?.id != null) {
|
|
prelimAgent = this.options.agent;
|
|
} else if (
|
|
memoryConfig.agent?.id == null &&
|
|
memoryConfig.agent?.model != null &&
|
|
memoryConfig.agent?.provider != null
|
|
) {
|
|
prelimAgent = { id: Constants.EPHEMERAL_AGENT_ID, ...memoryConfig.agent };
|
|
}
|
|
} catch (error) {
|
|
logger.error(
|
|
'[api/server/controllers/agents/client.js #useMemory] Error loading agent for memory',
|
|
error,
|
|
);
|
|
}
|
|
|
|
if (!prelimAgent) {
|
|
return;
|
|
}
|
|
|
|
/** Forward the same `execute_code` capability gate the chat flow uses —
|
|
* memory agents are unlikely to list `execute_code`, but if one does,
|
|
* Phase 8 relies on this flag to expand the string into
|
|
* `bash_tool` + `read_file` (pre-Phase 8 the legacy `execute_code`
|
|
* tool registered unconditionally; without this passthrough the
|
|
* memory path would silently lose code-execution tooling). */
|
|
const memoryCapabilities = new Set(appConfig?.endpoints?.[EModelEndpoint.agents]?.capabilities);
|
|
const agent = await initializeAgent(
|
|
{
|
|
req: this.options.req,
|
|
res: this.options.res,
|
|
agent: prelimAgent,
|
|
allowedProviders,
|
|
endpointOption: {
|
|
endpoint: !isEphemeralAgentId(prelimAgent.id)
|
|
? EModelEndpoint.agents
|
|
: memoryConfig.agent?.provider,
|
|
},
|
|
codeEnvAvailable: memoryCapabilities.has(AgentCapabilities.execute_code),
|
|
statefulSessionsAvailable: memoryCapabilities.has(AgentCapabilities.stateful_code_sessions),
|
|
},
|
|
{
|
|
getFiles: db.getFiles,
|
|
getUserKey: db.getUserKey,
|
|
getConvoFiles: db.getConvoFiles,
|
|
updateFilesUsage: db.updateFilesUsage,
|
|
getUserKeyValues: db.getUserKeyValues,
|
|
getToolFilesByIds: db.getToolFilesByIds,
|
|
getCodeGeneratedFiles: db.getCodeGeneratedFiles,
|
|
filterFilesByAgentAccess,
|
|
},
|
|
);
|
|
|
|
if (!agent) {
|
|
logger.warn(
|
|
'[api/server/controllers/agents/client.js #useMemory] No agent found for memory',
|
|
memoryConfig,
|
|
);
|
|
return;
|
|
}
|
|
|
|
const llmConfig = Object.assign(
|
|
{
|
|
provider: agent.provider,
|
|
model: agent.model,
|
|
},
|
|
agent.model_parameters,
|
|
);
|
|
|
|
/** @type {import('@librechat/api').MemoryConfig} */
|
|
const config = {
|
|
validKeys: memoryConfig.validKeys,
|
|
instructions: agent.instructions,
|
|
llmConfig,
|
|
tokenLimit: memoryConfig.tokenLimit,
|
|
};
|
|
|
|
const messageId = this.responseMessageId + '';
|
|
const conversationId = this.conversationId + '';
|
|
const streamId = this.options.req?._resumableStreamId || null;
|
|
const [withoutKeys, processMemory] = await createMemoryProcessor({
|
|
userId,
|
|
agentId: memoryAgentId,
|
|
config,
|
|
messageId,
|
|
streamId,
|
|
jobCreatedAt: this.jobCreatedAt,
|
|
conversationId,
|
|
memoryMethods: {
|
|
setMemory: db.setMemory,
|
|
deleteMemory: db.deleteMemory,
|
|
getFormattedMemories: db.getFormattedMemories,
|
|
},
|
|
res: this.options.res,
|
|
user: createSafeUser(this.options.req.user),
|
|
});
|
|
|
|
this.processMemory = processMemory;
|
|
let withKeys = withoutKeys;
|
|
try {
|
|
({ withKeys } = await getRequestMemories({
|
|
req: this.options.req,
|
|
userId,
|
|
agentId: memoryAgentId,
|
|
getFormattedMemories: db.getFormattedMemories,
|
|
}));
|
|
} catch (error) {
|
|
logger.error(
|
|
'[api/server/controllers/agents/client.js #useMemory] Error loading keyed memories',
|
|
error,
|
|
);
|
|
}
|
|
return { withKeys, withoutKeys };
|
|
}
|
|
|
|
/**
|
|
* Filters out image URLs from message content
|
|
* @param {BaseMessage} message - The message to filter
|
|
* @returns {BaseMessage} - A new message with image URLs removed
|
|
*/
|
|
filterImageUrls(message) {
|
|
if (!message.content || typeof message.content === 'string') {
|
|
return message;
|
|
}
|
|
|
|
if (Array.isArray(message.content)) {
|
|
const filteredContent = message.content.filter(
|
|
(part) => part.type !== ContentTypes.IMAGE_URL,
|
|
);
|
|
|
|
if (filteredContent.length === 1 && filteredContent[0].type === ContentTypes.TEXT) {
|
|
const MessageClass = message.constructor;
|
|
return new MessageClass({
|
|
content: filteredContent[0].text,
|
|
additional_kwargs: message.additional_kwargs,
|
|
});
|
|
}
|
|
|
|
const MessageClass = message.constructor;
|
|
return new MessageClass({
|
|
content: filteredContent,
|
|
additional_kwargs: message.additional_kwargs,
|
|
});
|
|
}
|
|
|
|
return message;
|
|
}
|
|
|
|
/**
|
|
* @param {BaseMessage[]} messages
|
|
* @returns {Promise<void | (TAttachment | null)[]>}
|
|
*/
|
|
async runMemory(messages) {
|
|
try {
|
|
if (this.processMemory == null) {
|
|
return;
|
|
}
|
|
const appConfig = this.options.req.config;
|
|
const memoryConfig = appConfig.memory;
|
|
const messageWindowSize = memoryConfig?.messageWindowSize ?? 5;
|
|
|
|
/**
|
|
* Strip skill-primed meta messages before memory extraction. The primes
|
|
* sit next to the latest user message and carry large SKILL.md bodies,
|
|
* so letting them into the window would crowd out real chat turns and
|
|
* pollute extracted memories with synthetic instruction content the
|
|
* user never typed.
|
|
*/
|
|
const chatMessages = messages.filter((m) => !isSkillPrimeMessage(m));
|
|
|
|
let messagesToProcess = [...chatMessages];
|
|
if (chatMessages.length > messageWindowSize) {
|
|
for (let i = chatMessages.length - messageWindowSize; i >= 0; i--) {
|
|
const potentialWindow = chatMessages.slice(i, i + messageWindowSize);
|
|
if (potentialWindow[0]?.role === 'user') {
|
|
messagesToProcess = [...potentialWindow];
|
|
break;
|
|
}
|
|
}
|
|
|
|
if (messagesToProcess.length === chatMessages.length) {
|
|
messagesToProcess = [...chatMessages.slice(-messageWindowSize)];
|
|
}
|
|
}
|
|
|
|
const filteredMessages = messagesToProcess.map((msg) => this.filterImageUrls(msg));
|
|
const bufferString = getBufferString(filteredMessages);
|
|
const configuredMaxInputTokens = Number.isFinite(memoryConfig?.maxInputTokens)
|
|
? Math.floor(memoryConfig.maxInputTokens)
|
|
: undefined;
|
|
const maxInputTokens =
|
|
configuredMaxInputTokens != null && configuredMaxInputTokens > 0
|
|
? configuredMaxInputTokens
|
|
: DEFAULT_MEMORY_MAX_INPUT_TOKENS;
|
|
const maxInputChars = maxInputTokens * MEMORY_INPUT_CHARS_PER_TOKEN;
|
|
const isCharTruncated = bufferString.length > maxInputChars;
|
|
const memoryInput = `# Current Chat:\n\n${
|
|
isCharTruncated
|
|
? `[Earlier chat content omitted due to memory input limit]\n\n${bufferString.slice(
|
|
-maxInputChars,
|
|
)}`
|
|
: bufferString
|
|
}`;
|
|
const {
|
|
text: limitedMemoryInput,
|
|
tokenCount,
|
|
wasTruncated,
|
|
} = await processTextWithTokenLimit({
|
|
text: memoryInput,
|
|
tokenLimit: maxInputTokens,
|
|
tokenCountFn: (text) => countTokens(text),
|
|
preserve: 'end',
|
|
});
|
|
if (isCharTruncated || wasTruncated) {
|
|
logger.warn('[MemoryAgent] Memory input truncated before processing', {
|
|
tokenCount,
|
|
messageId: this.responseMessageId,
|
|
conversationId: this.conversationId,
|
|
maxInputTokens,
|
|
wasTruncated,
|
|
maxInputChars,
|
|
originalLength: bufferString.length,
|
|
});
|
|
}
|
|
const bufferMessage = new HumanMessage(limitedMemoryInput);
|
|
return await this.processMemory([bufferMessage]);
|
|
} catch (error) {
|
|
logger.error('Memory Agent failed to process memory', error);
|
|
}
|
|
}
|
|
|
|
/** @type {sendCompletion} */
|
|
async sendCompletion(payload, opts = {}) {
|
|
await this.chatCompletion({
|
|
payload,
|
|
onProgress: opts.onProgress,
|
|
userMCPAuthMap: opts.userMCPAuthMap,
|
|
abortController: opts.abortController,
|
|
});
|
|
|
|
const completion = filterMalformedContentParts(this.contentParts);
|
|
const metadata = this.buildResponseMetadata();
|
|
return metadata ? { completion, metadata } : { completion };
|
|
}
|
|
|
|
/**
|
|
* Assembles the response message `metadata`: Vertex thought signatures plus
|
|
* the persisted context breakdown (Part A) and the usage/cost rollup (Part B),
|
|
* which rebuild the gauge breakdown and branch/total cost across reloads.
|
|
* Returns undefined when nothing was captured.
|
|
* @returns {{
|
|
* thoughtSignatures?: Record<string, string>,
|
|
* contextUsage?: import('librechat-data-provider').TContextUsageEvent,
|
|
* usage?: import('librechat-data-provider').TResponseUsage,
|
|
* } | undefined}
|
|
*/
|
|
buildResponseMetadata() {
|
|
/** @type {{
|
|
* thoughtSignatures?: Record<string, string>,
|
|
* contextUsage?: import('librechat-data-provider').TContextUsageEvent,
|
|
* usage?: import('librechat-data-provider').TResponseUsage,
|
|
* }} */
|
|
const metadata = {};
|
|
const signatures = this.collectedThoughtSignatures;
|
|
if (signatures && Object.keys(signatures).length > 0) {
|
|
metadata.thoughtSignatures = signatures;
|
|
}
|
|
const usageEvents = this.usageEmitSink ?? [];
|
|
/** Persist the breakdown only when the latest snapshot's OWN run completed —
|
|
* i.e. a PRIMARY usage event (usage_type == null) from that run's id arrived
|
|
* AFTER the snapshot. Matching by run id keeps `completedOutputTokens` a real
|
|
* post-snapshot delta even when parallel/direct runs interleave (A snapshot →
|
|
* B snapshot → A usage must NOT persist B's snapshot with A's output); an
|
|
* interrupted final call that emits no usage falls back to the per-message
|
|
* estimate. It still keeps the post-summary snapshot: the summarization detour
|
|
* emits an extra snapshot whose following primary usage shares that run's id,
|
|
* which the old snapshot-count guard miscounted and wrongly dropped. Events
|
|
* without a run id (older lib / resume) match any snapshot for back-compat. */
|
|
const latestSnapshot = this.contextUsageSink?.latest;
|
|
const latestSnapshotUsageIndex = this.contextUsageSink?.latestUsageIndex ?? 0;
|
|
const latestSnapshotRunId = latestSnapshot?.runId;
|
|
const hasPrimaryAfterSnapshot = usageEvents
|
|
.slice(latestSnapshotUsageIndex)
|
|
.some(
|
|
(event) =>
|
|
event.usage_type == null &&
|
|
(latestSnapshotRunId == null ||
|
|
event.runId == null ||
|
|
event.runId === latestSnapshotRunId),
|
|
);
|
|
if (latestSnapshot && hasPrimaryAfterSnapshot) {
|
|
metadata.contextUsage = buildPersistedContextUsage(latestSnapshot, usageEvents);
|
|
}
|
|
/** Lightweight summarization marker — persisted whenever this turn compacted
|
|
* the context, INDEPENDENT of the snapshot guard above. When the client has
|
|
* no usable snapshot on the branch and falls back to the per-message
|
|
* estimate, it caps the discarded pre-summary history at this baseline
|
|
* instead of re-summing it (the gauge otherwise reads 100% forever). Shared
|
|
* with the abort save path via `computeSummaryUsedTokens`. Subtract the
|
|
* response's earlier tool-loop outputs (the primaries that preceded the
|
|
* latest snapshot, same run): those tokens are inside the snapshot baseline
|
|
* AND in the response `tokenCount` the client estimate adds on top, so
|
|
* leaving them in the marker double-counts them on a multi-call turn. */
|
|
const priorOutputTokens = priorRunOutputTokens(
|
|
usageEvents,
|
|
latestSnapshotUsageIndex,
|
|
latestSnapshotRunId,
|
|
);
|
|
const summaryUsedTokens = computeSummaryUsedTokens(latestSnapshot, priorOutputTokens);
|
|
if (summaryUsedTokens != null) {
|
|
metadata.summaryUsedTokens = summaryUsedTokens;
|
|
}
|
|
const usage = aggregateEmittedUsage(usageEvents);
|
|
if (usage) {
|
|
metadata.usage = usage;
|
|
}
|
|
return Object.keys(metadata).length > 0 ? metadata : undefined;
|
|
}
|
|
|
|
/**
|
|
* Resolves the endpoint token config for a usage item by its producing agent
|
|
* (multi-endpoint graphs: connected agents + subagents). A known agent's
|
|
* config is authoritative — including `undefined`, which prices with built-in
|
|
* rates (e.g. a non-custom agent in a custom-primary graph). Only an
|
|
* untagged/unknown agent falls back to the primary config, so single-endpoint
|
|
* graphs are unchanged.
|
|
* @param {UsageMetadata} usage
|
|
* @returns {import('@librechat/api').EndpointTokenConfig | undefined}
|
|
*/
|
|
resolveAgentEndpointTokenConfig(usage) {
|
|
return resolveAgentTokenConfig({
|
|
agentId: usage?.agentId,
|
|
byAgentId: this.options.endpointTokenConfigByAgentId,
|
|
fallback: this.options.endpointTokenConfig,
|
|
});
|
|
}
|
|
|
|
/**
|
|
* @param {Object} params
|
|
* @param {string} [params.model]
|
|
* @param {string} [params.context='message']
|
|
* @param {AppConfig['balance']} [params.balance]
|
|
* @param {AppConfig['transactions']} [params.transactions]
|
|
* @param {UsageMetadata[]} [params.collectedUsage=this.collectedUsage]
|
|
*/
|
|
async recordCollectedUsage({
|
|
model,
|
|
balance,
|
|
transactions,
|
|
context = 'message',
|
|
collectedUsage = this.collectedUsage,
|
|
/**
|
|
* Rates for usage that did NOT run on the agent's endpoint — currently
|
|
* activity labels pointed at a different `activityEndpoint`. Without it
|
|
* the caller's config was dropped here and the balance transaction was
|
|
* written at the primary agent's rates while the UI cost was computed at
|
|
* the label's, so the two disagreed. `undefined` keeps the agent default.
|
|
*/
|
|
endpointTokenConfig,
|
|
/**
|
|
* True when this usage ran on a DIFFERENT endpoint than the agent, making
|
|
* `endpointTokenConfig` authoritative even when it is `undefined` (a
|
|
* built-in endpoint prices from the shared table). Presence of the value
|
|
* cannot express that, which is why the caller states it outright.
|
|
*/
|
|
crossEndpoint = false,
|
|
/**
|
|
* Whether this recording owns `getStreamUsage()`. Only the PRIMARY
|
|
* generation does. Secondary usage (activity labels) must still be
|
|
* billed, but writing it here would hand `BaseClient` the label's token
|
|
* counts as the assistant response's authoritative total — and because
|
|
* the primary call returns early when it collected nothing, the wrong
|
|
* value would never be replaced, suppressing the text-based token
|
|
* fallback and leaving the real generation unbilled.
|
|
*/
|
|
updateStreamUsage = true,
|
|
}) {
|
|
/** Per-agent resolution keys off the AGENT's config map, which cannot
|
|
* describe a label running on a different endpoint — so an explicit
|
|
* config wins outright rather than being second-guessed per usage row.
|
|
*
|
|
* Keyed on the caller's discriminator, NOT on `endpointTokenConfig !==
|
|
* undefined`: a built-in label endpoint prices from the shared table, so
|
|
* `undefined` is its meaningful value. Reading that as "no override" is
|
|
* what silently restored the primary's custom rates. */
|
|
const overrideTokenConfig = crossEndpoint === true;
|
|
const result = await recordCollectedUsage(
|
|
{
|
|
spendTokens: db.spendTokens,
|
|
spendStructuredTokens: db.spendStructuredTokens,
|
|
pricing: { getMultiplier: db.getMultiplier, getCacheMultiplier: db.getCacheMultiplier },
|
|
bulkWriteOps: { insertMany: db.bulkInsertTransactions, updateBalance: db.updateBalance },
|
|
},
|
|
{
|
|
user: this.user ?? this.options.req.user?.id,
|
|
conversationId: this.conversationId,
|
|
collectedUsage,
|
|
model: model ?? this.model ?? this.options.agent.model_parameters.model,
|
|
context,
|
|
messageId: this.responseMessageId,
|
|
balance,
|
|
transactions,
|
|
endpointTokenConfig: overrideTokenConfig
|
|
? endpointTokenConfig
|
|
: this.options.endpointTokenConfig,
|
|
...(overrideTokenConfig
|
|
? {}
|
|
: { resolveEndpointTokenConfig: (usage) => this.resolveAgentEndpointTokenConfig(usage) }),
|
|
},
|
|
);
|
|
|
|
if (result && updateStreamUsage) {
|
|
this.usage = result;
|
|
}
|
|
}
|
|
|
|
/**
|
|
* Get stream usage as returned by this client's API response.
|
|
* @returns {UsageMetadata} The stream usage object.
|
|
*/
|
|
getStreamUsage() {
|
|
return this.usage;
|
|
}
|
|
|
|
/**
|
|
* Builds the subagent usage emitter for {@link createSubagentUsageSink}.
|
|
* Streams each billed child-run usage to the client as an `on_token_usage`
|
|
* event tagged `subagent` (folds into session cost/totals, not the live
|
|
* gauge), with the authoritative cost when `interface.contextCost` is on.
|
|
* Returns undefined when there's no stream to write to.
|
|
* @param {AppConfig} [appConfig]
|
|
* @returns {((usage: UsageMetadata) => void) | undefined}
|
|
*/
|
|
buildSubagentUsageEmitter(appConfig) {
|
|
const res = this.options.res;
|
|
const streamId = this.options.req?._resumableStreamId || null;
|
|
if (!res && !streamId) {
|
|
return undefined;
|
|
}
|
|
const includeCost = appConfig?.interfaceConfig?.contextCost === true;
|
|
return (usage) => {
|
|
const data = {
|
|
input_tokens: usage.input_tokens,
|
|
output_tokens: usage.output_tokens,
|
|
total_tokens: usage.total_tokens,
|
|
input_token_details: this.subagentCacheDetails(usage),
|
|
model: usage.model,
|
|
provider: usage.provider,
|
|
usage_type: 'subagent',
|
|
runId: this.responseMessageId,
|
|
/** Unique per collected entry (post-push length) for resume dedupe */
|
|
seq: this.collectedUsage.length,
|
|
/** Price with the SUBAGENT's own endpoint token config (its endpoint may
|
|
* differ from the parent's); `usage.agentId` is tagged by the sink. */
|
|
cost: includeCost
|
|
? computeUsageCostUSD(
|
|
usage,
|
|
{ getMultiplier: db.getMultiplier, getCacheMultiplier: db.getCacheMultiplier },
|
|
this.resolveAgentEndpointTokenConfig(usage),
|
|
)
|
|
: undefined,
|
|
};
|
|
/** Fold into the response's usage rollup (synchronously, regardless of
|
|
* emit success) so the persisted total matches the live session, which
|
|
* also folds subagent usage into its cost/totals. */
|
|
if (this.usageEmitSink) {
|
|
this.usageEmitSink.push(data);
|
|
}
|
|
/** The sink fires this without awaiting, so retain the promise and flush
|
|
* it in chatCompletion's finally — emitChunk persists (HSET) before
|
|
* publishing, and job cleanup must not race that persist or resumed
|
|
* clients miss billed subagent usage. */
|
|
const emit = (async () => {
|
|
try {
|
|
if (streamId) {
|
|
await GenerationJobManager.emitChunk(
|
|
streamId,
|
|
{
|
|
event: UsageEvents.ON_TOKEN_USAGE,
|
|
data,
|
|
},
|
|
{ expectedCreatedAt: this.jobCreatedAt },
|
|
);
|
|
} else {
|
|
sendEvent(res, { event: UsageEvents.ON_TOKEN_USAGE, data });
|
|
}
|
|
} catch (err) {
|
|
logger.warn('[AgentClient] Failed to emit subagent usage', err);
|
|
}
|
|
})();
|
|
this.pendingSubagentEmits.push(emit);
|
|
return emit;
|
|
};
|
|
}
|
|
|
|
/** Normalizes a subagent usage event's cache token details for emission. */
|
|
subagentCacheDetails(usage) {
|
|
const cache_creation =
|
|
usage.input_token_details?.cache_creation ?? usage.cache_creation_input_tokens;
|
|
const cache_read = usage.input_token_details?.cache_read ?? usage.cache_read_input_tokens;
|
|
if (cache_creation == null && cache_read == null) {
|
|
return undefined;
|
|
}
|
|
return { cache_creation, cache_read };
|
|
}
|
|
|
|
/**
|
|
* @param {TMessage} responseMessage
|
|
* @returns {number}
|
|
*/
|
|
getTokenCountForResponse({ content }) {
|
|
return countFormattedMessageTokens({ role: 'assistant', content }, this.getEncoding());
|
|
}
|
|
|
|
/**
|
|
* @param {object} params
|
|
* @param {string | ChatCompletionMessageParam[]} params.payload
|
|
* @param {Record<string, Record<string, string>>} [params.userMCPAuthMap]
|
|
* @param {AbortController} [params.abortController]
|
|
*/
|
|
/**
|
|
* @deprecated Agent Chain — strip hidden intermediate sequential-agent content
|
|
* before persistence, keeping only the last part + tool_call parts. Mirrors the
|
|
* chat path so a HITL resume doesn't persist/emit intermediate outputs the
|
|
* agent's `hide_sequential_outputs` setting is meant to hide.
|
|
*/
|
|
applyHideSequentialOutputsFilter() {
|
|
if (!this.options.agent?.hide_sequential_outputs || !Array.isArray(this.contentParts)) {
|
|
return;
|
|
}
|
|
this.contentParts = this.contentParts.filter(
|
|
(part, index) =>
|
|
index >= this.contentParts.length - 1 ||
|
|
part.type === ContentTypes.TOOL_CALL ||
|
|
// Steer parts are user speech, not intermediate agent output — dropping
|
|
// one would erase the user's words from the persisted turn.
|
|
part.type === ContentTypes.STEER ||
|
|
// Activity labels summarize the hidden intermediate outputs — exactly
|
|
// the affordance hide_sequential_outputs wants to keep visible.
|
|
part.type === ContentTypes.ACTIVITY_LABEL ||
|
|
part.tool_call_ids,
|
|
);
|
|
}
|
|
|
|
/**
|
|
* Surface any human-in-the-loop interrupt the SDK captured during the most
|
|
* recent `processStream` / `resume`. When the run paused for tool approval (or
|
|
* an ask-user question), mark the job `requires_action`, persist the pending
|
|
* review record, and emit it to live clients — then set `this.pendingApproval`
|
|
* so the controller leaves the turn unfinalized for the resume route to continue.
|
|
*
|
|
* No-op when the run completed without an interrupt, or when the job was aborted
|
|
* between the interrupt firing and this mark (a late interrupt must not pause a
|
|
* dead job — the atomic `pause` transition returns false and we drop it).
|
|
*
|
|
* @param {AgentRun} run
|
|
* @param {string} [streamId]
|
|
*/
|
|
async handleRunInterrupt(run, streamId) {
|
|
if (!streamId || typeof run?.getInterrupt !== 'function') {
|
|
return;
|
|
}
|
|
const interrupt = run.getInterrupt();
|
|
if (!interrupt?.payload) {
|
|
return;
|
|
}
|
|
|
|
const appConfig = this.options.req?.config;
|
|
const checkpointerCfg = appConfig?.endpoints?.[EModelEndpoint.agents]?.checkpointer;
|
|
// Persist the generation params (temperature, max tokens, custom endpoint params, …)
|
|
// so an ephemeral-agent resume continues with the SAME settings the run paused on.
|
|
// The resume payload omits them and they aren't part of the fingerprint, so without
|
|
// this the rebuilt ephemeral run falls back to defaults. The paused request body is
|
|
// the primary source (UI-form, round-trips the compact-convo schema by construction);
|
|
// the resolved llmConfig fills gaps and is sanitized — it carries provider secrets
|
|
// (apiKey, credentials) and gateway config — resume re-resolves those server-side.
|
|
// (Saved agents source params from the DB record, so this is belt-and-suspenders.)
|
|
const resumeContext = pickResumeContext(this.options.req?.body);
|
|
const resumeModelParameters = captureResumeModelParameters(
|
|
this.options.req?.body,
|
|
this.options.agent?.model_parameters,
|
|
);
|
|
if (resumeModelParameters) {
|
|
resumeContext.model_parameters = resumeModelParameters;
|
|
}
|
|
// Persist the question onto the paused ask tool_call's args NOW: an
|
|
// abandoned/expired/stopped pause never reaches the answer-resume stamp,
|
|
// and the streamed args were dropped (name-less chunks) — without this the
|
|
// unfinished turn saves an empty ask part the record card can't render.
|
|
if (interrupt.payload?.type === 'ask_user_question' && Array.isArray(this.contentParts)) {
|
|
const stamped = attachAskUserQuestionArgs(this.contentParts, interrupt.payload.question);
|
|
if (stamped !== this.contentParts) {
|
|
this.contentParts.length = 0;
|
|
this.contentParts.push(...stamped);
|
|
}
|
|
}
|
|
const pendingAction = buildPendingAction(interrupt.payload, {
|
|
streamId,
|
|
conversationId: this.conversationId,
|
|
// runId mirrors the LangGraph checkpoint namespace when the SDK provides it
|
|
// (its documented meaning), falling back to the response message id.
|
|
runId: interrupt.checkpointNs ?? this.responseMessageId,
|
|
responseMessageId: this.responseMessageId,
|
|
interruptId: interrupt.interruptId,
|
|
// thread_id was bound to conversationId at run config (config.configurable);
|
|
// fall back to it when the SDK doesn't echo threadId on the interrupt.
|
|
threadId: interrupt.threadId ?? this.conversationId,
|
|
ttlMs: getApprovalTtlMs(checkpointerCfg),
|
|
// Pin the graph-determining request fields so resume can't rebuild this paused
|
|
// run on a different agent/tool set (esp. ephemeral agents, whose agent_id is
|
|
// undefined so the id guard can't tell two configs apart).
|
|
requestFingerprint: computeAgentRequestFingerprint(this.options.req?.body ?? {}),
|
|
// Persist those same fields verbatim so the resume route can REPLAY them — a
|
|
// reload/cross-replica resume can't reconstruct the ephemeral config client-side,
|
|
// so the server restores it and rebuilds the same graph (and the fingerprint matches).
|
|
resumeContext,
|
|
});
|
|
|
|
// Job-replacement guard: streamId == conversationId is reused per conversation, so a
|
|
// newer request can replace this run's job. If this (older) run hits an interrupt
|
|
// after a replacement, pausing would flip the NEWER job to requires_action with this
|
|
// stale run's pending action, blocking fresh work behind the wrong approval. Only
|
|
// pause when the live job is still the one THIS run created (mirrors request.js).
|
|
if (this.jobCreatedAt != null) {
|
|
const liveJob = await GenerationJobManager.getJobStore().getJob(streamId);
|
|
if (!liveJob || liveJob.createdAt !== this.jobCreatedAt) {
|
|
logger.debug(`[AgentClient] Interrupt fired but job ${streamId} was replaced; not pausing`);
|
|
return;
|
|
}
|
|
}
|
|
|
|
const paused = await GenerationJobManager.approvals.pause(streamId, pendingAction);
|
|
if (!paused) {
|
|
logger.debug(
|
|
`[AgentClient] Interrupt fired but job ${streamId} was not running; not pausing`,
|
|
);
|
|
return;
|
|
}
|
|
|
|
// Capture deferred tools discovered (via tool_search) earlier in THIS turn so resume
|
|
// can replay them into createRun. The resumed graph is rebuilt with `messages: []`
|
|
// (state comes from the checkpoint), so the in-turn tool_search results that mark a
|
|
// deferred tool discovered aren't present there — without this the paused deferred
|
|
// tool would be missing from the rebuilt schema-only toolMap and resume would fail
|
|
// with "unknown tool". Inert for non-deferred turns (the set comes back empty).
|
|
try {
|
|
const runMessages =
|
|
typeof run.getRunMessages === 'function' ? run.getRunMessages() : undefined;
|
|
if (Array.isArray(runMessages) && runMessages.length > 0) {
|
|
const discovered = extractDiscoveredToolsFromHistory(runMessages);
|
|
if (discovered.size > 0) {
|
|
await GenerationJobManager.updateMetadata(
|
|
streamId,
|
|
{
|
|
discoveredTools: Array.from(discovered),
|
|
},
|
|
this.jobCreatedAt,
|
|
);
|
|
}
|
|
}
|
|
} catch (err) {
|
|
logger.warn(
|
|
`[AgentClient] Failed to capture discovered tools for resume on ${streamId}`,
|
|
err?.message ?? err,
|
|
);
|
|
}
|
|
|
|
this.pendingApproval = pendingAction;
|
|
// Release the concurrency slot this request held the MOMENT the turn is durably
|
|
// paused — before the approval card is emitted — so the user's `/resume` can
|
|
// re-acquire one immediately. Otherwise a fast Approve races the HTTP-driver
|
|
// teardown (request.js pause branch / resume.js finally) that would otherwise
|
|
// release it, and `/resume` 429s under LIMIT_CONCURRENT_MESSAGES. Idempotent via
|
|
// the flag; if it fails here, the teardown still releases (it checks the flag).
|
|
if (!this.pendingRequestReleased) {
|
|
try {
|
|
await decrementPendingRequest(this.options.req?.user?.id);
|
|
this.pendingRequestReleased = true;
|
|
} catch (err) {
|
|
logger.error(`[AgentClient] Failed to release request slot on pause ${streamId}`, err);
|
|
}
|
|
}
|
|
await GenerationJobManager.emitChunk(
|
|
streamId,
|
|
{
|
|
event: ApprovalEvents.ON_PENDING_ACTION,
|
|
data: toClientPendingAction(pendingAction),
|
|
},
|
|
{ expectedCreatedAt: this.jobCreatedAt },
|
|
);
|
|
// Steers queued before this pause stay IN the store for the whole approval
|
|
// window: `resumeState.pendingSteers` re-seeds the client's chips on
|
|
// reload, and the resumed run drains them at its first tool boundary.
|
|
// Draining here would leave the only copy in ephemeral client state — a
|
|
// reload during the pause would silently lose the user's message. New
|
|
// steers are rejected while paused (enqueue is status-guarded), and the
|
|
// requires_action TTL extension keeps the queue key alive.
|
|
logger.debug(
|
|
`[AgentClient] Paused ${streamId} for ${interrupt.payload.type} (action ${pendingAction.actionId})`,
|
|
);
|
|
}
|
|
|
|
async chatCompletion({ payload, userMCPAuthMap, abortController = null }) {
|
|
/** @type {Partial<GraphRunnableConfig>} */
|
|
let config;
|
|
/** @type {ReturnType<createRun>} */
|
|
let run;
|
|
/** @type {Promise<(TAttachment | null)[] | undefined>} */
|
|
let memoryPromise;
|
|
const appConfig = this.options.req.config;
|
|
const balanceConfig = getBalanceConfig(appConfig);
|
|
const transactionsConfig = getTransactionsConfig(appConfig);
|
|
try {
|
|
if (!abortController) {
|
|
abortController = new AbortController();
|
|
}
|
|
|
|
/** Fire-and-forget: boot the per-conversation stateful sandbox in
|
|
* parallel with generation so the first execute_code/bash call lands
|
|
* on a warm VM. No-op unless a reachable agent resolved
|
|
* `statefulCodeSessions`. */
|
|
maybePrewarmCodeSandbox({
|
|
req: this.options.req,
|
|
conversationId: this.conversationId,
|
|
agents: [this.options.agent, ...(this.agentConfigs?.values() ?? [])],
|
|
});
|
|
|
|
/** @type {AppConfig['endpoints']['agents']} */
|
|
const agentsEConfig = appConfig.endpoints?.[EModelEndpoint.agents];
|
|
|
|
config = {
|
|
runName: 'AgentRun',
|
|
configurable: {
|
|
thread_id: this.conversationId,
|
|
last_agent_index: this.agentConfigs?.size ?? 0,
|
|
user_id: this.user ?? this.options.req.user?.id,
|
|
hide_sequential_outputs: this.options.agent.hide_sequential_outputs,
|
|
requestBody: {
|
|
messageId: this.responseMessageId,
|
|
conversationId: this.conversationId,
|
|
parentMessageId: this.parentMessageId,
|
|
},
|
|
user: createSafeUser(this.options.req.user),
|
|
},
|
|
recursionLimit: resolveRecursionLimit(agentsEConfig, this.options.agent),
|
|
signal: abortController.signal,
|
|
streamMode: 'values',
|
|
version: 'v2',
|
|
};
|
|
|
|
const toolSet = buildToolSet(this.options.agent);
|
|
const tokenCounter = createTokenCounter(this.getEncoding());
|
|
|
|
/** Pre-resolve invoked skill bodies + re-prime files before formatting messages */
|
|
const skillPrimeResult = this.options.primeInvokedSkills
|
|
? await this.options.primeInvokedSkills(payload)
|
|
: undefined;
|
|
|
|
/**
|
|
* Seed `Graph.sessions` with code-env files primed across every
|
|
* reachable agent (primary, handoff/addedConvo, and nested
|
|
* subagents) plus skill-priming output. The merge logic and its
|
|
* run-wide semantics live in `buildInitialToolSessions`; see that
|
|
* helper's doc for why this is intentionally NOT per-agent.
|
|
*/
|
|
const initialSessions = buildInitialToolSessions({
|
|
skillSessions: skillPrimeResult?.initialSessions,
|
|
agents: [this.options.agent, ...(this.agentConfigs ? this.agentConfigs.values() : [])],
|
|
});
|
|
|
|
/**
|
|
* Reconstruct `reasoning_content` on prior tool-call turns: DeepSeek
|
|
* thinking-mode (#13366) or custom endpoints opting in via
|
|
* `customParams.includeReasoningHistory` (e.g. Xiaomi MiMo, Kimi).
|
|
* Walks subagents too — the opted-in endpoint may appear only as a
|
|
* nested subagent, not the primary or a top-level handoff agent.
|
|
*/
|
|
const needsReasoningContentFormat = anyAgentReplaysReasoningContent([
|
|
this.options.agent,
|
|
...(this.agentConfigs ? Array.from(this.agentConfigs.values()) : []),
|
|
]);
|
|
/**
|
|
* Skills primed fresh this turn — manual ($ popover) and always-apply
|
|
* (frontmatter). `injectSkillPrimes` (below) splices their SKILL.md
|
|
* bodies in, so `formatAgentMessages` must NOT also reconstruct the
|
|
* same names from a historical `skill` tool_call — otherwise the body
|
|
* lands twice and a prompt-cache marker can pin to the duplicated
|
|
* synthetic prefix. Names NOT primed this turn still reconstruct from
|
|
* history, preserving sticky manual re-priming across turns.
|
|
*/
|
|
const manualSkillPrimes = this.options.agent?.manualSkillPrimes;
|
|
const alwaysApplySkillPrimes = this.options.agent?.alwaysApplySkillPrimes;
|
|
const freshSkillPrimeNames = collectFreshSkillPrimeNames({
|
|
manualSkillPrimes,
|
|
alwaysApplySkillPrimes,
|
|
});
|
|
const formatOptions =
|
|
needsReasoningContentFormat || freshSkillPrimeNames.size > 0
|
|
? {
|
|
...(needsReasoningContentFormat ? { preserveReasoningContent: true } : {}),
|
|
...(freshSkillPrimeNames.size > 0
|
|
? { skipSkillBodyNames: freshSkillPrimeNames }
|
|
: {}),
|
|
}
|
|
: undefined;
|
|
let {
|
|
messages: initialMessages,
|
|
indexTokenCountMap,
|
|
summary: initialSummary,
|
|
boundaryTokenAdjustment,
|
|
} = formatAgentMessages(
|
|
stripActivityLabelParts(payload),
|
|
this.indexTokenCountMap,
|
|
toolSet,
|
|
skillPrimeResult?.skills,
|
|
formatOptions,
|
|
);
|
|
if (boundaryTokenAdjustment) {
|
|
logger.debug(
|
|
`[AgentClient] Boundary token adjustment: ${boundaryTokenAdjustment.original} → ${boundaryTokenAdjustment.adjusted} (${boundaryTokenAdjustment.remainingChars}/${boundaryTokenAdjustment.totalChars} chars)`,
|
|
);
|
|
}
|
|
|
|
/**
|
|
* Skill priming — both manual ($ popover) and always-apply (frontmatter).
|
|
*
|
|
* Splice + index-shift logic lives in `injectSkillPrimes`
|
|
* (packages/api/src/agents/skills.ts) so the delicate position math
|
|
* can be unit-tested in TS without standing up AgentClient. The
|
|
* resolver enforces a combined ceiling (manual-first, always-apply
|
|
* truncated first when over cap) before reaching here; the splice
|
|
* re-applies the cap as defense-in-depth. Runs for both single-
|
|
* agent and multi-agent runs; how primes interact with handoff /
|
|
* added-convo agents' per-agent state is an agents-SDK concern,
|
|
* not this layer's to gate.
|
|
*
|
|
* `manualSkillPrimes` / `alwaysApplySkillPrimes` are resolved above
|
|
* (used to build `freshSkillPrimeNames` for dedupe against historical
|
|
* skill reconstruction).
|
|
*/
|
|
if (
|
|
(manualSkillPrimes && manualSkillPrimes.length > 0) ||
|
|
(alwaysApplySkillPrimes && alwaysApplySkillPrimes.length > 0)
|
|
) {
|
|
const primeResult = injectSkillPrimes({
|
|
initialMessages,
|
|
indexTokenCountMap,
|
|
manualSkillPrimes,
|
|
alwaysApplySkillPrimes,
|
|
});
|
|
indexTokenCountMap = primeResult.indexTokenCountMap;
|
|
if (primeResult.inserted > 0) {
|
|
const manualNames = (manualSkillPrimes ?? []).map((p) => p.name);
|
|
const alwaysApplyNames = (alwaysApplySkillPrimes ?? []).map((p) => p.name);
|
|
logger.debug(
|
|
`[AgentClient] Primed ${primeResult.inserted} skill(s) at message index ${primeResult.insertIdx} — manual: [${manualNames.join(', ')}], always-apply: [${alwaysApplyNames.join(', ')}]`,
|
|
);
|
|
}
|
|
if (primeResult.alwaysApplyDropped > 0) {
|
|
logger.warn(
|
|
`[AgentClient] Dropped ${primeResult.alwaysApplyDropped} always-apply prime(s) to stay within MAX_PRIMED_SKILLS_PER_TURN.`,
|
|
);
|
|
}
|
|
}
|
|
|
|
if (indexTokenCountMap && isEnabled(process.env.AGENT_DEBUG_LOGGING)) {
|
|
const entries = Object.entries(indexTokenCountMap);
|
|
const perMsg = entries.map(([idx, count]) => {
|
|
const msg = initialMessages[Number(idx)];
|
|
const type = msg ? msg._getType() : '?';
|
|
return `${idx}:${type}=${count}`;
|
|
});
|
|
logger.debug(
|
|
`[AgentClient] Token map after format: [${perMsg.join(', ')}] (payload=${payload.length}, formatted=${initialMessages.length})`,
|
|
);
|
|
}
|
|
indexTokenCountMap = hydrateMissingIndexTokenCounts({
|
|
messages: initialMessages,
|
|
indexTokenCountMap,
|
|
tokenCounter,
|
|
});
|
|
|
|
const memoryMessages =
|
|
this.processMemory && this.memoryPayload
|
|
? formatAgentMessages(
|
|
stripActivityLabelParts(this.memoryPayload),
|
|
undefined,
|
|
toolSet,
|
|
skillPrimeResult?.skills,
|
|
formatOptions,
|
|
).messages
|
|
: initialMessages;
|
|
|
|
/**
|
|
* @param {BaseMessage[]} messages
|
|
*/
|
|
const runAgents = async (messages) => {
|
|
const agents = [this.options.agent];
|
|
// Include additional agents when:
|
|
// - agentConfigs has agents (from addedConvo parallel execution or agent handoffs)
|
|
// - Agents without incoming edges become start nodes and run in parallel automatically
|
|
if (this.agentConfigs && this.agentConfigs.size > 0) {
|
|
agents.push(...this.agentConfigs.values());
|
|
}
|
|
|
|
// TODO: needs to be added as part of AgentContext initialization
|
|
// const noSystemModelRegex = [/\b(o1-preview|o1-mini|amazon\.titan-text)\b/gi];
|
|
// const noSystemMessages = noSystemModelRegex.some((regex) =>
|
|
// agent.model_parameters.model.match(regex),
|
|
// );
|
|
// if (noSystemMessages === true && systemContent?.length) {
|
|
// const latestMessageContent = _messages.pop().content;
|
|
// if (typeof latestMessageContent !== 'string') {
|
|
// latestMessageContent[0].text = [systemContent, latestMessageContent[0].text].join('\n');
|
|
// _messages.push(new HumanMessage({ content: latestMessageContent }));
|
|
// } else {
|
|
// const text = [systemContent, latestMessageContent].join('\n');
|
|
// _messages.push(new HumanMessage(text));
|
|
// }
|
|
// }
|
|
// let messages = _messages;
|
|
// if (agent.useLegacyContent === true) {
|
|
// messages = formatContentStrings(messages);
|
|
// }
|
|
// if (
|
|
// agent.model_parameters?.clientOptions?.defaultHeaders?.['anthropic-beta']?.includes(
|
|
// 'prompt-caching',
|
|
// )
|
|
// ) {
|
|
// messages = addCacheControl(messages);
|
|
// }
|
|
|
|
if (this.processMemory) {
|
|
memoryPromise = this.runMemory(memoryMessages);
|
|
}
|
|
|
|
/** Seed calibration state from previous run if encoding matches */
|
|
const currentEncoding = this.getEncoding();
|
|
const prevMeta = this.contextMeta;
|
|
const encodingMatch = prevMeta?.encoding === currentEncoding;
|
|
const calibrationRatio =
|
|
encodingMatch && prevMeta?.calibrationRatio > 0 ? prevMeta.calibrationRatio : undefined;
|
|
|
|
if (prevMeta) {
|
|
logger.debug(
|
|
`[AgentClient] contextMeta from parent: ratio=${prevMeta.calibrationRatio}, encoding=${prevMeta.encoding}, current=${currentEncoding}, seeded=${calibrationRatio ?? 'none'}`,
|
|
);
|
|
}
|
|
|
|
const streamId = this.options.req?._resumableStreamId;
|
|
// HITL: clear any checkpoint orphaned by a prior paused turn in this
|
|
// conversation (one that expired or was aborted while paused) so this fresh
|
|
// turn starts clean instead of rehydrating a stale interrupt — thread_id is
|
|
// the stable conversationId. No-op when HITL is off or nothing is orphaned.
|
|
// Deliberately UNCONDITIONAL per HITL turn: any cheaper gate (job metadata,
|
|
// a Redis flag) can go stale across replicas/restarts and skip the prune
|
|
// exactly when an orphan exists, while these are two indexed, usually-empty
|
|
// deleteMany ops — correctness over a micro-optimization.
|
|
// The gate mirrors createRun's checkpointer condition: the approval policy
|
|
// OR an ask_user_question-capable agent (which attaches a checkpointer
|
|
// WITHOUT the approval policy) — an ask pause abandoned via job replacement
|
|
// or Stop would otherwise rehydrate here and silently duplicate context.
|
|
//
|
|
// Start the prune alongside graph construction. The all-settled barrier
|
|
// below still guarantees it completes before the graph is exposed or run.
|
|
const shouldPruneCheckpoint =
|
|
streamId &&
|
|
(isHITLEnabled(agentsEConfig?.toolApproval) || agents.some(agentRequestsAskUserQuestion));
|
|
const checkpointPrunePromise = shouldPruneCheckpoint
|
|
? deleteAgentCheckpoint(this.conversationId, agentsEConfig?.checkpointer)
|
|
: Promise.resolve();
|
|
|
|
const createRunPromise = createRun({
|
|
agents,
|
|
messages,
|
|
// This controller implements the full HITL pause/resume lifecycle (handleRunInterrupt
|
|
// persists the pending action; the /resume route rebuilds + continues the run), so it
|
|
// opts into the tool-approval wiring. Non-resumable callers (OpenAI-compat, Responses)
|
|
// leave this off so an approval-gated tool can't pause where there's no resume path.
|
|
hitlCapable: true,
|
|
toolInputValidationErrors: this.toolInputValidationErrors,
|
|
// Mid-run steering: drain queued user messages at each tool-batch
|
|
// boundary and inject them into graph state. The offset wrapper
|
|
// shifts SDK content indices past any spliced steer parts.
|
|
steering: this.buildSteerWiring(streamId),
|
|
activityLabel: this.buildActivityLabelWiring(streamId, abortController.signal),
|
|
indexTokenCountMap,
|
|
initialSummary,
|
|
initialSessions,
|
|
calibrationRatio,
|
|
runId: this.responseMessageId,
|
|
signal: abortController.signal,
|
|
customHandlers: createSteerIndexOffsetHandlers(
|
|
this.options.eventHandlers,
|
|
this.steerOffsetState,
|
|
),
|
|
requestBody: config.configurable.requestBody,
|
|
user: createSafeUser(this.options.req?.user),
|
|
tenantId: this.options.req?.user?.tenantId,
|
|
summarizationConfig: appConfig?.summarization,
|
|
appConfig,
|
|
tokenCounter,
|
|
/** Bills subagent child-run model calls — child graphs execute
|
|
* outside the streamEvents loop, so ModelEndHandler never sees
|
|
* them. Entries land in collectedUsage tagged
|
|
* `usage_type: 'subagent'` and are spent by recordCollectedUsage.
|
|
* The sink also streams each as an `on_token_usage` event so the
|
|
* gauge's session cost/totals include billed subagent usage (the
|
|
* `subagent` tag keeps it out of the live context meter). */
|
|
subagentUsageSink: createSubagentUsageSink(
|
|
this.collectedUsage,
|
|
this.buildSubagentUsageEmitter(appConfig),
|
|
),
|
|
}).then((createdRun) => {
|
|
if (!createdRun) {
|
|
throw new Error('Failed to create run');
|
|
}
|
|
this.options.startupTelemetry?.mark('run_created');
|
|
return createdRun;
|
|
});
|
|
|
|
const [createRunResult, checkpointPruneResult] = await Promise.allSettled([
|
|
createRunPromise,
|
|
checkpointPrunePromise,
|
|
]);
|
|
if (createRunResult.status === 'rejected') {
|
|
throw createRunResult.reason;
|
|
}
|
|
if (checkpointPruneResult.status === 'rejected') {
|
|
throw checkpointPruneResult.reason;
|
|
}
|
|
run = createRunResult.value;
|
|
|
|
this.run = run;
|
|
if (this._resolveRun) {
|
|
this._resolveRun(run);
|
|
this._resolveRun = null;
|
|
}
|
|
|
|
if (streamId && run.Graph) {
|
|
GenerationJobManager.setGraph(streamId, run.Graph, this.jobCreatedAt);
|
|
}
|
|
|
|
if (userMCPAuthMap != null) {
|
|
config.configurable.userMCPAuthMap = userMCPAuthMap;
|
|
}
|
|
|
|
/** @deprecated Agent Chain */
|
|
config.configurable.last_agent_id = agents[agents.length - 1].id;
|
|
|
|
this.options.startupTelemetry?.mark('stream_processing_started');
|
|
/** Flag durable BEFORE the run can claim a label: gap reconciliation
|
|
* is gated on it, and ordering it here (one settled-on-failure
|
|
* await) keeps the claim-time reservation emit immediate — see
|
|
* `emitLabelEvent` in buildActivityLabelWiring. */
|
|
if (this.activityLabelsMarkedPromise != null) {
|
|
await this.activityLabelsMarkedPromise;
|
|
}
|
|
await run.processStream({ messages }, config, {
|
|
callbacks: {
|
|
[Callback.TOOL_ERROR]: logToolError,
|
|
},
|
|
});
|
|
|
|
// HITL: if the run paused for tool approval, mark the job
|
|
// `requires_action` + emit the prompt and leave the turn unfinalized
|
|
// (the resume route continues it). No-op when the run completed.
|
|
await this.handleRunInterrupt(run, streamId);
|
|
|
|
config.signal = null;
|
|
};
|
|
|
|
this.options.startupTelemetry?.mark('run_input_prepared');
|
|
await runAgents(initialMessages);
|
|
|
|
/**
|
|
* Surface a completed `skill` tool_call content part per *manually*-
|
|
* primed skill so the existing `SkillCall` frontend renderer shows
|
|
* a "Skill X loaded" card on the assistant response. Applied after
|
|
* the graph finishes to avoid clashing with the aggregator's own
|
|
* per-step content indexing. Prepended (not appended) so cards sit
|
|
* above the model's output — priming ran before the turn, the
|
|
* reply follows.
|
|
*
|
|
* Always-apply primes intentionally do NOT emit assistant-side
|
|
* cards. `extractInvokedSkillsFromPayload` scans history for
|
|
* `skill` tool_calls and feeds `primeInvokedSkills`, which is
|
|
* Phase 3's sticky-re-prime path — that's the right behavior for
|
|
* manual (user picked `$skill` once; re-prime on every subsequent
|
|
* turn from history). For always-apply, `resolveAlwaysApplySkills`
|
|
* already re-primes every turn from fresh DB state, so persisting
|
|
* the card would cause the skill body to get primed twice per
|
|
* turn starting on turn 2. The user-facing acknowledgement for
|
|
* always-apply lives on the user bubble as the pinned
|
|
* `SkillPills` row (`message.alwaysAppliedSkills`), which
|
|
* is the durable signal the user wants: "this skill auto-primes".
|
|
*
|
|
* Live streaming display of manual user-bubble pills is handled
|
|
* by `SkillPills` reading `message.manualSkills`. No
|
|
* separate SSE emit is needed here; trying to stream a mid-run
|
|
* tool_call at index 0 collided with the LLM's first text
|
|
* content, while emitting at a sparse offset pushed the card
|
|
* below the reply on finalize. Post-run unshift keeps the final
|
|
* responseMessage.content in the right order.
|
|
*/
|
|
/**
|
|
* Settle in-flight label fills BEFORE the content is reshaped below.
|
|
* A fill emits its claim-time index; the skill-card unshift and the
|
|
* hide-sequential filter both shift positions, so a fill landing after
|
|
* either would emit a stale index — and a client that already synced
|
|
* the reshaped array applies it onto the wrong part. A paused turn
|
|
* never gets a final event to repair that. Costs nothing extra: these
|
|
* are the same promises the finalization settle would wait on, and
|
|
* that later call then sees an empty pending list.
|
|
*/
|
|
await this.settleActivityLabels();
|
|
|
|
const manualPrimed = this.options.agent?.manualSkillPrimes ?? [];
|
|
if (manualPrimed.length > 0) {
|
|
const runId = this.responseMessageId ?? 'skill-prime';
|
|
const manualParts = buildSkillPrimeContentParts(manualPrimed, { runId });
|
|
this.contentParts.unshift(...manualParts);
|
|
}
|
|
|
|
this.applyHideSequentialOutputsFilter();
|
|
} catch (err) {
|
|
if (abortController.signal.aborted) {
|
|
logger.debug(
|
|
'[api/server/controllers/agents/client.js #sendCompletion] Operation aborted by user',
|
|
{ conversationId: this.conversationId, name: err?.name, code: err?.code },
|
|
);
|
|
} else {
|
|
logger.error(
|
|
'[api/server/controllers/agents/client.js #sendCompletion] Unhandled error type',
|
|
err,
|
|
);
|
|
const videoError = resolveGoogleVideoError({
|
|
error: err,
|
|
provider: this.options.agent?.provider,
|
|
hasYouTubeVideo: this.injectedYouTubeVideo,
|
|
});
|
|
this.contentParts.push({
|
|
type: ContentTypes.ERROR,
|
|
[ContentTypes.ERROR]:
|
|
videoError ??
|
|
`An error occurred while processing the request${err?.message ? `: ${err.message}` : ''}`,
|
|
});
|
|
}
|
|
} finally {
|
|
/** Capture calibration state from the run for persistence on the response message.
|
|
* Runs in finally so values are captured even on abort. */
|
|
const ratio = this.run?.getCalibrationRatio() ?? 0;
|
|
if (ratio > 0 && ratio !== 1) {
|
|
this.contextMeta = {
|
|
calibrationRatio: Math.round(ratio * 1000) / 1000,
|
|
encoding: this.getEncoding(),
|
|
};
|
|
} else {
|
|
this.contextMeta = undefined;
|
|
}
|
|
|
|
this.finalizeSubagentContent();
|
|
await this.settleActivityLabels();
|
|
|
|
/** Flush subagent usage emits the sink fired without awaiting, so their
|
|
* persist/publish completes before we return and the job is cleaned up
|
|
* (resumed clients read this persisted usage). */
|
|
if (this.pendingSubagentEmits.length > 0) {
|
|
await Promise.allSettled(this.pendingSubagentEmits);
|
|
this.pendingSubagentEmits = [];
|
|
}
|
|
|
|
try {
|
|
const attachments = await this.awaitMemoryWithTimeout(memoryPromise);
|
|
if (attachments && attachments.length > 0) {
|
|
this.artifactPromises.push(...attachments);
|
|
}
|
|
|
|
/** Skip token spending if aborted - the abort handler (abortMiddleware.js) handles it
|
|
This prevents double-spending when user aborts via `/api/agents/chat/abort` */
|
|
const wasAborted = abortController?.signal?.aborted;
|
|
if (!wasAborted) {
|
|
await this.recordCollectedUsage({
|
|
context: 'message',
|
|
balance: balanceConfig,
|
|
transactions: transactionsConfig,
|
|
});
|
|
} else {
|
|
logger.debug(
|
|
'[api/server/controllers/agents/client.js #chatCompletion] Skipping token spending - handled by abort middleware',
|
|
);
|
|
}
|
|
} catch (err) {
|
|
logger.error(
|
|
'[api/server/controllers/agents/client.js #chatCompletion] Error in cleanup phase',
|
|
err,
|
|
);
|
|
}
|
|
if (this._resolveRun) {
|
|
this._resolveRun(this.run ?? null);
|
|
this._resolveRun = null;
|
|
}
|
|
|
|
// HITL: a non-paused turn deliberately prunes nothing here. The lazy checkpointer
|
|
// (LazyMongoSaver) never persists a clean-exit checkpoint, so there is
|
|
// nothing this turn left to delete. A checkpoint orphaned by a PRIOR abandoned pause
|
|
// is cleared by the pre-run prune (before processStream) on the next turn, with the
|
|
// Mongo TTL as the backstop. Dropping this post-completion prune also removes its
|
|
// job-replacement race: an older run's late finally can no longer delete a newer
|
|
// paused run's checkpoint, because there is no longer a clean-path prune to race.
|
|
|
|
run = null;
|
|
config = null;
|
|
memoryPromise = null;
|
|
}
|
|
}
|
|
|
|
/**
|
|
* Resume a run that paused for human-in-the-loop review.
|
|
*
|
|
* The original run lives in a detached background task that exits when the run
|
|
* pauses, so resume REBUILDS the run on a fresh graph bound to the same
|
|
* `thread_id` (= conversationId) and the durable checkpointer. LangGraph rehydrates
|
|
* the paused graph state from the checkpoint; `run.resume(value)` re-enters the
|
|
* interrupted node with the user's decision. State comes from the checkpoint, so
|
|
* no message history is rebuilt here — `createRun` only needs the agent(s) to
|
|
* reconstruct the graph structure.
|
|
*
|
|
* `seedContent` is the content streamed before the pause (the assistant message +
|
|
* its tool call). In Redis mode the job store's append log already spans the pause,
|
|
* so the finalized message is complete regardless; seeding keeps the in-memory store
|
|
* complete too. The run drives events through the same `streamId`, so the client's
|
|
* open SSE receives the continuation live.
|
|
*
|
|
* Unlike `chatCompletion`, this does NOT prune the checkpoint in its `finally` — the
|
|
* resume controller owns checkpoint lifecycle (it must also clean up on failures that
|
|
* happen before this method runs, and keep the checkpoint on a re-pause).
|
|
*
|
|
* @param {object} params
|
|
* @param {Agents.ToolApprovalDecisionMap | { answer: string }} params.resumeValue
|
|
* @param {Array} [params.seedContent] - content aggregated before the pause
|
|
* @param {Array<import('@librechat/agents').RunStep>} [params.runSteps] - run steps emitted before the pause
|
|
* @param {AbortController} [params.abortController]
|
|
* @param {Pick<import('@langchain/langgraph').Command, 'update' | 'goto'>} [params.commandOptions]
|
|
*/
|
|
async resumeCompletion({
|
|
resumeValue,
|
|
seedContent = [],
|
|
runSteps = [],
|
|
abortController = null,
|
|
commandOptions,
|
|
userMCPAuthMap,
|
|
discoveredToolNames,
|
|
}) {
|
|
/** @type {Partial<GraphRunnableConfig>} */
|
|
let config;
|
|
/** @type {ReturnType<createRun>} */
|
|
let run;
|
|
const appConfig = this.options.req.config;
|
|
const balanceConfig = getBalanceConfig(appConfig);
|
|
const transactionsConfig = getTransactionsConfig(appConfig);
|
|
try {
|
|
if (!abortController) {
|
|
abortController = new AbortController();
|
|
}
|
|
|
|
/** @type {AppConfig['endpoints']['agents']} */
|
|
const agentsEConfig = appConfig.endpoints?.[EModelEndpoint.agents];
|
|
|
|
config = {
|
|
runName: 'AgentRun',
|
|
configurable: {
|
|
thread_id: this.conversationId,
|
|
last_agent_index: this.agentConfigs?.size ?? 0,
|
|
user_id: this.user ?? this.options.req.user?.id,
|
|
hide_sequential_outputs: this.options.agent.hide_sequential_outputs,
|
|
requestBody: {
|
|
messageId: this.responseMessageId,
|
|
conversationId: this.conversationId,
|
|
parentMessageId: this.parentMessageId,
|
|
},
|
|
user: createSafeUser(this.options.req.user),
|
|
},
|
|
recursionLimit: resolveRecursionLimit(agentsEConfig, this.options.agent),
|
|
signal: abortController.signal,
|
|
streamMode: 'values',
|
|
version: 'v2',
|
|
};
|
|
|
|
// Seed pre-pause content so the in-memory job store reports the complete turn
|
|
// (Redis aggregates across the pause via its append log; this covers in-memory).
|
|
if (Array.isArray(seedContent) && seedContent.length > 0) {
|
|
this.contentParts.push(...seedContent);
|
|
}
|
|
|
|
const tokenCounter = createTokenCounter(this.getEncoding());
|
|
const agents = [this.options.agent];
|
|
if (this.agentConfigs && this.agentConfigs.size > 0) {
|
|
agents.push(...this.agentConfigs.values());
|
|
}
|
|
|
|
// Re-prime skill files invoked in the pre-pause segment (mirrors the normal path's
|
|
// `primeInvokedSkills(payload)`), so an approved code/file-backed tool keeps the
|
|
// injected skill-file session refs instead of running without them. The pre-pause
|
|
// content carries the `skill` tool_calls, so it stands in for the message payload.
|
|
let skillSessions;
|
|
if (
|
|
typeof this.options.primeInvokedSkills === 'function' &&
|
|
Array.isArray(seedContent) &&
|
|
seedContent.length > 0
|
|
) {
|
|
try {
|
|
const primed = await this.options.primeInvokedSkills([
|
|
{ role: 'assistant', content: seedContent },
|
|
]);
|
|
skillSessions = primed?.initialSessions;
|
|
} catch (err) {
|
|
logger.warn(
|
|
'[api/server/controllers/agents/client.js #resumeCompletion] Failed to re-prime skill sessions',
|
|
err?.message ?? err,
|
|
);
|
|
}
|
|
}
|
|
|
|
// Seed code-env / skill tool sessions so an approved code/file/skill-backed tool
|
|
// runs with the same uploaded-file context the pre-pause run had — the rebuilt
|
|
// graph otherwise has no `Graph.sessions` entries (especially cross-replica).
|
|
const initialSessions = buildInitialToolSessions({ skillSessions, agents });
|
|
|
|
const streamId = this.options.req?._resumableStreamId;
|
|
run = await createRun({
|
|
agents,
|
|
// State (messages, tool calls) is rehydrated from the checkpoint by
|
|
// run.resume; createRun only needs the agents to rebuild the graph.
|
|
messages: [],
|
|
// The resumed run can pause AGAIN (another tool, a follow-up question), and this
|
|
// controller owns that lifecycle, so it must keep the HITL wiring on the rebuilt run.
|
|
hitlCapable: true,
|
|
toolInputValidationErrors: this.toolInputValidationErrors,
|
|
// Steering stays live across a pause/resume cycle: steers queued while
|
|
// the resumed segment runs drain at its tool-batch boundaries.
|
|
steering: this.buildSteerWiring(streamId),
|
|
// Activity labels likewise survive pause/resume: post-resume tool
|
|
// batches keep claiming slots and generating group headers.
|
|
activityLabel: this.buildActivityLabelWiring(streamId, abortController.signal),
|
|
// Replay deferred tools discovered before the pause. With `messages: []` the
|
|
// discovery scan finds nothing, so a deferred tool the paused call targets
|
|
// would be absent from the rebuilt toolMap; these names (captured at pause)
|
|
// force it back in. Undefined/empty for non-deferred turns — a harmless no-op.
|
|
discoveredToolNames,
|
|
initialSessions,
|
|
runId: this.responseMessageId,
|
|
signal: abortController.signal,
|
|
// The rebuilt graph numbers content indices from 0, but the aggregator was
|
|
// just seeded with the pre-pause parts at those same indices — shift every
|
|
// resumed step index past the seed, or the new output merges into (or, on a
|
|
// type mismatch, is silently dropped against) the pre-pause content. The
|
|
// steer wrapper composes on top: resumed indices shift by seed + any
|
|
// steer parts spliced in while the resumed segment streams.
|
|
customHandlers: createSteerIndexOffsetHandlers(
|
|
createContentIndexOffsetHandlers(
|
|
this.options.eventHandlers,
|
|
Array.isArray(seedContent) ? seedContent : [],
|
|
),
|
|
this.steerOffsetState,
|
|
),
|
|
requestBody: config.configurable.requestBody,
|
|
user: createSafeUser(this.options.req?.user),
|
|
tenantId: this.options.req?.user?.tenantId,
|
|
summarizationConfig: appConfig?.summarization,
|
|
appConfig,
|
|
tokenCounter,
|
|
subagentUsageSink: createSubagentUsageSink(
|
|
this.collectedUsage,
|
|
this.buildSubagentUsageEmitter(appConfig),
|
|
),
|
|
});
|
|
|
|
if (!run) {
|
|
throw new Error('Failed to create run for resume');
|
|
}
|
|
|
|
hydrateResumeRunSteps(runSteps, this.stepMap, run.Graph, seedContent);
|
|
|
|
this.run = run;
|
|
if (this._resolveRun) {
|
|
this._resolveRun(run);
|
|
this._resolveRun = null;
|
|
}
|
|
|
|
// Do NOT cache the rebuilt graph on resume: it was created with `messages: []`, so
|
|
// RedisJobStore.getContentParts() (which prefers a cached graph over reconstructing
|
|
// from the chunk log) would return only the resumed segment and drop the pre-pause
|
|
// assistant/tool-call content on a same-replica reload/status poll. Skipping it makes
|
|
// introspection fall back to the durable chunk reconstruction, which is complete.
|
|
// `setContentParts` still points the in-memory store at the seeded client content.
|
|
if (streamId && this.contentParts) {
|
|
GenerationJobManager.setContentParts(streamId, this.contentParts, this.jobCreatedAt);
|
|
}
|
|
|
|
// Carry the user's MCP auth into the rebuilt run so an approved MCP tool executes
|
|
// with the same OAuth/user credentials it had before the pause.
|
|
if (userMCPAuthMap != null) {
|
|
config.configurable.userMCPAuthMap = userMCPAuthMap;
|
|
}
|
|
|
|
/** @deprecated Agent Chain */
|
|
config.configurable.last_agent_id = agents[agents.length - 1].id;
|
|
|
|
/** Same flag-before-run ordering as chatCompletion's processStream. */
|
|
if (this.activityLabelsMarkedPromise != null) {
|
|
await this.activityLabelsMarkedPromise;
|
|
}
|
|
await run.resume(
|
|
resumeValue,
|
|
config,
|
|
{ callbacks: { [Callback.TOOL_ERROR]: logToolError } },
|
|
commandOptions,
|
|
);
|
|
|
|
config.signal = null;
|
|
|
|
// The model may pause AGAIN (another tool needs approval, or a follow-up
|
|
// question). Re-arm the same interrupt gate so the cycle can repeat.
|
|
await this.handleRunInterrupt(run, streamId);
|
|
|
|
// Mirror chatCompletion: settle label fills before the filter below can
|
|
// shift part positions out from under an in-flight fill's claimed index.
|
|
await this.settleActivityLabels();
|
|
|
|
// Strip hidden intermediate sequential-agent content
|
|
// before resume finalize/re-pause persistence reads `this.contentParts`, so a
|
|
// resumed sequential chain doesn't persist/emit outputs hide_sequential_outputs
|
|
// is meant to hide.
|
|
this.applyHideSequentialOutputsFilter();
|
|
} catch (err) {
|
|
if (abortController.signal.aborted) {
|
|
logger.debug(
|
|
'[api/server/controllers/agents/client.js #resumeCompletion] Aborted by user',
|
|
{
|
|
conversationId: this.conversationId,
|
|
name: err?.name,
|
|
code: err?.code,
|
|
},
|
|
);
|
|
} else {
|
|
logger.error(
|
|
'[api/server/controllers/agents/client.js #resumeCompletion] Unhandled error',
|
|
err,
|
|
);
|
|
this.contentParts.push({
|
|
type: ContentTypes.ERROR,
|
|
[ContentTypes.ERROR]: `An error occurred while resuming the request${err?.message ? `: ${err.message}` : ''}`,
|
|
});
|
|
}
|
|
} finally {
|
|
const ratio = this.run?.getCalibrationRatio() ?? 0;
|
|
if (ratio > 0 && ratio !== 1) {
|
|
this.contextMeta = {
|
|
calibrationRatio: Math.round(ratio * 1000) / 1000,
|
|
encoding: this.getEncoding(),
|
|
};
|
|
} else {
|
|
this.contextMeta = undefined;
|
|
}
|
|
|
|
this.finalizeSubagentContent();
|
|
await this.settleActivityLabels();
|
|
|
|
if (this.pendingSubagentEmits.length > 0) {
|
|
await Promise.allSettled(this.pendingSubagentEmits);
|
|
this.pendingSubagentEmits = [];
|
|
}
|
|
|
|
try {
|
|
const wasAborted = abortController?.signal?.aborted;
|
|
if (!wasAborted) {
|
|
await this.recordCollectedUsage({
|
|
context: 'message',
|
|
balance: balanceConfig,
|
|
transactions: transactionsConfig,
|
|
});
|
|
}
|
|
} catch (err) {
|
|
logger.error(
|
|
'[api/server/controllers/agents/client.js #resumeCompletion] Error in cleanup phase',
|
|
err,
|
|
);
|
|
}
|
|
if (this._resolveRun) {
|
|
this._resolveRun(this.run ?? null);
|
|
this._resolveRun = null;
|
|
}
|
|
run = null;
|
|
config = null;
|
|
}
|
|
}
|
|
|
|
/**
|
|
* Resolves with the agent run once it is initialized, or `null` if
|
|
* initialization fails. Lets immediate-mode title generation await the run
|
|
* instead of throwing when fired before `chatCompletion` assigns `this.run`.
|
|
* Rejects promptly if the provided signal aborts before the run is ready.
|
|
* @param {AbortSignal} [signal]
|
|
* @returns {Promise<AgentRun | null>}
|
|
*/
|
|
_waitForRun(signal) {
|
|
if (this.run) {
|
|
return Promise.resolve(this.run);
|
|
}
|
|
if (!this._runReady) {
|
|
this._runReady = new Promise((resolve) => {
|
|
this._resolveRun = resolve;
|
|
});
|
|
}
|
|
if (!signal) {
|
|
return this._runReady;
|
|
}
|
|
if (signal.aborted) {
|
|
return Promise.reject(new Error('Aborted before run initialization'));
|
|
}
|
|
return new Promise((resolve, reject) => {
|
|
const onAbort = () => reject(new Error('Aborted before run initialization'));
|
|
signal.addEventListener('abort', onAbort, { once: true });
|
|
this._runReady.then((run) => {
|
|
signal.removeEventListener('abort', onAbort);
|
|
resolve(run);
|
|
});
|
|
});
|
|
}
|
|
|
|
/**
|
|
* @param {Object} params
|
|
* @param {string} params.text
|
|
* @param {AbortController} params.abortController
|
|
* @param {boolean} [params.immediate] When true, the title is generated as soon
|
|
* as the request is made — the run is awaited (instead of throwing) and the
|
|
* title derives from the user's input only (`contentParts` is empty).
|
|
*/
|
|
async titleConvo({ text, abortController, immediate = false }) {
|
|
if (!this.run) {
|
|
if (!immediate) {
|
|
throw new Error('Run not initialized');
|
|
}
|
|
await this._waitForRun(abortController?.signal);
|
|
if (!this.run) {
|
|
logger.debug(
|
|
'[api/server/controllers/agents/client.js #titleConvo] Run unavailable for immediate title generation',
|
|
);
|
|
return;
|
|
}
|
|
}
|
|
const { handleLLMEnd, collected: collectedMetadata } = createMetadataAggregator();
|
|
const { req, agent } = this.options;
|
|
|
|
if (req?.body?.isTemporary) {
|
|
logger.debug(
|
|
`[api/server/controllers/agents/client.js #titleConvo] Skipping title generation for temporary conversation`,
|
|
);
|
|
return;
|
|
}
|
|
|
|
const appConfig = req.config;
|
|
let endpoint = agent.endpoint;
|
|
|
|
/** @type {import('@librechat/agents').ClientOptions} */
|
|
let clientOptions = {
|
|
model: agent.model || agent.model_parameters.model,
|
|
};
|
|
|
|
let titleProviderConfig = getProviderConfig({ provider: endpoint, appConfig });
|
|
|
|
/** @type {TEndpoint | undefined} */
|
|
const endpointConfig =
|
|
appConfig.endpoints?.all ??
|
|
appConfig.endpoints?.[endpoint] ??
|
|
titleProviderConfig.customEndpointConfig;
|
|
if (!endpointConfig) {
|
|
logger.debug(
|
|
`[api/server/controllers/agents/client.js #titleConvo] No endpoint config for "${endpoint}"`,
|
|
);
|
|
}
|
|
|
|
if (endpointConfig?.titleConvo === false) {
|
|
logger.debug(
|
|
`[api/server/controllers/agents/client.js #titleConvo] Title generation disabled for endpoint "${endpoint}"`,
|
|
);
|
|
return;
|
|
}
|
|
|
|
if (endpointConfig?.titleEndpoint && endpointConfig.titleEndpoint !== endpoint) {
|
|
try {
|
|
titleProviderConfig = getProviderConfig({
|
|
provider: endpointConfig.titleEndpoint,
|
|
appConfig,
|
|
});
|
|
endpoint = endpointConfig.titleEndpoint;
|
|
} catch (error) {
|
|
logger.warn(
|
|
`[api/server/controllers/agents/client.js #titleConvo] Error getting title endpoint config for "${endpointConfig.titleEndpoint}", falling back to default`,
|
|
error,
|
|
);
|
|
// Fall back to original provider config
|
|
endpoint = agent.endpoint;
|
|
titleProviderConfig = getProviderConfig({ provider: endpoint, appConfig });
|
|
}
|
|
}
|
|
|
|
if (
|
|
endpointConfig &&
|
|
endpointConfig.titleModel &&
|
|
endpointConfig.titleModel !== Constants.CURRENT_MODEL
|
|
) {
|
|
clientOptions.model = endpointConfig.titleModel;
|
|
}
|
|
|
|
const options = await titleProviderConfig.getOptions({
|
|
req,
|
|
endpoint,
|
|
model_parameters: clientOptions,
|
|
db: {
|
|
getUserKey: db.getUserKey,
|
|
getUserKeyValues: db.getUserKeyValues,
|
|
},
|
|
});
|
|
|
|
let provider = options.provider ?? titleProviderConfig.overrideProvider ?? agent.provider;
|
|
if (
|
|
endpoint === EModelEndpoint.azureOpenAI &&
|
|
options.llmConfig?.azureOpenAIApiInstanceName == null
|
|
) {
|
|
provider = Providers.OPENAI;
|
|
} else if (
|
|
endpoint === EModelEndpoint.azureOpenAI &&
|
|
options.llmConfig?.azureOpenAIApiInstanceName != null &&
|
|
provider !== Providers.AZURE
|
|
) {
|
|
provider = Providers.AZURE;
|
|
}
|
|
|
|
/** @type {import('@librechat/agents').ClientOptions} */
|
|
clientOptions = { ...options.llmConfig };
|
|
if (options.configOptions) {
|
|
clientOptions.configuration = options.configOptions;
|
|
}
|
|
|
|
if (clientOptions.maxTokens != null) {
|
|
delete clientOptions.maxTokens;
|
|
}
|
|
if (clientOptions?.modelKwargs?.max_completion_tokens != null) {
|
|
delete clientOptions.modelKwargs.max_completion_tokens;
|
|
}
|
|
if (clientOptions?.modelKwargs?.max_output_tokens != null) {
|
|
delete clientOptions.modelKwargs.max_output_tokens;
|
|
}
|
|
|
|
/** `omitTitleOptions` drops the Anthropic `clientOptions` carrier (thinking,
|
|
* streaming, etc.), which would also drop its `defaultHeaders` — preserve the
|
|
* original `clientOptions` object so gateway/reverse-proxy metadata still
|
|
* reaches title requests (the proxy may require it for auth/routing). Restore
|
|
* the SAME object reference, not a copy: the Vertex `createClient` closure from
|
|
* `getLLMConfig` closes over this object, so `resolveConfigHeaders` must mutate
|
|
* the very object the client is built from. */
|
|
const anthropicClientOptions = clientOptions?.clientOptions;
|
|
|
|
clientOptions = Object.assign(
|
|
Object.fromEntries(
|
|
Object.entries(clientOptions).filter(([key]) => !omitTitleOptions.has(key)),
|
|
),
|
|
);
|
|
|
|
if (anthropicClientOptions?.defaultHeaders != null && clientOptions.clientOptions == null) {
|
|
clientOptions.clientOptions = anthropicClientOptions;
|
|
}
|
|
|
|
if (
|
|
provider === Providers.GOOGLE &&
|
|
(endpointConfig?.titleMethod === TitleMethod.FUNCTIONS ||
|
|
endpointConfig?.titleMethod === TitleMethod.STRUCTURED)
|
|
) {
|
|
clientOptions.json = true;
|
|
}
|
|
|
|
/** Resolve request-based headers across provider-specific header locations:
|
|
* OpenAI `configuration.defaultHeaders`, Anthropic `clientOptions.defaultHeaders`
|
|
* (preserved above), and Google `customHeaders`.
|
|
*/
|
|
resolveConfigHeaders({
|
|
llmConfig: clientOptions,
|
|
user: createSafeUser(this.options.req?.user),
|
|
body: {
|
|
messageId: this.responseMessageId,
|
|
conversationId: this.conversationId,
|
|
parentMessageId: this.parentMessageId,
|
|
},
|
|
});
|
|
|
|
try {
|
|
const titleResult = await this.run.generateTitle({
|
|
provider,
|
|
clientOptions,
|
|
inputText: text,
|
|
contentParts: immediate ? [] : this.contentParts,
|
|
titleMethod: endpointConfig?.titleMethod,
|
|
titlePrompt: endpointConfig?.titlePrompt,
|
|
titlePromptTemplate: endpointConfig?.titlePromptTemplate,
|
|
chainOptions: {
|
|
runName: 'TitleRun',
|
|
signal: abortController.signal,
|
|
callbacks: [
|
|
{
|
|
handleLLMEnd,
|
|
},
|
|
],
|
|
configurable: {
|
|
thread_id: this.conversationId,
|
|
user_id: this.user ?? this.options.req.user?.id,
|
|
},
|
|
},
|
|
});
|
|
|
|
const collectedUsage = collectedMetadata.map((item) => {
|
|
let input_tokens, output_tokens;
|
|
|
|
if (item.usage) {
|
|
input_tokens =
|
|
item.usage.prompt_tokens || item.usage.input_tokens || item.usage.inputTokens;
|
|
output_tokens =
|
|
item.usage.completion_tokens || item.usage.output_tokens || item.usage.outputTokens;
|
|
} else if (item.tokenUsage) {
|
|
input_tokens = item.tokenUsage.promptTokens;
|
|
output_tokens = item.tokenUsage.completionTokens;
|
|
} else if (item.usage_metadata) {
|
|
input_tokens = item.usage_metadata.input_tokens;
|
|
output_tokens = item.usage_metadata.output_tokens;
|
|
}
|
|
|
|
return {
|
|
input_tokens: input_tokens,
|
|
output_tokens: output_tokens,
|
|
};
|
|
});
|
|
|
|
const balanceConfig = getBalanceConfig(appConfig);
|
|
const transactionsConfig = getTransactionsConfig(appConfig);
|
|
await this.recordCollectedUsage({
|
|
collectedUsage,
|
|
context: 'title',
|
|
model: clientOptions.model,
|
|
balance: balanceConfig,
|
|
transactions: transactionsConfig,
|
|
messageId: this.responseMessageId,
|
|
}).catch((err) => {
|
|
logger.error(
|
|
'[api/server/controllers/agents/client.js #titleConvo] Error recording collected usage',
|
|
err,
|
|
);
|
|
});
|
|
|
|
return sanitizeTitle(titleResult.title);
|
|
} catch (err) {
|
|
logger.error('[api/server/controllers/agents/client.js #titleConvo] Error', err);
|
|
return;
|
|
}
|
|
}
|
|
|
|
/**
|
|
* @param {object} params
|
|
* @param {number} params.promptTokens
|
|
* @param {number} params.completionTokens
|
|
* @param {string} [params.model]
|
|
* @param {OpenAIUsageMetadata} [params.usage]
|
|
* @param {AppConfig['balance']} [params.balance]
|
|
* @param {string} [params.context='message']
|
|
* @returns {Promise<void>}
|
|
*/
|
|
async recordTokenUsage({
|
|
model,
|
|
usage,
|
|
balance,
|
|
promptTokens,
|
|
completionTokens,
|
|
context = 'message',
|
|
}) {
|
|
try {
|
|
await db.spendTokens(
|
|
{
|
|
model,
|
|
context,
|
|
balance,
|
|
messageId: this.responseMessageId,
|
|
conversationId: this.conversationId,
|
|
user: this.user ?? this.options.req.user?.id,
|
|
endpointTokenConfig: this.options.endpointTokenConfig,
|
|
},
|
|
{ promptTokens, completionTokens },
|
|
);
|
|
|
|
if (
|
|
usage &&
|
|
typeof usage === 'object' &&
|
|
'reasoning_tokens' in usage &&
|
|
typeof usage.reasoning_tokens === 'number'
|
|
) {
|
|
await db.spendTokens(
|
|
{
|
|
model,
|
|
balance,
|
|
context: 'reasoning',
|
|
messageId: this.responseMessageId,
|
|
conversationId: this.conversationId,
|
|
user: this.user ?? this.options.req.user?.id,
|
|
endpointTokenConfig: this.options.endpointTokenConfig,
|
|
},
|
|
{ completionTokens: usage.reasoning_tokens },
|
|
);
|
|
}
|
|
} catch (error) {
|
|
logger.error(
|
|
'[api/server/controllers/agents/client.js #recordTokenUsage] Error recording token usage',
|
|
error,
|
|
);
|
|
}
|
|
}
|
|
|
|
/** Anthropic Claude models use a distinct BPE tokenizer; all others default to o200k_base. */
|
|
getEncoding() {
|
|
if (this.model && this.model.toLowerCase().includes('claude')) {
|
|
return 'claude';
|
|
}
|
|
return 'o200k_base';
|
|
}
|
|
}
|
|
|
|
module.exports = AgentClient;
|