mirror of
https://github.com/danny-avila/LibreChat.git
synced 2026-11-18 14:08:21 +00:00
63 commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
8718338728
|
⏱️ fix: Bound Outbound Calls Made Inside a Request (#17263)
Some checks are pending
Backend Unit Tests / Build packages (push) Waiting to run
Backend Unit Tests / Codegraph select (push) Waiting to run
Backend Unit Tests / TypeScript type checks (push) Blocked by required conditions
Backend Unit Tests / Circular dependency checks (push) Waiting to run
Backend Unit Tests / Tests: api (shard 1/3) (push) Blocked by required conditions
Backend Unit Tests / Tests: api (shard 2/3) (push) Blocked by required conditions
Backend Unit Tests / Tests: api (shard 3/3) (push) Blocked by required conditions
Backend Unit Tests / Tests: data-provider (push) Blocked by required conditions
Backend Unit Tests / Tests: data-schemas (push) Blocked by required conditions
Backend Unit Tests / Tests: @librechat/api (shard 1/4) (push) Blocked by required conditions
Backend Unit Tests / Tests: @librechat/api (shard 2/4) (push) Blocked by required conditions
Backend Unit Tests / Tests: @librechat/api (shard 3/4) (push) Blocked by required conditions
Backend Unit Tests / Tests: @librechat/api (shard 4/4) (push) Blocked by required conditions
Codegraph E2E Votes / vote (full suite) (push) Waiting to run
Docker Dev Branch Images Build / publish (push) Waiting to run
Frontend Unit Tests / Client build recovery regression (push) Waiting to run
Frontend Unit Tests / Codegraph select (push) Waiting to run
Frontend Unit Tests / Build packages (push) Waiting to run
Frontend Unit Tests / TypeScript type checks (client) (push) Blocked by required conditions
Frontend Unit Tests / Tests: @librechat/chat (push) Blocked by required conditions
Frontend Unit Tests / Tests: @librechat/client (push) Blocked by required conditions
Frontend Unit Tests / Tests: Ubuntu (shard 1/2) (push) Blocked by required conditions
Frontend Unit Tests / Tests: Ubuntu (shard 2/2) (push) Blocked by required conditions
Frontend Unit Tests / Vite build verification (push) Blocked by required conditions
* fix: Bound Outbound Calls Made Inside a Request RAG queries for file search and message context, OCR provider calls, action HTTP calls, the moderation check, the stored-image fetch for base64 providers, and Flux result polling had no timeout. A stalled upstream hung the chat turn, the upload or the tool call indefinitely, and Flux kept polling forever on statuses that never become Ready. Each now runs under a budget from a new librechat.yaml timeouts block with finite defaults (getOutboundTimeoutMs in @librechat/api). Action calls also receive the run's abort signal, so stopping a run closes the socket. * test: Cover a Stalled RAG Query in the Mock Harness * fix: Share One Flux Budget Across Submit and Polls, Bound the Vertex Token Exchange * refactor: Poll Flux Results Through pollUntilSettled * refactor: Leave Flux Polling for a Separate Migration * fix: Cap Configured Timeouts at the Node Timer Limit * fix: Bound the Action Token Refresh and the Whole Image Download * fix: Cap Every Outbound Budget by Wall-Clock Time Axios's timeout only fires on an idle socket, so a RAG, OCR, moderation, action or token response that keeps trickling bytes never ended. outboundRequestBudget pairs each timeout with an AbortSignal deadline, and action calls combine it with the run's own signal. * fix: Name the Budget That Ran Out and Give Actions Five Minutes The actions default rises to five minutes, since action endpoints are arbitrary third-party APIs and the old minute could cut off legitimately slow ones after an upgrade. A budget overrun now fails with an error naming the timeouts key and its value, whether it was the idle timeout or the wall-clock deadline, and logAxiosError logs it as a warning so operators know what to raise. A deadline cancellation is no longer read as a user stop, so an action that runs out of time reports a timeout instead of looking like a stopped run. * fix: Hold Outbound Deadlines Weakly So Finished Requests Leave No Timers * fix: Check the Budget Before Reading an Error Message * fix: Report a Wrapped Deadline as a Failure, Not a Stop * fix: Bound the OAuth Code Exchange and Keep the OCR Budget Hint When Protected * chore: Sort Imports in axios.ts After Rebase |
||
|
|
5fc4371c6a
|
🪦 refactor: Remove Dead Legacy API Modules and Root Tooling (#17114) | ||
|
|
44703da3c6
|
🫖 fix: Skip Blank RAG Queries Before Embedding (#16231)
Some checks are pending
Backend Unit Tests / Build packages (push) Waiting to run
Backend Unit Tests / Codegraph select (push) Waiting to run
Backend Unit Tests / TypeScript type checks (push) Blocked by required conditions
Backend Unit Tests / Circular dependency checks (push) Waiting to run
Backend Unit Tests / Tests: api (shard 1/3) (push) Blocked by required conditions
Backend Unit Tests / Tests: api (shard 2/3) (push) Blocked by required conditions
Backend Unit Tests / Tests: api (shard 3/3) (push) Blocked by required conditions
Backend Unit Tests / Tests: data-provider (push) Blocked by required conditions
Backend Unit Tests / Tests: data-schemas (push) Blocked by required conditions
Backend Unit Tests / Tests: @librechat/api (shard 1/4) (push) Blocked by required conditions
Backend Unit Tests / Tests: @librechat/api (shard 2/4) (push) Blocked by required conditions
Backend Unit Tests / Tests: @librechat/api (shard 3/4) (push) Blocked by required conditions
Backend Unit Tests / Tests: @librechat/api (shard 4/4) (push) Blocked by required conditions
Codegraph E2E Votes / vote (full suite) (push) Waiting to run
Docker Dev Branch Images Build / publish (push) Waiting to run
Frontend Unit Tests / Client build recovery regression (push) Waiting to run
Frontend Unit Tests / Codegraph select (push) Waiting to run
Frontend Unit Tests / Build packages (push) Waiting to run
Frontend Unit Tests / TypeScript type checks (client) (push) Blocked by required conditions
Frontend Unit Tests / Tests: @librechat/client (push) Blocked by required conditions
Frontend Unit Tests / Tests: Ubuntu (shard 1/2) (push) Blocked by required conditions
Frontend Unit Tests / Tests: Ubuntu (shard 2/2) (push) Blocked by required conditions
Frontend Unit Tests / Vite build verification (push) Blocked by required conditions
Co-authored-by: Lia <lia@librechat.ai> |
||
|
|
676cd512b0
|
🫙 fix: Drop Blank Content Blocks From Promptless Sends (#15530)
* 🫙 fix: Drop Blank Content Blocks From Promptless Sends
* fix: Preserve the promptless turn instead of dropping it
|
||
|
|
a2ad0aa0c8
|
🤐 feat: Allow Promptless Sends When Files Are Attached (#13717)
* ✨ feat: Allow sending file attachments without a text message When an agent asks the user to upload a document, the user could attach the file but still had to type a placeholder message ("OK", "Here is the file") before the send button enabled and the submit guard let the message through. Attachments now count as submittable content: - New isSubmittableMessage(text, fileCount) util: non-whitespace text OR at least one attached file. - ask() in useChatFunctions uses it instead of bailing on empty text, so an empty draft with attached files submits. - SendButton receives the attached file count and enables accordingly. - ChatForm only marks the text field as required when no files are attached, so react-hook-form validation no longer blocks handleSubmit. Submitting an empty draft with no attachments is still rejected at all three layers. Fixes #13646 * Address review: support replayed file-only turns + drop empty vision text - ask(): count replayed attachments (overrideFiles) in the submittable check and skip it entirely for regenerate, so a file-only message can be regenerated or saved-and-resubmitted instead of being rejected as empty. - formatVisionMessage(): omit the text content part when the message text is empty. Anthropic rejects empty text content blocks with HTTP 400, and an empty block adds nothing for other providers; image-only sends now format cleanly. Added formatMessages tests for with-text and image-only (Anthropic + other) cases. * Address review: keep attachment-only turns valid for providers, answer mode, and titles - formatMessage: substitute minimal text when a user turn carries files but no inline content, so Anthropic does not reject an empty user message for RAG or code-environment attachments. - assistants chatV1: send the same stand-in for attachment-only Threads messages, which reject an empty body. The persisted message keeps empty text. - ChatForm: attachments no longer make an empty draft submittable in answer mode, where submitText consumes the click without answering or sending. - agents request: seed title generation from attachment filenames when the turn has no text, so immediate-mode titles are not invented from an empty string. - useChatFunctions.regenerate.spec: mock the utils barrel over the real module so new exports resolve. * Cover the agents path for attachment-only turns AgentClient formats its payload with the SDK's formatMessage, not the local one, so the earlier guard missed the endpoint the feature actually targets: an attachment-only turn still reached Anthropic as an empty user message. Apply the same stand-in after the file-context and quote merges, so a turn that already gained inline content is untouched. * Carry filenames on freshly attached files The fresh-file submission mapping copied only file_id, filepath, type, and dimensions, so the attachment-only title fallback read an undefined filename and produced nothing. Include filename, and cover it with a test that submits an empty draft with one attachment. * Address review: cover assistants v2, fresh agent attachments, editor, and title fallback - agents client: the current turn has no files during buildMessages, so read the resolved attachments from message_file_map instead. The previous guard only ever fired for persisted historical turns. - assistants chatV2: the default assistants endpoint routes here, so it needs the same stand-in body chatV1 got. - assistants title: fall back to filenames, then the response, and keep the default title rather than saving an empty one. - EditMessage: retained attachments make an empty edit submittable, matching the composer, so the overrideFiles replay path is reachable from the UI. --------- Co-authored-by: Marco Beretta <81851188+berry-13@users.noreply.github.com> |
||
|
|
7b6900d556
|
🏷️ feat: Activity Groups With Fast-Model Headers (#14391)
* ✨ feat: Activity Groups with Fast-Model Labels Groups each contiguous block of reasoning + tool calls into a collapsible unit headed by a fast-model label (claude.ai-style hierarchy), off the critical path: a PostToolBatch hook claims a live content slot at the batch boundary (steering index-offset pattern), renders a deterministic counts phrase instantly, and swaps in the generated label ~1s later while the next model call streams. Labels are UI-only — stripped before the SDK formatter and skipped in the legacy formatter — and reach live clients via a dedicated on_activity_label SSE event (live/replay/pending paths). Grouping preserves legacy rendering byte-for-byte when no label part is present. Generation bridges to Run.generateActivityLabel() when the SDK ships it (session-grouped Langfuse tracing); falls back to a direct call today. Env-gated: ACTIVITY_LABELS_POC=true, ACTIVITY_LABEL_MODEL. * 🧷 fix: Address Codex and Copilot Review Findings for Activity Labels - Settle in-flight label fills (bounded 3s) before finalization on both the main and resume paths, so a label resolving during the final batch still reaches the durable log and saved message. - Overlay on_activity_label chunks in RedisJobStore content reconstruction (splice path last-wins per index; replay path chronological overwrite), matching steer handling. - Wire activity labels into the HITL resume createRun so post-resume batches keep claiming slots. - Guard against out-of-order publishes: fill() awaits the claim emit before emitting the resolved label, and the client applier ignores a stale pending placeholder once a resolved label is present. - Stamp the batch's groupId onto label parts so parallel-column runs place them inside their group instead of filtering them out. - Localize the counts fallback phrase (10 keys, singular/plural) through useLocalize across chat rendering and exports. - Type the hook with Providers/ClientOptions instead of stringly types; drop the unknown cast in the spec; add a dedicated rAF retry ref for label events with effect cleanup. * 🛡️ fix: Address Independent Review — Abort, Usage, Lane Context, Redis Test - Propagate the run abort signal into label generation (both wiring call sites; runtime combines host + dispatch signals with the timeout) so a user abort cancels in-flight label calls instead of paying to timeout. - Record label-call usage like titles: the SDK bridge aggregates via chainOptions callbacks, the fallback path via a per-generation callback factory; both feed recordCollectedUsage under context 'activity-label'. - Scope block-context capture: reasoning collection stops at the previous block's label part and filters by executingAgentId, so consecutive or parallel batches can no longer bleed another block's thinking into the payload; intent text still scans past labels (persists across batches). - Forward the effective charLimit to the SDK call so host and SDK prompts agree (SDK default aligned to 600 in agents#327). - Add a Redis integration test proving last-write-wins reconstruction of on_activity_label chunks per claimed index. - Rebased onto main: only the two activity commits replay (the nine steering commits belonged to the old base branch), zero conflicts, steering suites green. * 📐 refactor: Move Activity-Label Wiring to TypeScript, Address Codex Round 2 - [P1] Slot claiming, lane stamping, emit ordering, context capture, and settle tracking now live in packages/api (createActivityLabelWiring + captureActivityBlockContext); client.js is a thin closure wrapper. - Register the activity-label hook BEFORE the steer drain so a steer draining at the same batch boundary cannot flush the tool block and orphan the label outside its group. - Resolve request-based header placeholders in resolveActivityLabelLLM (titleConvo parity) so metadata-keyed proxies work on label calls. - Trim labels centrally before filling so whitespace-only output from either generation path keeps the deterministic counts fallback. * 🧭 fix: Codex Round 3 — Capture Order, Shared Strip, Token Estimator, Hide Filter - Capture block context BEFORE pushing the label part: the scan stops at ACTIVITY_LABEL parts, so post-push capture hit the just-inserted label and silently collected no reasoning excerpts (regression test added). - Share stripActivityLabelParts from packages/api and apply it in the Responses and OpenAI-compatible controllers, closing the replay leak for entry points still running SDKs without the formatter skip. - Exclude activity_label parts from the fallback response-token estimator (UI-only parts must not inflate no-usage provider billing). - Keep label parts explicitly under hide_sequential_outputs — they summarize exactly the outputs that mode hides. * 🔁 fix: Codex Round 4 — Resume Gap, Delta Flush, Agent-Scoped Intent, Token Counter - Synthesize on_activity_label events for labels claimed or filled in the snapshot→subscribe window (the publish is fire-and-forget, so Redis-mode reconnects missed them). Feature-gated so the default path adds no content re-read; the client applier already ignores duplicates. - Flush queued deltas before applying a label part, matching the pending- action and steer appliers — without it the handler read a stale message cache and syncStepMessage pushed a pre-delta copy back. - Skip another agent's tail text when resolving intent, so parallel runs cannot seed a label prompt with a sibling agent's narration. - Exclude activity_label parts from countFormattedMessageTokens (the agent-path counter), not just the legacy BaseClient one. * 🏗️ refactor: Codex Round 5 — Extract Label Host Logic, Report Usage, Icon Strip - Move provider/model resolution, usage-metadata mapping, and the settle loop into packages/api (activityLabels/host.ts); client.js keeps only thin delegations, per the repo's TypeScript-implementation convention. - Fold label usage into the response rollup with an 'activity-label' tag (subagent precedent) so metadata.usage and the live cost gauge account for it; tagged, so it stays out of PRIMARY usage/context pairing. - Narrow tool metadata once in ToolCallGroup so THINK parts in a labeled block no longer render phantom generic icons in the stacked strip. - Import the activity-label helpers by deep path in GenerationJobManager: the package barrel now reaches provider-config/cache modules that import back into the stream layer, and the cycle broke suite loading. Declined: resetting steerOffsetState before HITL resume — resume builds a FRESH AgentClient via initializeClient (initialize.js:978), so the offset is already zero; the seed wrapper alone accounts for pre-pause parts. * 🚦 fix: Codex Round 6 — Stream Label Usage, Close Late Fills - Emit an on_token_usage chunk for label calls (sink push alone left the live session gauge blind); retained in pendingSubagentEmits so job cleanup cannot race the persist, tagged 'activity-label' as before. - Close the label scope when settle times out: the wiring gates fill() on isClosed and the client fires a label-scoped AbortController, so a straggling generation can neither mutate a saved response nor emit into a job whose runtime is gone. The controller also chains to the run signal, so a user abort still cancels label work. * 🩹 fix: Repair CI — Package Typecheck and Module Mocks Local runs covered the client tsconfig and jest, but never packages/api's own tsconfig, so nine type errors in the extracted host module shipped. - Type host.ts against the real contracts: ServerRequest, EndpointDbMethods, AppConfig from @librechat/data-schemas, IUser for createSafeUser, and a MaybeAzureConfig view for the azure instance-name probe and configuration. - Widen resolveConfigHeaders' llmConfig to Partial<RunLLMConfig>: it only reads the three provider header carriers, so auxiliary generations with a bare ClientOptions can resolve headers without assembling a run config. Type-only widening; every existing caller still satisfies it. - Add stripActivityLabelParts to the @librechat/api mock in the OpenAI and Responses controller specs — those mocks enumerate exports, so a new import read as undefined and threw before the assertions ran. - Use the real activity-label helpers in the ToolCallGroup spec's ~/utils mock; stubbing them out would hide the header logic under test. * ⚙️ feat: Configure Activity Labels via librechat.yaml, Drop Env Vars Replaces the ACTIVITY_LABELS_POC / ACTIVITY_LABEL_MODEL env gate with per-endpoint settings, following the title options convention rather than a top-level block — each endpoint picks its own cheap label model. - Add activity, activityModel, activityEndpoint, activityPrompt, activityMaxPerRun, and activityCharLimit to the endpoint schema, and to the endpoints.all pick list (enumerated, so 'all:' would otherwise drop them silently). - resolveActivityConfig reads them with title-style precedence: endpoints.all > named endpoint > custom endpoint config. - Model precedence is now activityModel > titleModel > the agent's model. activityEndpoint runs labels on another endpoint's credentials, with titleConvo's fallback-on-unknown-name behavior. - Thread activityPrompt/MaxPerRun/CharLimit through the wiring into the hook and the SDK bridge; they were hardcoded defaults. - The resume gap-repair gate keyed on the env var; it now keys on the snapshot actually containing label parts, so deployments without the feature still perform no extra content read. - Document the fields in librechat.example.yaml; add host.spec.ts covering precedence, custom-endpoint fallback, and opt-out. * 📝 refactor: Rename Enable Flag to activityLabel, Document Schema Inheritance - Rename the boolean from `activity` to `activityLabel`, matching the titleConvo/titleModel shape: a verb-object toggle whose prefix matches its modifiers (activityModel, activityPrompt, ...). `activity: true` alone read ambiguously — it could mean tracking or logging activity. - Document the two endpoint-schema inheritance paths, which behave oppositely and are ~900 lines apart: * `endpoints.all` omits from baseEndpointSchema, so new options are inherited automatically — nothing to maintain. * `azureEndpointSchema` enumerates via .pick(), so a new option is silently unavailable on Azure endpoints until listed there. The activity block now carries a pointer to the Azure caveat. * 🔍 fix: Address Codex Findings on the Config Rework - Pass the matched custom-endpoint config into the label gate. Custom endpoints live in the `endpoints.custom` ARRAY, so without it every custom endpoint resolved as disabled — including the example this PR added to librechat.example.yaml. - Give label usage a unique `runId:seq`. Label usage is billed but never appended to `collectedUsage`, so its length was static: every label event reused the last primary usage's pair and collided with itself, and the client dedupes on exactly that. - Attach `cost` to label usage when `interface.contextCost` is on; aggregateEmittedUsage treats coverage as all-or-nothing, so an event without it suppressed the whole response's cost. - Honor `activityPrompt` on the direct fallback path, not just the SDK bridge — it previously always used the built-in instruction. - Seed the per-response label cap from labels already on the response so a HITL resume cannot mint a fresh quota after every approval. - Reconcile label gaps on resume via a durable per-job `activityLabels` flag instead of probing the snapshot: the FIRST label of a run can be claimed inside the snapshot->subscribe window, which the old signal missed. The flag is read from a job record already fetched there, so runs without the feature still add no content read. - Auto-collapse labeled single-tool groups; one-call batches are common in agent runs and rendering them expanded defeats the grouping. * 🎯 fix: Correct Label Usage Seq, Cross-Endpoint Pricing, Close Scopes - Give label usage a NEGATIVE seq namespace. The previous fix was wrong: seq is a position in `collectedUsage` (push, then emit with the new length), so sink-length + array-length still lands on a real position — primary emits 1, the label computes 2, the next primary also emits 2. Labels have no position at all (billed separately, never appended), so they now occupy a namespace positional sequences cannot reach. The client key is a string used for Set membership, so the sign is inert. - Price cross-endpoint labels with the LABEL endpoint's token config: resolveActivityLabelModel now returns the resolved endpointTokenConfig, and both the streamed cost and recordCollectedUsage use it instead of the agent endpoint's rates. - Make close state per-wiring rather than per-client. A HITL resume rebuilds the wiring, and resetting a shared flag re-opened closures from the pre-pause segment whose provider call ignored the abort; settle now closes every retained scope, past generations included. * 🎯 fix: Make the Activity Header Say Something the Cards Cannot The header read "ran 1 command" next to a card already labeled "Code" — it restated the UI beneath it instead of adding to it. Two causes, both about content rather than timing: - A deterministic tool-type tally was the primary display and also fed the prompt, so the best case was a tally and the worst case was a tally dressed as prose. Removed from the metadata, the prompt, the part type, and the client. - The instruction only ever reached the fallback path. The wiring passed a prompt only when was configured, so the preferred SDK path silently used the published package default. The wiring now always supplies one and the hook forwards it on both paths. The register is rewritten around what the cards cannot show: past-tense git-commit-subject, leading with the distinctive noun, outcome over attempt, tool names and counts and arguments explicitly forbidden. The batch entries are labeled as reference material so the model stops transcribing them. Claiming a slot no longer emits. The slot still reserves its index so streamed parts never collide, but with nothing to say there is nothing to render: until a description exists the block looks exactly as it does without the feature. * 🧹 fix: Drop the Localize Hook Left Unused by the Counts Removal * ✅ test: Add Activity-Label e2e Coverage with a Recording Label Server Activity labels are the one model call a mock run does not already fake: fake-model.js swaps the GRAPH model via overrideTestModel, while run.generateActivityLabel() calls the endpoint resolved client options over HTTP. The custom endpoints already point baseURL at 127.0.0.1:8889, so serving that port exercises the real path with no production seam. fake-label-server.js answers it in both JSON and SSE form, records each prompt, and can inject blank/error responses. Recording is what lets the spec assert the CONTRACT rather than the rendering: that this repo register and the tool OUTPUTS actually reach the model. That is the bug class that produced unusable labels before, and rendered text looks identical whether or not the instruction arrived. Labels get a dedicated endpoint (Mock Provider E). A labeled block auto-collapses even at one tool call, which hides the tool cards other specs assert on -- enabling this on a shared endpoint broke steering.spec.ts. Provider D is the unlabeled control. Request-count assertions are scoped to a per-test token: a 5xx label response is retried by the provider client, and a retry can land after the next test has reset the server. * 🩹 fix: Address Review Findings on Activity-Label Indexing and Pricing Replay index (P1). Reserving the slot only in server memory left no event for it, so a cross-instance replay rebuilt content as [tool, hole, later], compacted the hole away, and the fill for the reserved index landed on the following part and overwrote it. The claim now publishes the empty, pending part so the index is real for every consumer, and fill publishes even when generation returned nothing so the client cannot stay pending. It stays invisible: an empty label still DELIMITS its batch in groupSequentialToolCalls but is not attached as the header, so grouping does not re-shuffle when the text lands and the block renders exactly as it does with the feature off. Edited-response index (P1). Edit-and-resubmit replays the kept prefix and the server indexes only new content, so run steps offset by that prefix. Labels are claimed in the same space and now take the identical shift; without it a label could land inside the prefix and overwrite it. Redis flag. deserializeJob never read activityLabels back, so every Redis reload left it undefined and resume skipped label gap reconciliation. Executing agent. RunActivityLabelOptions.agentId selects the executing agent tracing metadata AND its tool-output redaction policy; omitting it let a handoff be redacted under the default agent configuration. Label pricing. An undefined endpointTokenConfig is meaningful for a built-in label endpoint (priced from the shared table), so the nullish fallback billed those labels at a custom primary rates. Inherit only when the label runs on the agent own endpoint. HITL usage sequence. runId is the response message id and the counter was instance-local, so a resume restarted at -1 and the client runId:seq deduper discarded the post-approval label usage. Seeded past the labels already on the response. Also distinguishes "cannot serve" (undefined) from "no label" (null) in the SDK bridge, so a missing run falls back to the direct call instead of filling the slot empty. Version gating already happens at wiring time via the sdkCapable prototype probe. * 🩹 fix: Keep Unfilled Activity Labels Invisible and Unmask Endpoint Settings Follow-up review round. Publishing the reservation on every batch made two latent rendering paths reachable on every run, and both are fixed here. Empty labels no longer change grouping. The previous pass still formed a tool-group for a textless label, which wrapped even a single tool call and pulled THINK parts inside it — and since a reservation is published the moment each batch ends, that applied during every generation and permanently after a blank or failed fill. An empty label now flushes the legacy way instead: it still delimits its batch, but the block re-splits exactly as it renders with the feature off. Parallel lanes no longer show a blank line. Lanes render raw parts, so an unfilled label had nothing to draw; empty ones are dropped. Making labels act as collapsible headers inside lanes is still a separate gap. Edited responses no longer offset on resume. The sync replaces initialResponse.content with the server's aggregatedContent, which already contains the kept prefix AND everything generated since — so its length is not the prefix length, and indices reconciled from that snapshot are already absolute. Offsetting again pushed the label past its slot onto a later part. The shift now applies only to a fresh edited submission. Activity settings resolve field by field. Selecting one config object whole meant any endpoints.all block — even one carrying nothing but headers — shadowed the named or custom endpoint and silently disabled activity labels everywhere. Global still wins per field. Adds groupToolCalls coverage for the invisible-while-empty contract, which is the part most likely to regress: it is normal state on every run, not an edge case. * 🔒 fix: Scope Detached Label Writes to Their Generation Epoch Epoch scoping (P1). Label generation is detached and can outlive the generation that started it. emitChunk only proves that SOME runtime is current, not that the caller belongs to it, so an aborted generation's fill(null) -- and its usage event -- could be attributed to whichever generation replaced it, landing an index from the abandoned response on top of the new one. Because an empty label renders nothing, that overwrote content silently. emitChunk now takes an optional jobCreatedAt and drops the event when the runtime epoch differs, mirroring the existing setGraph/setContentParts convention, and both label emitters pass it. An abort now CLOSES the label scope instead of only cancelling the call: the rejected generation still runs its catch and calls fill(null), which would otherwise emit into a stream the next generation may already own. Edited-response indexing (P1). The previous pass skipped the prefix offset on resume, which was the wrong half of the problem: a sync replaces initialResponse.content with the server's aggregatedContent, which is completion-local, so after a reconnect its length is not the kept-prefix length and the offset is wrong -- but it is wrong for run steps in exactly the same way. Tool cards and the label that heads them must share one index space; a label shifting differently from its tools lands on another part. The label path now uses the identical expression as useStepHandler, with no resume special-case. Correcting the post-resume prefix length belongs in calculateContentIndex, where it fixes both at once. titleModel masking. The activity settings were made per-field last pass, but the titleModel fallback a few lines below still selected an entire config object, so a partial endpoints.all (for example one carrying only headers) hid a named endpoint's titleModel and quietly fell the label back to the main agent model. Both now read through one shared per-field helper. Resume reconciliation no longer depends solely on markActivityLabels, which is best-effort yet had come to gate correctness: a lost flag write silently dropped a label. The snapshot is consulted as a fallback. The exported host type for generateLabel now admits undefined, which is the documented "cannot serve, fall back to the direct call" signal the hook keys on -- distinct from null, meaning it ran and produced nothing. * 🧷 fix: Keep Group Identity Stable and Memoize Label Endpoint Resolution Group remount. Tool-group identity was keyed on the first part in the block. An activity label absorbs the block's leading THINK part the moment its text lands, so the key flipped from tool:<id> to fallback:<scope>:<idx> mid-run, remounting the group and discarding whatever the user had expanded. The key now scans for the first tool call, which does not move when the block re-forms. Label endpoint resolution is memoized per response. It reads provider config and can hit the database for user keys, yet nothing it depends on changes between batches of one run — and it ran twice per batch, once for generation and once for usage accounting. The promise is cached rather than the value so concurrent batches share a single in-flight resolution, and a rejection is evicted so one transient credential failure cannot disable labels for the rest of the response. * 🎯 fix: Offset Edited Resubmissions by a Prefix Length That Survives Resume The server indexes only NEW content for an edited resubmission, so the client offsets incoming indices by the prefix it retained. That prefix was read as initialResponse.content.length, which is correct only until a resume: the sync replaces that array with the server's completion-local snapshot, whose length is unrelated to the prefix. After a reconnect every offset was therefore wrong -- run steps and activity labels alike -- and could write over content the edit kept. For a label the symptom is worse than a bad position: the fill misses its own reservation, so the pending placeholder is never resolved. The prefix length is now captured when the submission is built, while initialResponse.content still IS the retained prefix, and carried on the submission as editPrefixLength. calculateContentIndex takes that length instead of deriving it from an array that a resume may have replaced, so run steps and labels share one index space by construction rather than by both happening to read the same field. Note the prefix is the FULL original content with the edited part substituted in place (useChatFunctions clones latestMessage.content and mutates one entry) -- it is not a slice, so the length cannot be inferred from editedContent.index. Group identity no longer changes when a label fills. Tool-group keys were derived from the first part in the block; an activity label absorbs the leading THINK part when its text lands, flipping the key mid-run and remounting the group, which discarded the user's expansion state. The key now scans for the first tool call, which does not move. Label endpoint resolution is memoized per response. It reads provider config and can hit the database for user keys, yet ran twice per batch -- once to generate, once for usage accounting -- while nothing it depends on changes within a run. The promise is cached so concurrent batches share one in-flight resolution, and rejections are evicted so a transient credential failure cannot disable labels for the rest of the response. The resume gap passes for steers and activity labels now share a single lazy content read instead of each issuing its own. The label pass stays gated on the run flag with a snapshot fallback: reconciling unconditionally would also close the residual first-label window, but it would bill a read to every resume of every run, including deployments with the feature off -- which the steer pass deliberately avoids. That residual requires a lost flag write, which shares fate with the content writes the labels live in. * 💵 fix: Bill Cross-Endpoint Labels at Their Own Rates recordCollectedUsage never accepted an endpointTokenConfig, so the value the activity-label caller passed was dropped and the balance transaction was written at the primary agent's rates. Only the UI cost honored the label endpoint, so a custom primary pointing activityEndpoint at another endpoint showed one price and charged another. The parameter is now accepted, and an explicit config wins outright over per-agent resolution: that map is keyed by AGENT, so it cannot describe usage that ran on a different endpoint. Group identity is stable for id-less tool calls too. The previous pass anchored the key to the first tool call ID; where a supported tool call carries no id the fallback still used the block's first part index, which shifts when a filled label absorbs the leading THINK part. The fallback now anchors to the first TOOL entry's index, so only a block containing no tool call at all keys off parts[0]. markActivityLabels is retried rather than fire-and-forget. It gates resume gap reconciliation and is a SEPARATE write from the durable label append, so a single lost write silently drops a label the content itself recorded. The earlier "shared fate with content writes" reasoning was wrong. One retry at run setup costs nothing and removes the only realistic way the gate goes stale, without billing a content read to every resume. * 🧮 fix: Stop Offsetting Once SYNC Drops the Edited Prefix The edit offset was applied unconditionally, but whether it is correct depends on which branch SYNC took. SYNC either preserves the content already loaded for the response -- which still contains the retained prefix, so the offset is required -- or replaces it with the server's aggregatedContent, which is completion-local and indexed from zero, after which any offset writes past the end of a now shorter array. That is why the two previous attempts each fixed half of it: skipping the offset on resume was right for the replace branch, applying it unconditionally was right for the preserve branch, and neither holds on its own. The offset now tracks the actual state of the rendered content. For an activity label the replace branch was worse than a bad position: the fill landed past its own reservation, so the pending placeholder was never resolved and the block kept its generic header for the rest of the run. Applied to run steps as well, not just labels. useStepHandler reads the prefix from the same submission and had the same unconditional offset, so after a mid-session resume of an edited response tool cards were misplaced too. Normalizing at the dispatch boundary keeps both in ONE index space by construction: a label that shifted differently from the tools it heads would land on another part. Note the reload path was already coherent -- useResumeOnLoad rebuilds the submission without editedContent or editPrefixLength, giving no offset against server-supplied content -- so only the mid-session SYNC path was inconsistent. * 🧾 fix: Keep Label Accounting Out of the Primary Usage Slot Label usage no longer owns getStreamUsage(). recordCollectedUsage assigned its result to this.usage unconditionally, so when the primary provider reported no usage metadata but the label provider did, BaseClient took the label's output tokens as the assistant response's authoritative count. The later primary call returns early on an empty collectedUsage and never replaced it, so the wrong value stood, the text-based fallback was skipped, and the real generation went unbilled. Secondary usage is still billed but no longer writes that slot. Cross-endpoint pricing keys off an explicit discriminator rather than the presence of a value. A built-in label endpoint prices from the shared table, so an undefined endpointTokenConfig is its MEANINGFUL value -- reading that as "no override" fell back to the primary's custom rates and restored the exact mismatch the previous pass set out to fix. The caller already knows whether the label ran elsewhere and now says so. markActivityLabels rejects on failure instead of swallowing it. The flag gates resume gap reconciliation and the caller retries it, but the internal catch resolved successfully and made that retry unreachable -- so the two changes cancelled out and a transient write failure still left the flag absent. Late label accounting is suppressed with the same gate as the late fill. A straggler that outlived the settle timeout still ran its finally block, so it charged the balance and appended to usageEmitSink after the response had passed its usage flush and metadata snapshot: a cost the user pays but is never shown. The cleared-prefix state is scoped to one generation. It was set on a resume SYNC that replaced the response and then never reset, so a later edited resubmission in the same mounted hook dispatched run steps and labels with no offset against content that still held its retained prefix. Reconnects pass isResume and keep the state; a new generation clears it. * 🔑 fix: Key Prefix State to the Stream and Honor current_model for Labels The cleared-prefix reset keyed on isResume, which skips exactly the case it was added for: a submission whose POST succeeded server-side but lost its response is retried, comes back resumed: true, and subscribes in resume mode even though it is a NEW generation. A previous generation's cleared state then survived into it, and incoming run steps and labels applied no offset against content that still held its retained prefix. The state is now keyed to the stream id, which changes with the generation and stays put across reconnects of one. activityModel now honors current_model. The options are documented as title-shaped and the titleModel fallback already excludes the sentinel, but the higher-precedence activity override passed the literal string through to getOptions and the provider, so an endpoint following that convention failed every label instead of using the agent model. * 🎯 fix: Key Prefix State to the Generation and Resolve the Run Model The cleared-prefix state was keyed to the stream id, which never changes within a conversation: request.js sets streamId = conversationId, so once a reconnect cleared the state every later edited resubmission in that conversation dispatched run steps and labels with no offset and could overwrite the prefix it retained. It is now keyed to the response message id, the only per-generation identity available here -- minted per submission and carried through a resume unchanged. That is the third identity tried for this state. isResume missed the deduplicated-retry path (a lost response returns resumed: true for a new generation); the stream id is conversation-scoped. The response id is the boundary that actually matches a generation. current_model labels now resolve the model the run is really using. initializeAgent merges the request's endpointOption override into model_parameters and the run gives it precedence, so preferring the saved agent.model could send labels to a different, potentially unavailable or more expensive model than the conversation is on. * 🆔 fix: Key Prefix State to the Submission and Keep the Origin Title Model Editing an assistant response reuses that response's messageId as editedMessageId, and useChatFunctions carries it onto initialResponse.messageId -- so re-editing the same response produced two generations with the same key and the cleared-prefix state survived between them, leaving run steps and labels with no offset against content the edit retained. Keyed now to clientRequestId, the per-submission uuid, which is minted fresh per edit attempt and forwarded unchanged on retries. That is the fourth key this state has had, and each earlier one failed at a real boundary: isResume missed the deduplicated-retry path, the stream id is the conversation id, and the response message id is reused across edits of one response. clientRequestId is the identity that actually means "this submission". The titleModel fallback is read from the ORIGINATING endpoint again, matching how titleConvo captures its config before switching credentials. Reading it after an activityEndpoint switch meant an OpenAI endpoint configured with titleModel claude-haiku and activityEndpoint anthropic fell through to the OpenAI run model and sent that name to Anthropic, failing every label. The destination endpoint supplies credentials, not the model choice. * 🧷 fix: Close the Remaining Edit, Epoch, and Scope Gaps for Labels SYNC clears the edit prefix on the new-row branch too. When a resumed edited submission cannot match an existing assistant row, that branch builds the response straight from the server's completion-local aggregatedContent, so it holds no retained prefix -- but the reset lived only in the matched branch, leaving later steps and labels adding an offset to indices that were already absolute. Label usage is keyed per GENERATION. Editing one assistant response reuses its responseMessageId while each fresh generation restarts activityLabelUsageSeq, so a second edit re-emitted the same runId:seq and the client discarded the newer usage while its balance transaction was still written. The key now carries jobCreatedAt, the run's own epoch: stable across reconnects and HITL resumes, distinct between generations. The scope is revalidated at commit time. Checking once before the await let a scope that closed mid-flight still charge the balance after finalization, while the matching fill saw the closed scope and dropped the label -- billed but never surfaced, the exact outcome the guard exists to prevent. The titleModel fallback no longer reaches the destination endpoint. With activityEndpoint set and no titleModel on the originating endpoint, it picked up the destination's, so changing only the credential target silently changed the model and its cost. Precedence is activityModel, then the originating endpoint's titleModel, then the run model; the destination supplies credentials only. * ✂️ refactor: Confine the Edit-Prefix Offset to Activity Labels useStepHandler is now byte-identical to dev again. The resume-aware prefix offset was applied there too, which was more correct in principle -- the post-resume prefix length is genuinely wrong for run steps as well -- but it changed index math that EVERY run step flows through, for every user, including everyone who never enables activityLabel. That shared correction needed five revisions in two days (isResume, the stream id, the response message id, clientRequestId, and the SYNC new-row branch), each passing the full suite and each failing at a boundary only review found. Carrying it inside an opt-in feature put every user behind logic with that track record. It belongs in its own change, with tests that construct the edit-plus-resume states none of the current suites reach. The offset now applies only where the label handler places its part, so this PR cannot alter rendering for anyone with the feature off. The known consequence is recorded in the description: with activity labels ENABLED, an edited response that reconnects mid-generation can place its label and its tool cards in different index spaces. That is a bug for opt-in users rather than a regression for everyone, and it disappears once the shared fix lands. submission.editPrefixLength stays: the label path still needs a prefix length that survives a SYNC replacing initialResponse.content. * 🧾 fix: Commit Labels Before Billing and Keep Blank Slots Invisible Round-nine review (all P2, feature-scoped): - Billing ordering (client.js:409, runtime.ts): usage accounting ran BEFORE the slot commit on both generation paths, so the settlement deadline could expire during the balance write — charged, then the fill dropped as out-of-scope: billed, never shown. `slot.fill` now resolves a commit flag, generators register their accounting via `deferUsage`, and the hook runs it only after a committed fill. - Scope gates (client.js:757): the direct-fallback `collect` omitted `scopeOpen`; both paths now gate on the OWNING wiring's scope, so a pre-pause straggler cannot bill because the resumed generation's scope is still open. - Blank-label grouping (groupToolCalls.ts:81): a blank slot forced a flush, splitting adjacent single-call batches into standalone cards where the feature-off path merges them. Blank labels now only mark the claim boundary — structurally invisible, while a later filled label still cannot claim an earlier batch. - Stale fill indices (wiring.ts:301): the skill-card unshift and the hide-sequential filter reshape contentParts before the finalization settle, so an in-flight fill emitted its claim-time index against a shifted array. Both completion paths now settle label fills before any post-run content reshaping (the finally settle stays as the error-path net; the second call sees an empty pending list). - Bounded serialization (runtime.ts:238): `JSON.stringify` fully materialized unbounded tool results to keep 200/600 chars per entry. A budget-bounded serializer stops at the limit (which also bounds cyclic values) and preserves the exact truncate-with-ellipsis output. Tests: fill/bill ordering + suppression on dropped fills (runtime.spec), blank-slot merging and claim boundaries (groupToolCalls.test), bounded serialization equivalence and giant-output truncation (runtime.spec). * 🧮 fix: Keep Deferred Label Billing Inside the Settle Window Self-review follow-up to the billing reorder: deferring usage until after the commit moved it PAST the fill's resolution, so a settle keyed on fills alone could let finalization flush the usage sink and snapshot metadata while the label's billing was still in flight — the usage row would silently miss the message rollup even on the happy path. The hook now reports its whole detached task (generate → fill → deferred usage) via a `trackTask` option, wired to the same settle tracker as the fills, so finalization waits for billing exactly as it did when accounting preceded the fill. The task never rejects. Pinned in runtime.spec: the tracked task resolves only after usage collection. * 🧰 fix: Harden Label Resolution, Output Bounds, and Cache Billing Round-ten review (all P2, feature-scoped); the sixth finding is the documented edited+reconnect index-space limitation, answered on-thread as deliberately out of scope for this PR. - Rejected-LLM memoization (runtime.ts): the hook cached a rejected `resolveLLM()` promise permanently, failing every later batch and silently defeating the host resolver's own rejected-cache eviction. The memo now evicts on rejection so the next batch retries. - `current_model` precedence (host.ts): an explicit `activityModel: current_model` resolved to `undefined` and then lost to a configured `titleModel`. The sentinel now resolves straight to the run model; the title fallback applies only when `activityModel` is absent. - Output bounds (runtime.ts): label text was persisted verbatim; a model ignoring the 4–9-word instruction (or steered by injection in untrusted tool output) could emit thousands of tokens duplicated through SSE, the chunk log, persistence, and the UI. `normalizeLabelOutput` keeps the first non-empty line, collapses whitespace, and hard-caps at 200 chars on both generation paths. - Cache-token billing (host.ts, client.js): the usage mapper dropped cache fields, vanishing Anthropic cache tokens from billing and charging OpenAI cache reads at the full input rate. The mapper now normalizes Anthropic/OpenAI/LangChain cache shapes into `input_token_details`, and the emit + cost path carries them with the label endpoint's `provider` (additive-provider adjustment). - Usage-type union (runs.ts): `TTokenUsageEvent.usage_type` now includes the emitted `activity-label` literal; the lone consumer keys on `usage_type != null`, so this is type-level completion. Tests: sentinel/title/explicit model precedence and all three cache shapes (host.spec), transient-resolution retry and output normalization with truncation (runtime.spec), the new usage literal (runs.spec). * 🪗 fix: Let Settled Labels Collapse Void Tools and Keep the Tail Cursor Round-eleven review (all P2, client-side). Two fixed; the other two findings restate documented Known limitations (edited+reconnect run-step index space; parallel-lane collapsible headers), answered on-thread. - Void-tool auto-collapse (ToolCallGroup.tsx): `allCompleted` keyed solely on output truthiness, so a tool that legitimately returns an empty string kept its labeled group expanded forever. A settled, filled label is itself a completion proof — the PostToolBatch claim only happens after every output in the batch returned — so it now satisfies `allCompleted`; pending labels keep the group live. - Trailing-reservation cursor (ContentParts.tsx): a blank label reservation at the content tail renders nothing but still counted as the last part, stripping the streaming cursor and last-item affordances from the last VISIBLE part until the next delta. `lastContentIdx` now walks back past empty label slots. Tests: labeled void-tool group auto-collapses, pending-label group stays expanded (ToolCallGroup.test). * 💳 fix: Price Label Cache Correctly, Honor endpoints.agents, Cancel Every Retry Round-twelve review: four fixed here; the remaining P1 (move the client.js bridge into packages/api) is an architecture call answered on-thread for the maintainer. - Provider on billed entries (client.js, P1): round ten added cache details to label usage entries but not `provider`, and `splitUsage` treats an unknown provider as additive — re-adding cache_read and cache_creation on top of an input count that already contains them, double-charging Anthropic/OpenAI cached label calls while the streamed cost (which carried the provider) disagreed. Every mapped entry now carries the label endpoint's provider. - endpoints.agents honored (host.ts, client.js): `initializeAgent` rewrites `agent.endpoint` to the backing provider, so activity settings under the PUBLIC `agents` endpoint — valid config, inherited by `agentsEndpointSchema` — were silently ignored. Field resolution is now `all` > public endpoint > backing provider/custom, applied to both the enable gate and the model/titleModel resolution. - E2E_LABEL_PORT reaches the YAML (playwright.config.mock.ts): an overridden port moved the fake label server and its health check but not the generated config's hard-coded 8889 baseURLs, so readiness passed while every label request targeted the wrong port. The override is now substituted into the generated copy. - Every retry frame cancelled (useResumableSSE.ts): concurrent label retry chains (reservation + fill per slot) overwrote one rAF handle, so cleanup cancelled only the newest chain; the rest ran up to 120 frames past unmount and could apply a stale label to a replacement generation reusing the same response id. Outstanding frame ids now live in a Set that cleanup drains. Tests: public-endpoint gate/precedence/all-above-public (host.spec). * 🖱️ fix: Keep the Last-Part Cursor in Parallel Lanes Too Round-thirteen review (single P2): `ParallelContentRenderer` computed `lastContentIdx` from the unfiltered array, so a trailing blank label reservation — filtered out of every lane — left NO rendered part carrying the last-part cursor and running-subagent affordances until the label filled. The sequential renderer's walk-back is extracted into a shared `lastVisibleContentIdx` helper (utils/activityLabels) used by both `ContentParts` and `ParallelContentRenderer`, so the two index spaces cannot drift again. Behavior pinned in activityLabels.spec: trailing blank skipped, consecutive blanks skipped, filled label counts, label-free content unchanged. * 🧹 chore: Alias the Retry-Frame Set for the Effect Cleanup Lint Rule * 📏 fix: Let activityCharLimit Reach Tool Inputs Round-fifteen review: `activityCharLimit` is documented as the per-entry truncation for tool input AND output, but `buildPrompt` hard-coded inputs at 200 characters — so raising the setting could never surface a distinguishing path, query, or operation that appears past the first 200 characters of a long argument. Inputs now truncate at the configured limit alongside outputs; the 200-char constant remains only for the intent line (renamed INTENT_CHAR_LIMIT to match). Config fidelity pinned in runtime.spec: a 400-char argument survives a 450 limit and truncates under a 50 limit. The round's other finding is the fifth restatement of the documented edited+reconnect index-space limitation, answered on-thread with the prior four cross-references. * 🤝 fix: No Labels for Pure Handoff Batches Round-sixteen review: a PostToolBatch containing only `transfer_to_*` calls claimed a label slot, but transfer parts are never groupable — the client flushed the handoff card standalone and the label orphaned into a stray line after it, restating what the card already says. Two-sided fix: - Hook (runtime.ts): a batch whose every entry is a transfer call claims nothing — no slot, no model call, no `maxPerRun` consumption. Mixed batches still label (the header describes the real work). - Renderer (groupToolCalls.ts): an orphan label whose `tool_call_ids` are all transfer calls is dropped instead of rendered standalone, covering content persisted before the hook-side skip. The round's two P1s are repeats answered on-thread: the packages/api extraction (maintainer-decided follow-up, recorded in the description) and the sixth restatement of the edited+reconnect index limitation. Tests: transfer-only batch claims nothing, mixed batch still claims (runtime.spec); transfer-only orphan label dropped, real-batch orphan label still renders (groupToolCalls.test). * 🎛️ fix: Sanitize Label Client Options and Bound the Batch Prompt Round-seventeen review: two fixed; the other two findings repeat the maintainer-decided packages/api extraction (follow-up) and the edited+reconnect index limitation (seventh instance), answered on-thread. - Primary-option strip (host.ts): the label client copied the resolved `llmConfig` wholesale, so an endpoint whose defaults enable extended thinking or carry model-specific output caps forwarded them to the (often cheaper) label model — unsupported options failed every label, and supported thinking spent real tokens and the settlement window on a 4–9 word header. The copy now strips `omitTitleOptions` keys and the `modelKwargs` output caps exactly like the title path, restoring the Anthropic `clientOptions` carrier by reference so proxy `defaultHeaders` still reach label requests. - Batch prompt budget (runtime.ts): per-entry truncation left the batch dimension unbounded — hundreds of parallel calls could build a prompt past the fast model's window. The entries section now has a total budget (8k chars, scaling with `activityCharLimit` so a raised limit still fits several entries); entries past it are skipped without paying their serialization cost, and the list notes how many were omitted. The first entry always renders in full. Tests: option strip with header-carrier survival (host.spec); giant batch bounded with omission marker, small batch untouched (runtime.spec). * 🛡️ fix: Keep SSRF Guards on Label Calls, Skip Mixed Handoff Batches Round-eighteen review: four fixed; the fifth repeats the maintainer-decided packages/api extraction (eighth instance), answered on-thread. - SSRF-safe carrier (host.ts, P1): the sanitize step restored the Anthropic `clientOptions` carrier only when `defaultHeaders` existed, but for user-provided base URLs `getLLMConfig` stores the guarded Undici dispatcher and `redirect: 'error'` there — dropping it reopened DNS-rebinding/redirect paths on label calls to user-controlled URLs. The carrier (client CONSTRUCTION options, not generation params) is now restored whenever present, same reference. - Primary maxTokens (host.ts): top-level `maxTokens` is not in `omitTitleOptions` and survived the strip; the title path deletes it explicitly, and a cap sized for the primary model can be rejected by the substitute. Deleted on the copy. - Bounded keys (runtime.ts): the object branch materialized every key via `Object.keys` and quoted oversized keys in full before the budget check. Enumeration is now lazy (`for..in` + own-property guard) and keys slice to the budget before quoting, like string values. - Mixed handoff batches (runtime.ts, groupToolCalls.ts): the client flushes the block at the transfer card, so a mixed batch's label orphaned exactly like a pure one. The hook now skips ANY batch containing a transfer call, and the renderer drops orphan labels covering one (legacy content). Tests: carrier survival without headers by same reference, maxTokens strip (host.spec); mixed batch claims nothing (runtime.spec); mixed orphan dropped, real-batch orphan kept (groupToolCalls.test). * 🧢 fix: Cap Label Generation, Order the Flag Persist, Detach Settled Listeners Round-nineteen review: three fixed; the fourth is the ninth instance of the edited+reconnect index limitation, answered on-thread. - Generation cap (host.ts): stripping the primary output caps left label calls with NO cap at all — `normalizeLabelOutput` bounds what persists, not what the provider generates and bills, so a model ignoring the 4–9-word instruction (or steered by injected tool output) could emit its provider-default output per batch. The sanitize step now installs a 256-token label cap (per provider family: `maxOutputTokens` for Google-style wrappers, `maxTokens` otherwise), after the filter so the omit set cannot remove it. - Flag-persist ordering (client.js): the `markActivityLabels` write was fire-and-forget, so an immediate cross-replica reconnect could read the job between the write and the first claim, see neither flag nor snapshot label, and skip gap reconciliation. Label emission now awaits the (settled-on-failure) persist chain, making "a label event exists" imply "the flag is durable" — the race window is gone; only the documented double-write-failure residual remains. - Listener detach (client.js): each HITL approval cycle's wiring adds a `once` abort listener to the shared job signal that only an actual abort removes; settled segments now detach theirs in `settleActivityLabels`, so long multi-approval runs cannot accumulate dead closures toward the listener-limit warning. Tests: the primary cap is REPLACED by the 256-token label cap (host.spec). * 🎯 fix: Route the Label Cap Per Model Family Round-twenty review: the 256-token label cap set maxTokens unconditionally, but GPT-5+ rejects max_tokens (the OpenAI builder routes its cap into modelKwargs.max_completion_tokens / max_output_tokens) and o-series models reject it with no stable kwargs alternative — every label on those models would have failed. The cap now mirrors the builder: modelKwargs for GPT-5+ (responses-API aware), no cap for o-series (title parity; the 200-char persistence bound still applies), maxOutputTokens for Google, maxTokens otherwise. Pinned in host.spec for both reasoning families. The round's other finding is the tenth instance of the documented edited+reconnect index limitation, answered on-thread. * ⏱️ fix: Persist the Label Flag at Run Start, Not on the Emit Path Round-twenty-one review: two fixed; the other three repeat the maintainer-decided packages/api extraction, the edited+reconnect index limitation, and the parallel-lane header limitation — all answered on-thread with their standing decisions. - Flag ordering, corrected (client.js): sequencing label emission behind the flag persist (previous round) delayed the claim-time reservation while the shared index offset had ALREADY shifted subsequent SDK chunks — reopening the cross-instance hole-compaction overwrite the reservation emit exists to prevent. The reservation emits immediately again; instead, run start (processStream and resume alike) awaits the settled-on-failure persist chain, so the flag is durable before any batch can claim a label. Same guarantee, zero latency on the emit path. - Tail-label cursor (ContentParts.tsx): a filled label at the content tail is consumed into the group header rather than listed in `group.parts`, so the `isLast` check missed it and nothing held the streaming cursor until the next delta. The check now includes `labelPart.idx`. * 🔌 fix: Detach Label Abort Listeners Even Without Claims A segment with labels enabled can end without a single claim (text-only, or handoff batches, which skip labels); the early return in settleActivityLabels skipped the detach added for HITL listener accumulation. The detach now runs on both paths. * ⚖️ fix: Make the Commit Flag the Sole Billing Authority Round-twenty-three review: a committed fill racing a late scope close (user abort or settle timeout during the durable emit) stayed visible — the part is mutated and persisted before the close — yet the deferred accounting's scope gates then skipped the charge: a completed provider call escaping both the label charge and the primary abort accounting. The scope gates on the deferred-usage path are removed; the hook's commit flag is now the single billing authority in BOTH directions. A dropped fill never reaches the accounting callback (billed-never-shown stays impossible), and a committed fill bills regardless of when its scope closed (shown-never-billed now impossible too). The dead `scopeOpen` payload threading is removed with it; the `recordActivityLabelUsage` parameter survives, defaulting open, for callers that own no commit signal. The round's other finding is the twelfth instance of the documented edited+reconnect index limitation, answered on-thread. * 🧮 feat: Bill Labels by Estimate When Providers Omit Usage Maintainer decision: follow the title convention rather than leaving label calls unbilled when a provider returns no usage metadata. The hook now passes a LAZY estimate thunk with the deferred accounting on the success path — the EXACT prompt the direct path sent (or the locally built equivalent for the SDK path: same entries, context, instruction, truncation contract, and continuity headers) plus the final normalized label. `recordActivityLabelUsage` invokes it only when no collected entry carries a real token count, counts both texts with the shared o200k_base tokenizer, and feeds the synthesized entry through the SAME pipeline (provider-tagged, streamed event, cost, balance transaction). Real provider usage always wins when present. The failure path passes NO estimate: a throw before a response bills only real collected metadata, never a full phantom prompt. Tests: the estimate thunk carries the exact invoked prompt and final label; the failure path defers with no estimate (runtime.spec). * 💵 fix: Estimate From the Raw Completion, Not the Normalized Label The fallback estimate counted the normalized label (first line, 200-char cap) while the provider generated and would bill the raw output up to the 256-token generation cap — under-recording verbose replies. The estimate thunk now carries the raw pre-normalization text; the persisted label is unchanged. Pinned with a multi-line reply test. * 🧾 fix: Commit Label Text Only After the Durable Emit, Estimate the Real SDK Prompt Round review on the billing work: two fixed; the third is the fourteenth instance of the edited+reconnect index limitation, answered on-thread. - Copy-first fill (wiring.ts): the fill mutated the shared content part BEFORE its durable emit, so a failed emit left the label text on `contentParts` anyway — persistence could save and display a label no client ever received and billing (keyed on the commit flag) never charged. The new state is staged on a copy; the shared part mutates only after the emit succeeds, so content, delivery, and billing move together. - Real SDK prompt for estimates (client.js): the estimate thunk carried this module's locally built prompt, but the SDK path frames entries differently — the estimated input count was for a prompt never sent. Chain-start callbacks (handleLLMStart/handleChatModelStart) now capture the prompt the SDK actually rendered, and the deferred accounting substitutes it into the estimate when capture succeeded, falling back to the local approximation otherwise. |
||
|
|
7447fddfb2
|
🙊 refactor: Clarify Ask Question Schema Errors and Retry Guidance (#14279)
* fix(agents): clarify ask question validation errors * fix(agents): narrow question failure detection * fix(agents): persist question validation failures * fix(agents): track question validation failures |
||
|
|
9bb351ad9c
|
🧭 feat: Mid-Run Steering and Queued Messages for Agent Runs (#14220)
* 🧭 feat: Mid-Run Steering and Queued Messages for Agent Runs Steering: submit a message while a run is generating; the server queues it in the job store (cross-instance) and a run-scoped PostToolBatch hook injects it into graph state at the next tool-batch boundary, records an inline 'steer' content part on the response (replayed as a user message on later turns), and streams on_steer_applied to the client. Queuing: messages composed during a run auto-send as normal follow-up turns after clean completion (one per final event, FIFO); user aborts leave them as chips unless armed by interrupt-and-send. Requires hook injectedMessages support in @librechat/agents (danny-avila/agents#299); hard-gated via a capability probe so older SDKs 501 the steer route instead of draining and dropping messages. * 🧵 fix: Harden Steering Against Finalization Races and Route Guard Gaps Addresses local Codex review findings on the steering feature: - Close-and-drain the steer queue atomically at finalization (final event, abort) so a steer POST racing teardown is rejected instead of 202-ACKed and then silently cleared; the closed flag lives on the job hash and is reset when a replacement job reuses the stream id. - Clear inherited steer queues on createJob — a job replacement must not drain the replaced run's messages. - Keep steers queued across a HITL pause instead of draining them into ephemeral client state: resumeState re-seeds chips on reload and the resumed run injects them at its first tool boundary (steers key TTL now extends to the approval window; on_steers_pending event removed). - Queue the NO_ACTIVE_RUN steer fallback while the final SSE is still settling — a direct send would be dropped by ask()'s in-flight guard. - Reconcile the 202 ACK against on_steer_applied events that beat it over the SSE, so a chip can't be re-minted after its removal event passed. - Allow the per-send Steer override when the default action is queue. - Apply the configured message rate limiters and the PII filter to POST /chat/steer — a steer is model-bound user text. * ✅ ci: Assert Steering Capability Probe Against the Installed SDK CI installs the published @librechat/agents pin (pre-injectedMessages), where isSteeringSupported() is legitimately false — the probe test now asserts it mirrors the installed SDK's capability flag instead of hardcoding the capability-bearing build's value. Verified against both the published 3.2.61 dist and the agents#299 build. * 🛟 fix: Preserve Steer Text Across Run-End, Error, and Abort Races Codex round 2 (4 P2s): - Applied-steer-id set survives run end (capped at 100) and converted ids join it, so a 202 ACK that lands after final/abort drops its chip instead of re-minting a stranded pending one. - Failed runs no longer strand acknowledged chips: both error paths convert local pending chips to queued follow-ups (chip text is client-local), and the server closes the steer queue before emitting the error so a racing steer POST gets 404 fallback instead of a 202 whose payload dies with the job. - sendQueuedNow keys on steer availability, not the default action — send-now on a queued chip is an explicit override for queue-preferring users. - Stop path consumes pendingSteers from the abort HTTP response as a fallback for the SSE final event it may close before processing; conversion is deduped so double delivery is a no-op (shared useSteerConvert hook). * 📎 feat: Carry Attachments Through During-Run Queued Messages Steering stays text-only (SDK injection, inline STEER part, and replay are all text), so a during-run submit with media now queues the whole message as one unit instead of silently stranding the files: - QueuedMessage gains `files`; composer attachments are consumed into the queued item at queue time (steerFromComposer / queueFromComposer / interruptAndSend), fixing the latent hazard where lingering composer files glued onto whatever `ask` vacuumed up next. - Enter-steer with attachments degrades to queue with an explanatory toast; the per-send menu routes through the same composer-aware wrappers. - The drain and sendQueuedNow pass the item's files as `overrideFiles`; media items never steer (send as a normal turn when idle, re-front otherwise). ask() no longer clears composer state for caller-supplied overrideFiles — only regenerate keeps that behavior. - During-run submits hold while uploads are in flight, mirroring the send button's filesLoading gate; queued chips show a paperclip count. * 🎛️ feat: Rework During-Run Chips into Action Rows Full-width rows above the composer (reference-UI parity): each queued message shows a primary Steer/Send-now action, delete, and a "…" menu with Edit message (restores text + attachments into the composer) and a Turn on queueing/steering toggle that flips the Enter default. Steer rows share the layout with status text; failed steers keep retry / edit / queue-convert. The per-send menu gains the same default toggle. Queued file refs now retain filename + bytes so edit-restore rebuilds real composer entries (draft-recovery shape). * 🖇️ feat: Steer With Attachments (Multimodal Mid-Run Injection) Steering now carries media end-to-end instead of degrading to queue: - The steer POST accepts sanitized attachment refs (cap 10; only file_id is trusted — the drain re-fetches owner-scoped and re-derives everything else). SteerQueueItem/TPendingSteer/SteerContentPart carry `files` refs; encoded data is never persisted or queued. - New api/server/services/Files/steering.js decouples attachment building from the request path: encodeSteerContent reuses the exact per-turn pipeline (addFileContextToMessage + processAttachments' single-pass categorize/encode, SDK formatMessage assembly, prependFileContext for extracted text) with zero new encoding code. buildSteerMedia feeds the drain hook's new buildMedia seam (any failure degrades that steer to text-only — words always land); stampSteerPartMedia re-encodes past steer parts per turn with ONE batched owner-scoped fetch and stamps a transient `media` array, replaced immutably so it can never leak into a save. Replay honors resendFiles like regular message media. - The SDK's formatAgentMessages (the formatter agents actually use) gained the steer replay branch on the PR branch; the local formatMessages.js branch now mirrors the media preference. - Client: steerFromComposer consumes composer files into the POST, chips/seeding/conversions carry files everywhere (retry, queue convert, abort/error recovery), queued media items steer for real, and SteerBubble renders the steered attachments inline. * 🧵 fix: Harden Steer Recovery Races and Drain Isolation Codex round 3 (7 fixes): - A 202 ACK landing after the run ended converts straight to a queued follow-up (server queue is gone; no event will ever resolve a pending chip for a finished run). Covers stream errors with in-flight POSTs. - A Stop that lands pre-completion can arrive as a final with unfinished:true and no aborted flag — runEnd now treats it as aborted so queued messages are not auto-sent against the user's Stop. - Leftover-steer conversion merges chronologically by createdAt instead of appending, preserving the order the user composed. - Auto-drained queued messages pass explicit (possibly empty) overrideFiles/overrideQuotes/overrideManualSkills: a drain can no longer vacuum up files, quotes, or skill picks staged in the composer for the user's NEXT message (ask() treats overrideFiles != null as authoritative). - Failed-steer Retry and resume-on-load chip restoration keep the steer's attachments. - The job-replacement guard moved INSIDE the store's atomic drain/close-and-drain (Lua createdAt compare; in-memory equivalent): a stale run's hook or finalization can neither consume, close, nor steal a replacement job's steer queue, and the drain hook drops its separate check-then-drain round trip. * 🧰 refactor: Typed Steer Controller, Single-Query Media Pass, Round-4 Fixes Codex round 4 + efficiency tightening in one pass: - Moved the steer guard ladder (validation, file sanitization via a shared toSteerFileRef picker, ownership/tenant checks, status-guarded enqueue) into packages/api as handleSteerRequest; api/steer.js is now a thin wrapper. Ladder covered against the REAL in-memory job manager in request.spec.ts; the api spec pins only the wrapper contract. - Folded the steer replay stamp into the turn's ONE historical-files query: collectHistoricalFileRefs also gathers steer-part refs, the owner-scoped doc map rides client state, and stampSteerPartMedia consumes it (no second round trip) while encoding parts in parallel. - Stamped steer media now counts against the run budget (existing multimodal counter over the non-text parts, folded into indexTokenCountMap/promptTokens after the stamp). - Steer route runs the PII filter BEFORE moderateText, matching chat.js so blocked sensitive text never reaches the external moderation API. - Interrupt & send survives the abort-response-beats-SSE-final race: stopGenerating writes the run-end signal itself when the one-shot interrupt flag is armed and no signal landed (double-fire safe). - Resume reconciles chips against the server's still-queued list even when EMPTY, clearing chips for steers applied while disconnected. - The local formatter's steer flush preserves non-text assistant parts (array-content AIMessage) instead of folding to text. * 🔒 fix: Replay-Aware Capability Gate and Round-5 Race Closures - isSteeringSupported now requires BOTH halves of the SDK contract: injection (HOOK_INJECTED_MESSAGES_CAPABLE) AND replay (ContentTypes.STEER, shipped in the same SDK commit as the formatAgentMessages steer branch). An SDK that can inject but not replay 501s the steer route — no release window can create steer parts that would leak into provider-facing assistant content. - The local formatter mirrors the SDK's anchor reset: a post-steer tool_call mints a fresh AIMessage instead of attaching to the pre-steer anchor (invalid provider ordering). - Queued-chip send-now and the NO_ACTIVE_RUN fallback pass explicit (possibly empty) overrideFiles so an idle send can't vacuum composer files staged for a different draft. - Redis createJob deletes the stale steer list BEFORE the replacement hash is written as running — a steer 202-accepted against the new job can never be wiped by the reset. - Resumed-turn finalization mirrors the normal path's terminal drain: createdAt-guarded close-and-drain, leftovers ride the resumed final event as pendingSteers instead of being cleared by completeJob. - buildSteerMedia restores composer order over the $in result so multi-attachment steers reach the model in the order the user saw. * ⚛️ fix: Atomic Job Replacement and Boundary-Clean Steering Module Codex round 6 (5 fixed, 1 standing deferral): - createJob resets the steer queue and writes the job hash in ONE same-slot Lua script (JOB_CREATE_LUA): a steer POST can no longer interleave between them on cluster, so a steer accepted against one run can never be drained into another. Redis-validated. - The steering media pipeline moved to packages/api (agents/steering/media.ts) with injected getFiles and a structural client interface — /api keeps zero steering logic; specs ported to the DI seam. - handleSteerRequest checks the job BEFORE the capability gate: a steer racing completion on an unsupported SDK gets 404 (send-now) instead of a 501 queue with no run-end signal left to drain it. - useQueueDrain binds to the active conversation: navigating away between the final SSE and the drain effect leaves the signal unconsumed instead of submitting A's follow-up into B; the drain fires on return. - abortJob closes and drains the steer queue BEFORE the content snapshot, so a drain-hook apply that lands pre-drain is captured inline rather than lost between the snapshot and the terminal drain. * 🚦 fix: Parked Run-End Signals, Interrupt Priority, Settled-Run Fallbacks Codex round 7 (5 fixes): - Run-end signals for a non-active conversation are PARKED per conversation instead of squatting the shared index slot: a later run finishing on the same pane can no longer overwrite them, and the parked drain fires when the user returns. - "Interrupt & send" front-inserts carry a priority flag that outranks createdAt when abort leftovers merge back chronologically — the urgent redirect drains first, not the oldest steer. - STEER_UNSUPPORTED/RUN_PAUSED/QUEUE_FULL rejections landing after the run settled mirror the NO_ACTIVE_RUN fallback and send immediately (queueing would strand the text with no run-end signal left); on the pinned SDK this is the common Enter-near-run-end path. - A failed abort (e.g. 404 when the run completed first) still signals the interrupt drain, so the queued interrupt message can't strand and the armed flag can't leak onto a later run. - Steered-image fallback alt text is localized (com_ui_attached_image). * 📌 chore: Adopt Published @librechat/agents Types Post-Bump dev's pin bump to ^3.2.62 (the release carrying injection + steer replay) landed via merge; the steering runtime now uses the SDK's real InjectedMessage/hook-output types instead of the local structural mirrors that bridged the pre-publish window. The two-half capability probe stays as the defensive gate for mismatched deployments — and the capability spec now exercises its TRUE path against the published package in CI. * 🛅 feat: Park-and-Claim Steer Recovery + Host-View Content Reads Codex round 8 (6 fixed incl. both P1s, 1 push-back): - The long-deferred no-subscriber gap is closed: every terminal drain (final, aborted-final, error, abortJob, resumed finalize) PARKS acknowledged leftovers on the job hash (unrecoveredSteers), and the status route claims them exactly once for inactive jobs — a client that closed/reloaded past the transient final event restores its steers as queued chips within the post-terminal TTL. A replacement run clears the parked copy (a live client started it). - Same-instance content reads are steer-complete: RedisJobStore now caches the HOST content array (WeakRef) via setContentParts and prefers it over the SDK graph cache, whose view never contains host-authored steer parts; the graph fallback splice-INSERTS steer chunks at their recorded host-view indices (the graph array is unshifted, so assignment would overwrite SDK parts). - Replay token accounting now counts prepended file-context text: full stamped content minus the steer body (already counted), so large steered documents hit the budget instead of bypassing pruning. - The queue drain restores an item when ask() refuses without sending (history not yet in cache after navigating back) — text is never silently dropped. - The armed interrupt flag travels WITH a parked run-end signal, so another run on the same pane can neither consume nor clear it. - parseTextParts extracts steer text (search indexing / audio). * 🎛️ refactor: Single Send Slot + In-Thread Steer Messages - Merge the during-run send affordance into the send/stop button slot: with composer text the send button replaces Stop (Enter = default action), hover reveals Steer/Queue/Interrupt rows with shortcuts; drop the separate DuringRunActionsMenu chevron - Add during-run keyboard chords: Cmd/Ctrl+Enter = non-default action, Alt+Enter = interrupt & send (plain-Enter submitters only) - Render steers as standard user messages in the thread: SteerPart (icon + author header + user text presentation) replaces the SteerBubble, and submitted steers appear immediately at the projected injection point via the PendingSteers slot on the streaming message - Keep composer rows only for recoverable states: failed steers (retry/edit/queue) and queued follow-ups * 🩹 fix: Keep the Replacement Submission Alive Across Abort Settlement The aborted run's final SSE event fires before the abort HTTP response resolves, so an armed interrupt & send drains and starts the NEXT submission while the abort POST is still in flight. The response handler's unconditional clearAllSubmissions() then reset the new submission, aborting its stream attach before the subscribe — the follow-up ran and persisted server-side but the live placeholder finalized empty (content appeared only after reload). useAbortCleanup captures the submission before the abort round-trip and both settlement paths (success and 404-catch) clear only when the captured submission is still current; a replacement stays untouched. Plain Stop behavior is unchanged. * 🧭 test: Playwright E2E for Mid-Run Steering and Queuing - Add e2e/specs/mock/steering.spec.ts: steer mid-run (202 + immediate in-thread pending part + real MCP tool boundary + words survive run end), Cmd/Ctrl+Enter queue with auto-send after clean completion, and Alt+Enter interrupt & send with the follow-up streaming into the live view - Add the E2E_STEER_TOOL_REPLY fake-model marker: slow preamble, a real remember_fact MCP tool call (PostToolBatch boundary), then a final turn - Test 1 pins the run-end degradation contract while the SDK's top-level agentId stamping bug blocks live injection; its header documents the assertions to flip once the fixed SDK is pinned * 🧷 fix: Job-Independent Steer Recovery + Expiry and Resume-Gap Parking Codex round 10: the park-and-claim recovery had lifecycle holes. - Move parked steers off the job hash onto their own bounded-TTL store key (JOB_CREATE_LUA resets it; deleteJob leaves it alone): the default completeJob path deletes the job record immediately, and the Redis read path never deserialized the old hash field — recovery previously worked only with STREAM_KEEP_COMPLETED_JOBS on the in-memory store - Carry the owner identity inside the parked payload and authorize the claim against it, so the status route recovers steers on its jobless branch too (the common reload-after-terminal case); a non-owner claim returns nothing and re-parks the payload - Park queued steers on approval expiry: snapshot the frozen queue before the requires_action→aborted CAS (whose terminal cleanup drops the steers key) and park only when the CAS wins - Mirror the terminal drain/park block in resume.js's failure path, which previously let completeJob's backstop clear 202-accepted steers - Close the Redis snapshot→subscribe resume gap: re-peek the queue after attaching and re-surface missed on_steer_applied events from the durable content view (synthesizeAppliedSteerEvents), updating resumeState.pendingSteers to the live queue * 📌 chore: Require @librechat/agents 3.2.63 + Applied-Steer E2E Contract - Bump the @librechat/agents pin to ^3.2.63 in api/ and packages/api/: it scopes the hook agentId marker to subagent child graphs, so the steering drain hook fires at top-level tool-batch boundaries and mid-run injection is active (danny-avila/agents PR 307) - Flip e2e steering test 1 from the documented degradation contract to the applied-steer contract: the optimistic in-thread part transitions to the persisted part at the tool boundary and survives inside the response after run end, with no queued follow-up turn * 🎗️ feat: Steered Messages Join the Message-Nav Ribs Steers are user messages, so they get their own clickable rib on the navigation rail, interleaved at their in-thread position inside the response that absorbed them (one DOM query in document order). SteerPart anchors itself as #steer-<id> with a steer-render marker — both the optimistic pending entry and the persisted part — and the rib carries the user role label with a preview drawn from the steer's text body, skipping the author header. * ❎ feat: Cancel a Queued Steer Before Injection + True User-Message Alignment - Add POST /chat/steer/cancel: removes ONE still-queued steer by id via an atomic list rebuild (Redis Lua preserves order and TTL), authorized against the job owner; removed:false is advisory — the cancel lost its race to the drain or the run end, never an error - Surface an × on the in-thread pending steer (server-acknowledged entries only): optimistic removal, restored if the POST fails since the server would still inject the words - Outdent SteerPart past the response's icon column so steers sit flush with top-level message rows, reading as regular user messages * 🧯 fix: Round-11 Recovery Hardening + Provider-Free Pending Slot - Reconcile the resume steer gap by steerId SETS, not queue length — a steer added in the gap (or an equal-length drain+enqueue swap) now refreshes resumeState.pendingSteers and still synthesizes the missed on_steer_applied events - Make completeJob's terminal backstop park: direct error-path callers without the controllers' close-and-park no longer silently clear 202-accepted steers (createdAt-guarded closeAndDrain + owner park before the terminal write) - Persist the steer part BEFORE media encoding in the drain hook: an abort inside the encode window can no longer lose a file-steer (the part refs come from the enqueue-sanitized item; replay re-encodes per turn unchanged) - Move the parked-claim owner check INSIDE the atomic store claim (substring gate in the Lua / in-memory equivalent): a non-owner probe can no longer transiently delete the recovery payload; the app-side parse stays authoritative - Park queued steers in BOTH stores' own requires_action expiry cleanup, which bypassed the manager-level sweep - Sweep expired parked steers from the in-memory store's periodic cleanup; restore a queued chip when send-now's submit is refused; upsert steer ACKs so an SSE reconnect reseed cannot duplicate chips - Mount the cancel mutation per steer item so the pending slot needs no QueryClient on ordinary streaming renders (fixes the CI failure in ContentParts.integration.test) - Skipped delivery-gated parking (finding 8): transport receiver counts cannot prove browser delivery, and gating the only durable copy on them trades cosmetic chip resurrection for real text loss; the window is already bounded by claim-on-read, createJob reset, and the TTL * 🩺 fix: Annotate PARKED_STEERS_TTL_MS for isolatedDeclarations tsdown's d.ts generation requires explicit types on exported consts with computed initializers; tsc --noEmit does not run that check, so the round-11 export slipped past local verification and broke Build packages (and every downstream CI job that consumes the built dist). * 🛟 fix: Round-12 Terminal-Path Recovery + Durable Steer Events - Park queued steers before the stale-running reap deletes a crashed or hung job in BOTH stores — the one terminal path with no controller finalization; requires_action expiry parking refactored onto the same snapshot/park helpers - Enqueue instead of dropping when a steer fallback send is refused: both the NO_ACTIVE_RUN branch and the settled-run rejection branch now observe sendNow's false return - Recover on the SSE reconnect-404 terminal path: convert local pending steers to queued, claim parked steers via /chat/status, and write a non-completed run-end signal so interrupt flags release without auto-sending an unknown outcome - Fall back to a positive parked-recovery TTL when completedTtl is 0 (SET EX 0 is invalid and silently killed recovery) - Make on_steer_applied durable before publish: emitChunk gains a durable option that awaits the chunk-log append (best-effort) ahead of the transport publish; the default delta path stays fire-and-forget * 🔐 fix: Round-13 Steer Authorization + Trusted File Refs - Resolve client-supplied steer file refs against the DB owner-scoped at enqueue and queue only DB-derived shapes (same filter as the injection fetch, shared via refs.ts); any unresolved id fails loud with 400 — spoofed type/filepath metadata can no longer be persisted into assistant content or rendered in chat/share views - Enforce agent authorization on /chat/steer against the ORIGINATING run's job identity: the chat path's role gate (AGENTS:USE, with the same non-agents-endpoint skip) plus the per-agent ACL check with the capability bypass — revoked access mid-run can no longer inject; cancel stays ownership-only (nothing model-bound) - Mark steered uploads used after a successful enqueue (owner-scoped, best-effort) so the upload-window TTL cannot reap a file the persisted steer part references - Consume the parked recovery copy after live delivery: converting final/abort/error pendingSteers fires one owner-gated claim-on-read, so dismissed chips can no longer resurrect on a later reload * 🎙️ fix: Round-14 Composer-Context Fidelity + TTS and Queue-State Gaps - Keep steer text out of generic assistant text extraction: parseTextParts excludes STEER parts by default with an includeSteer opt-in for the full-record surfaces (Meili indexing, aborted-response persistence) — TTS callers no longer speak the user's own mid-run words - Mark queued uploads used at enqueue time via a minimal owner-scoped POST /files/usage (fail-closed without a user; upload limiters do not apply to a metadata touch), fired once wherever composer files enter the queued state — the upload-window TTL can no longer reap a file waiting out a long run or approval pause - Carry quote chips and manual skill picks on queued items: captured and consumed from the composer at queue/interrupt time exactly like files, threaded through the drain and send-now overrides, and restored by the queued row's Edit message - Key an early-aborted FIRST turn's run-end signal to NEW_CONVO (resolveRunEndTarget) so queued follow-ups stay visible on the restored new-chat composer instead of parking under an optimistic stream id the user never sees again * 🧿 fix: Round-15 Gap Coverage + Consolidated Sweep (Share Leak, Abort Ids, Chip Hygiene) - Run the resume steer-gap check for every still-active job: an empty snapshot no longer skips the re-peek, and synthesis now keys on the FRESH content view so an applied-in-gap steer that was never snapshotted still re-surfaces (over-emission is benign — applied-id dedupe, index-stable parts) - Thread queued context through steer degradation: sendQueuedNow passes the item's quotes/skills into submitSteer, and every fallback (requeue or settled send) restores them instead of dropping to text+files - Stop shared links from leaking steer attachment refs: the share snapshot now walks content — files-excluded shares strip steer-part files entirely; files-included shares sanitize and share-route them like top-level files (copy-on-write, non-steer content by reference) - Seed pending-steer chips unconditionally on load/return so a steer applied while away cannot linger as a stale chip beside its part - Use the abort response's resolved job id: chips/drain-signal land where the user actually is (NEW_CONVO for a new-held first turn, consistent with resolveRunEndTarget) while the parked-copy claim hits the resolved id instead of a no-op /chat/status/new - Open steered documents like normal message files (FilePreviewDialog) - Cap the applied-steer id set on the live path via a shared helper; kept surviving run end deliberately (late-ACK race depends on it) and fixed the atom comment that claimed otherwise * 💡 fix: Un-light Steer Ribs When Their Node Is Replaced Two stacked gaps kept a steer rib lit after scrolling away: the pending→applied swap replaces the DOM node under the same id, which produces no IntersectionObserver exit and — because the entry list dedupes on (id, preview) — no entries change either, so the observer kept watching a detached node; and the rail's mutation filter only reacted to .message-render nodes, so steer-node swaps and removals never triggered a refresh at all. - reconcileObservedElements re-points the observer at replaced nodes from the mutation-driven refresh regardless of entries identity, dropping stale visibility until the fresh node reports (the observer fires its initial intersection immediately, so a truly visible part re-lights within a frame) - The mutation filter now recognizes steer-render nodes alongside message rows * 🪪 fix: Round-16 Recovery Owner Fields + Context Stickiness + Share Labels - Park resumed-run leftovers with the manager facade's metadata owner fields: a bare job.userId is undefined on that shape, which made every parked payload from a resumed HITL run unclaimable - Keep a queued item's quotes/skills sticky through a successful steer ACK: the pending chip carries them (client-only), reseeds preserve them across reconnects, and every terminal conversion — local or server-list, merged by steerId — restores them onto the queued item - Convert resumeState.pendingSteers on the inactive status branch (deduped against unrecoveredSteers) so steers observed in the expired-pause-before-sweeper window convert instead of vanishing until a later reload - Label shared steer parts share-safely via the existing ShareContext: a viewer's own name no longer appears on the sharer's steered messages * ✂️ fix: Carry Steer Context Through the Failed-Chip Edit Action Retry and convert-to-queue already preserve a failed steer's carried quotes/skills; Edit message dropped them on the way back to the composer. It now restores them through the same context path. |
||
|
|
8fcb77fe6f
|
🧵 fix: Preserve Fenced Markdown Artifacts (#14121)
Some checks failed
Docker Dev Branch Images Build / build (Dockerfile, lc-dev, node) (push) Waiting to run
Docker Dev Branch Images Build / build (Dockerfile.multi, lc-dev-api, api-build) (push) Waiting to run
Docker Dev Images Build / build (Dockerfile, librechat-dev, node) (push) Has been cancelled
Docker Dev Images Build / build (Dockerfile.multi, librechat-dev-api, api-build) (push) Has been cancelled
GitNexus Index / index (push) Has been cancelled
Sync Locize Translations & Create Translation PR / Sync Translation Keys with Locize (push) Has been cancelled
Sync Helm Chart Tags / Ignore non-main push (push) Has been cancelled
Sync Helm Chart Tags / Sync chart tags (push) Has been cancelled
GitNexus Index / post-index (push) Has been cancelled
Sync Locize Translations & Create Translation PR / Create Translation PR on Version Published (push) Has been cancelled
* fix: Preserve fenced markdown artifacts * fix: Satisfy artifact CI checks * fix: Handle longer artifact fences in updates |
||
|
|
84329ab0ff
|
fix: use logAxiosError at the RAG file_search/context call sites (#14014)
|
||
|
|
d90567204e
|
🛟 fix: persist Vertex Gemini 3 thoughtSignatures across DB round-trips (#13026)
When a tool round-trip is interrupted between the tool result and the
model's text reply (user aborted, network drop, pod restart, ...) and
LibreChat persists the partial assistant message, the next conversation
turn reconstructs an `AIMessage` from `formatAgentMessages` that has
`tool_calls` populated but no `additional_kwargs.signatures`. Vertex
Gemini 3 rejects the resumed request with 400 because the most recent
historical functionCall has no `thought_signature`.
## Storage shape
Capture as `Record<tool_call_id, signature>` rather than a flat array.
This addresses the codex P1 review:
> When an assistant turn contains multiple sequential tool-call batches,
> this restoration path writes all persisted thoughtSignatures onto only
> the last tool-bearing AIMessage. Vertex/Gemini validates signatures
> for each step in the current tool-calling turn, so earlier
> functionCall steps reconstructed without their signature can still
> fail with 400.
A single agent run can fire multiple `chat_model_end` events when the
loop cycles the LLM with intervening tool results — each cycle owns a
distinct `tool_call_id`. Per-id storage maps each signature back onto
the right reconstructed `AIMessage`, not just the last one.
## Mapping
`additional_kwargs.signatures` is a flat array indexed by *response part*
(text + functionCall interleaved). `tool_calls` is just the function
calls in their original order. Non-empty signatures correspond 1:1 with
tool_calls in order — see `partsToSignatures` in
`@langchain/google-common`. Single-pass walk maps `signatures[i]` (when
non-empty) onto the i-th `tool_call.id`.
## Pipeline
| Stage | File | Change |
|---|---|---|
| Capture | callbacks.js | `ModelEndHandler` accepts `Record<string,string>` map; walks signatures + tool_calls in tandem to record per-id. Gated on the map being provided — non-Vertex flows are no-op (and also no-op even when provided, since they don't emit signatures). |
| Plumbing | initialize.js | Allocate `collectedThoughtSignatures = {}`, share with handler + client. Always allocated; the JSDoc explicitly documents that it stays empty for non-Vertex providers. |
| Surface | client.js | `sendCompletion` returns `metadata.thoughtSignatures` when the map has entries; falls through unchanged when empty. |
| Persist | (existing BaseClient.handleRespCompletion) | Writes `metadata` from `sendCompletion` onto `responseMessage.metadata`. Mongoose `Mixed` — no migration. |
| Restore | formatMessages.js | Track every tool-bearing AIMessage produced from a TMessage. For each, build a position-aligned `additional_kwargs.signatures` array (empty placeholders for tool_calls without a stored sig). Agents' `fixThoughtSignatures` dispatches non-empty entries to functionCall parts in order. |
## Live verification
- **Single-step:** real Vertex `gemini-3.1-flash-lite-preview` resume-after-tool case. With fix ✅ / without ❌ 400.
- **Multi-step (codex case):** real two-step agent loop (list /tmp → echo done). Each step's signature attaches to its own reconstructed AIMessage. With fix ✅ / without ❌ 400.
- **Cross-provider:** Anthropic Claude haiku-4.5 + OpenAI gpt-5-mini accept the persisted/restored shape unchanged.
## Tests
`modelEndHandler.spec.js` (new) — 6 tests:
- maps non-empty signatures onto tool_call_ids in order
- accumulates per-id across multiple `model_end` events (multi-step)
- no-op when `collectedThoughtSignatures` is null
- no-op when `signatures` field missing (non-Vertex)
- no-op when `tool_calls` missing
- preserves existing `collectedUsage` array contract
`formatAgentMessages.spec.js` — 6 new tests:
- restores onto the AIMessage that owns the tool_call
- per-step attachment for multi-step turns (codex review case)
- preserves tool_call ordering when signatures are partial
- no-op when metadata.thoughtSignatures absent
- no-op when assistant has no tool_calls
- no-op when stored ids don't match any current tool_call
37 passing across 3 suites; 15 existing formatAgentMessages tests unchanged.
## Compatibility
- Backward-compatible — restore gated on `metadata.thoughtSignatures` being a populated object; capture gated on the map being provided.
- No schema migration — uses `Message.metadata: Mixed` already in place.
- Cross-provider safe — non-Vertex providers tolerate the field (verified live against Anthropic + OpenAI converters).
- Pairs with [agents#159](https://github.com/danny-avila/agents/pull/159) for full coverage on histories that mix plain-text and toolcall AIMessages.
|
||
|
|
1b79e0b785
|
🧬 chore: Align LibreChat With Agents LangChain Upgrade (#12922)
* 🔧 chore: Update dependencies in package-lock.json and package.json - Bump version of @librechat/agents to 3.1.75-dev.0 in multiple package.json files. - Upgrade various AWS SDK and Smithy dependencies to their latest versions in package-lock.json for improved stability and performance. * 🔧 chore: Update AWS SDK and Smithy dependencies in package-lock.json - Bump version of @aws-sdk/client-bedrock-runtime to 3.1041.0 and update related dependencies for improved performance and stability. - Upgrade various AWS SDK and Smithy packages to their latest versions, ensuring compatibility and enhanced functionality. * chore: Align LibreChat with agents LangChain upgrade - Route LangChain imports through @librechat/agents facade exports - Update @librechat/agents to 3.1.75-dev.1 and remove direct LangChain deps - Normalize nullable agent model params and API key override typing - Update Google thinking config typing for newer LangChain packages - Refresh targeted audit-related dependency overrides * chore: Add Jest types for API specs * test: Fix LangChain upgrade CI specs * test: Exercise agents env facade * fix: Clean up TS preview diagnostics * fix: Address Codex review feedback |
||
|
|
b5c097e5c7
|
⚗️ feat: Agent Context Compaction/Summarization (#12287)
* chore: imports/types
Add summarization config and package-level summarize handler contracts
Register summarize handlers across server controller paths
Port cursor dual-read/dual-write summary support and UI status handling
Selectively merge cursor branch files for BaseClient summary content
block detection (last-summary-wins), dual-write persistence, summary
block unit tests, and on_summarize_status SSE event handling with
started/completed/failed branches.
Co-authored-by: Cursor <cursoragent@cursor.com>
refactor: type safety
feat: add localization for summarization status messages
refactor: optimize summary block detection in BaseClient
Updated the logic for identifying existing summary content blocks to use a reverse loop for improved efficiency. Added a new test case to ensure the last summary content block is updated correctly when multiple summary blocks exist.
chore: add runName to chainOptions in AgentClient
refactor: streamline summarization configuration and handler integration
Removed the deprecated summarizeNotConfigured function and replaced it with a more flexible createSummarizeFn. Updated the summarization handler setup across various controllers to utilize the new function, enhancing error handling and configuration resolution. Improved overall code clarity and maintainability by consolidating summarization logic.
feat(summarization): add staged chunk-and-merge fallback
feat(usage): track summarization usage separately from messages
feat(summarization): resolve prompt from config in runtime
fix(endpoints): use @librechat/api provider config loader
refactor(agents): import getProviderConfig from @librechat/api
chore: code order
feat(app-config): auto-enable summarization when configured
feat: summarization config
refactor(summarization): streamline persist summary handling and enhance configuration validation
Removed the deprecated createDeferredPersistSummary function and integrated a new createPersistSummary function for MongoDB persistence. Updated summarization handlers across various controllers to utilize the new persistence method. Enhanced validation for summarization configuration to ensure provider, model, and prompt are properly set, improving error handling and overall robustness.
refactor(summarization): update event handling and remove legacy summarize handlers
Replaced the deprecated summarization handlers with new event-driven handlers for summarization start and completion across multiple controllers. This change enhances the clarity of the summarization process and improves the integration of summarization events in the application. Additionally, removed unused summarization functions and streamlined the configuration loading process.
refactor(summarization): standardize event names in handlers
Updated event names in the summarization handlers to use constants from GraphEvents for consistency and clarity. This change improves maintainability and reduces the risk of errors related to string literals in event handling.
feat(summarization): enhance usage tracking for summarization events
Added logic to track summarization usage in multiple controllers by checking the current node type. If the node indicates a summarization task, the usage type is set accordingly. This change improves the granularity of usage data collected during summarization processes.
feat(summarization): integrate SummarizationConfig into AppSummarizationConfig type
Enhanced the AppSummarizationConfig type by extending it with the SummarizationConfig type from librechat-data-provider. This change improves type safety and consistency in the summarization configuration structure.
test: add end-to-end tests for summarization functionality
Introduced a comprehensive suite of end-to-end tests for the summarization feature, covering the full LibreChat pipeline from message creation to summarization. This includes a new setup file for environment configuration and a Jest configuration specifically for E2E tests. The tests utilize real API keys and ensure proper integration with the summarization process, enhancing overall test coverage and reliability.
refactor(summarization): include initial summary in formatAgentMessages output
Updated the formatAgentMessages function to return an initial summary alongside messages and index token count map. This change is reflected in multiple controllers and the corresponding tests, enhancing the summarization process by providing additional context for each agent's response.
refactor: move hydrateMissingIndexTokenCounts to tokenMap utility
Extracted the hydrateMissingIndexTokenCounts function from the AgentClient and related tests into a new tokenMap utility file. This change improves code organization and reusability, allowing for better management of token counting logic across the application.
refactor(summarization): standardize step event handling and improve summary rendering
Refactored the step event handling in the useStepHandler and related components to utilize constants for event names, enhancing consistency and maintainability. Additionally, improved the rendering logic in the Summary component to conditionally display the summary text based on its availability, providing a better user experience during the summarization process.
feat(summarization): introduce baseContextTokens and reserveTokensRatio for improved context management
Added baseContextTokens to the InitializedAgent type to calculate the context budget based on agentMaxContextNum and maxOutputTokensNum. Implemented reserveTokensRatio in the createRun function to allow configurable context token management. Updated related tests to validate these changes and ensure proper functionality.
feat(summarization): add minReserveTokens, context pruning, and overflow recovery configurations
Introduced new configuration options for summarization, including minReserveTokens, context pruning settings, and overflow recovery parameters. Updated the createRun function to accommodate these new options and added a comprehensive test suite to validate their functionality and integration within the summarization process.
feat(summarization): add updatePrompt and reserveTokensRatio to summarization configuration
Introduced an updatePrompt field for updating existing summaries with new messages, enhancing the flexibility of the summarization process. Additionally, added reserveTokensRatio to the configuration schema, allowing for improved management of token allocation during summarization. Updated related tests to validate these new features.
feat(logging): add on_agent_log event handler for structured logging
Implemented an on_agent_log event handler in both the agents' callbacks and responses to facilitate structured logging of agent activities. This enhancement allows for better tracking and debugging of agent interactions by logging messages with associated metadata. Updated the summarization process to ensure proper handling of log events.
fix: remove duplicate IBalanceUpdate interface declaration
perf(usage): single-pass partition of collectedUsage
Replace two Array.filter() passes with a single for-of loop that
partitions message vs. summarization usages in one iteration.
fix(BaseClient): shallow-copy message content before mutating and preserve string content
Avoid mutating the original message.content array in-place when
appending a summary block. Also convert string content to a text
content part instead of silently discarding it.
fix(ui): fix Part.tsx indentation and useStepHandler summarize-complete handling
- Fix SUMMARY else-if branch indentation in Part.tsx to match chain level
- Guard ON_SUMMARIZE_COMPLETE with didFinalize flag to avoid unnecessary
re-renders when no summarizing parts exist
- Protect against undefined completeData.summary instead of unsafe spread
fix(agents): use strict enabled check for summarization handlers
Change summarizationConfig?.enabled !== false to === true so handlers
are not registered when summarizationConfig is undefined.
chore: fix initializeClient JSDoc and move DEFAULT_RESERVE_RATIO to module scope
refactor(Summary): align collapse/expand behavior with Reasoning component
- Single render path instead of separate streaming vs completed branches
- Use useMessageContext for isSubmitting/isLatestMessage awareness so
the "Summarizing..." label only shows during active streaming
- Default to collapsed (matching Reasoning), user toggles to expand
- Add proper aria attributes (aria-hidden, role, aria-controls, contentId)
- Hide copy button while actively streaming
feat(summarization): default to self-summarize using agent's own provider/model
When no summarization config is provided (neither in librechat.yaml nor
on the agent), automatically enable summarization using the agent's own
provider and model. The agents package already provides default prompts,
so no prompt configuration is needed.
Also removes the dead resolveSummarizationLLMConfig in summarize.ts
(and its spec) — run.ts buildAgentContext is the single source of truth
for summarization config resolution. Removes the duplicate
RuntimeSummarizationConfig local type in favor of the canonical
SummarizationConfig from data-provider.
chore: schema and type cleanup for summarization
- Add trigger field to summarizationAgentOverrideSchema so per-agent
trigger overrides in librechat.yaml are not silently stripped by Zod
- Remove unused SummarizationStatus type from runs.ts
- Make AppSummarizationConfig.enabled non-optional to reflect the
invariant that loadSummarizationConfig always sets it
refactor(responses): extract duplicated on_agent_log handler
refactor(run): use agents package types for summarization config
Import SummarizationConfig, ContextPruningConfig, and
OverflowRecoveryConfig from @librechat/agents and use them to
type-check the translation layer in buildAgentContext. This ensures
the config object passed to the agent graph matches what it expects.
- Use `satisfies AgentSummarizationConfig` on the config object
- Cast contextPruningConfig and overflowRecoveryConfig to agents types
- Properly narrow trigger fields from DeepPartial to required shape
feat(config): add maxToolResultChars to base endpoint schema
Add maxToolResultChars to baseEndpointSchema so it can be configured
on any endpoint in librechat.yaml. Resolved during agent initialization
using getProviderConfig's endpoint resolution: custom endpoint config
takes precedence, then the provider-specific endpoint config, then the
shared `all` config.
Passed through to the agents package ToolNode, which uses it to cap
tool result length before it enters the context window. When not
configured, the agents package computes a sensible default from
maxContextTokens.
fix(summarization): forward agent model_parameters in self-summarize default
When no explicit summarization config exists, the self-summarize
default now forwards the agent's model_parameters as the
summarization parameters. This ensures provider-specific settings
(e.g. Bedrock region, credentials, endpoint host) are available
when the agents package constructs the summarization LLM.
fix(agents): register summarization handlers by default
Change the enabled gate from === true to !== false so handlers
register when no explicit summarization config exists. This aligns
with the self-summarize default where summarization is always on
unless explicitly disabled via enabled: false.
refactor(summarization): let agents package inherit clientOptions for self-summarize
Remove model_parameters forwarding from the self-summarize default.
The agents package now reuses the agent's own clientOptions when the
summarization provider matches the agent's provider, inheriting all
provider-specific settings (region, credentials, proxy, etc.)
automatically.
refactor(summarization): use MessageContentComplex[] for summary content
Unify summary content to always use MessageContentComplex[] arrays,
matching the pattern used by on_message_delta. No more string | array
unions — content is always an array of typed blocks ({ type: 'text',
text: '...' } for text, { type: 'reasoning_content', ... } for
reasoning).
Agents package:
- SummaryContentBlock.content: MessageContentComplex[] (was string)
- tokenCount now optional (not sent on deltas)
- Removed reasoning field — reasoning is now a content block type
- streamAndCollect normalizes all chunks to content block arrays
- Delta events pass content blocks directly
LibreChat:
- SummaryContentPart.content: Agents.MessageContentComplex[]
- Updated Part.tsx, Summary.tsx, useStepHandler.ts, BaseClient.js
- Summary.tsx derives display text from content blocks via useMemo
- Aggregator uses simple array spread
refactor(summarization): enhance summary handling and text extraction
- Updated BaseClient.js to improve summary text extraction, accommodating both legacy and new content formats.
- Modified summarization logic to ensure consistent handling of summary content across different message formats.
- Adjusted test cases in summarization.e2e.spec.js to utilize the new summary text extraction method.
- Refined SSE useStepHandler to initialize summary content as an array.
- Updated configuration schema by removing unused minReserveTokens field.
- Cleaned up SummaryContentPart type by removing rangeHash property.
These changes streamline the summarization process and ensure compatibility with various content structures.
refactor(summarization): streamline usage tracking and logging
- Removed direct checks for summarization nodes in ModelEndHandler and replaced them with a dedicated markSummarizationUsage function for better readability and maintainability.
- Updated OpenAIChatCompletionController and responses handlers to utilize the new markSummarizationUsage function for setting usage types.
- Enhanced logging functionality by ensuring the logger correctly handles different log levels.
- Introduced a new useCopyToClipboard hook in the Summary component to encapsulate clipboard copy logic, improving code reusability and clarity.
These changes improve the overall structure and efficiency of the summarization handling and logging processes.
refactor(summarization): update summary content block documentation
- Removed outdated comment regarding the last summary content block in BaseClient.js.
- Added a new comment to clarify the purpose of the findSummaryContentBlock method, ensuring consistency in documentation.
These changes enhance code clarity and maintainability by providing accurate descriptions of the summarization logic.
refactor(summarization): update summary content structure in tests
- Modified the summarization content structure in e2e tests to use an array format for text, aligning with recent changes in summary handling.
- Updated test descriptions to clarify the behavior of context token calculations, ensuring consistency and clarity in the tests.
These changes enhance the accuracy and maintainability of the summarization tests by reflecting the updated content structure.
refactor(summarization): remove legacy E2E test setup and configuration
- Deleted the e2e-setup.js and jest.e2e.config.js files, which contained legacy configurations for E2E tests using real API keys.
- Introduced a new summarization.e2e.ts file that implements comprehensive E2E backend integration tests for the summarization process, utilizing real AI providers and tracking summaries throughout the run.
These changes streamline the testing framework by consolidating E2E tests into a single, more robust file while removing outdated configurations.
refactor(summarization): enhance E2E tests and error handling
- Added a cleanup step to force exit after all tests to manage Redis connections.
- Updated the summarization model to 'claude-haiku-4-5-20251001' for consistency across tests.
- Improved error handling in the processStream function to capture and return processing errors.
- Enhanced logging for cross-run tests and tight context scenarios to provide better insights into test execution.
These changes improve the reliability and clarity of the E2E tests for the summarization process.
refactor(summarization): enhance test coverage for maxContextTokens behavior
- Updated run-summarization.test.ts to include a new test case ensuring that maxContextTokens does not exceed user-defined limits, even when calculated ratios suggest otherwise.
- Modified summarization.e2e.ts to replace legacy UsageMetadata type with a more appropriate type for collectedUsage, improving type safety and clarity in the test setup.
These changes improve the robustness of the summarization tests by validating context token constraints and refining type definitions.
feat(summarization): add comprehensive E2E tests for summarization process
- Introduced a new summarization.e2e.test.ts file that implements extensive end-to-end integration tests for the summarization pipeline, covering the full flow from LibreChat to agents.
- The tests utilize real AI providers and include functionality to track summaries during and between runs.
- Added necessary cleanup steps to manage Redis connections post-tests and ensure proper exit.
These changes enhance the testing framework by providing robust coverage for the summarization process, ensuring reliability and performance under real-world conditions.
fix(service): import logger from winston configuration
- Removed the import statement for logger from '@librechat/data-schemas' and replaced it with an import from '~/config/winston'.
- This change ensures that the logger is correctly sourced from the updated configuration, improving consistency in logging practices across the application.
refactor(summary): simplify Summary component and enhance token display
- Removed the unused `meta` prop from the `SummaryButton` component to streamline its interface.
- Updated the token display logic to use a localized string for better internationalization support.
- Adjusted the rendering of the `meta` information to improve its visibility within the `Summary` component.
These changes enhance the clarity and usability of the Summary component while ensuring better localization practices.
feat(summarization): add maxInputTokens configuration for summarization
- Introduced a new `maxInputTokens` property in the summarization configuration schema to control the amount of conversation context sent to the summarizer, with a default value of 10000.
- Updated the `createRun` function to utilize the new `maxInputTokens` setting, allowing for more flexible summarization based on agent context.
These changes enhance the summarization capabilities by providing better control over input token limits, improving the overall summarization process.
refactor(summarization): simplify maxInputTokens logic in createRun function
- Updated the logic for the `maxInputTokens` property in the `createRun` function to directly use the agent's base context tokens when the resolved summarization configuration does not specify a value.
- This change streamlines the configuration process and enhances clarity in how input token limits are determined for summarization.
These modifications improve the maintainability of the summarization configuration by reducing complexity in the token calculation logic.
feat(summary): enhance Summary component to display meta information
- Updated the SummaryContent component to accept an optional `meta` prop, allowing for additional contextual information to be displayed above the main content.
- Adjusted the rendering logic in the Summary component to utilize the new `meta` prop, improving the visibility of supplementary details.
These changes enhance the user experience by providing more context within the Summary component, making it clearer and more informative.
refactor(summarization): standardize reserveRatio configuration in summarization logic
- Replaced instances of `reserveTokensRatio` with `reserveRatio` in the `createRun` function and related tests to unify the terminology across the codebase.
- Updated the summarization configuration schema to reflect this change, ensuring consistency in how the reserve ratio is defined and utilized.
- Removed the per-agent override logic for summarization configuration, simplifying the overall structure and enhancing clarity.
These modifications improve the maintainability and readability of the summarization logic by standardizing the configuration parameters.
* fix: circular dependency of `~/models`
* chore: update logging scope in agent log handlers
Changed log scope from `[agentus:${data.scope}]` to `[agents:${data.scope}]` in both the callbacks and responses controllers to ensure consistent logging format across the application.
* feat: calibration ratio
* refactor(tests): update summarizationConfig tests to reflect changes in enabled property
Modified tests to check for the new `summarizationEnabled` property instead of the deprecated `enabled` field in the summarization configuration. This change ensures that the tests accurately validate the current configuration structure and behavior of the agents.
* feat(tests): add markSummarizationUsage mock for improved test coverage
Introduced a mock for the markSummarizationUsage function in the responses unit tests to enhance the testing of summarization usage tracking. This addition supports better validation of summarization-related functionalities and ensures comprehensive test coverage for the agents' response handling.
* refactor(tests): simplify event handler setup in createResponse tests
Removed redundant mock implementations for event handlers in the createResponse unit tests, streamlining the setup process. This change enhances test clarity and maintainability while ensuring that the tests continue to validate the correct behavior of usage tracking during on_chat_model_end events.
* refactor(agents): move calibration ratio capture to finally block
Reorganized the logic for capturing the calibration ratio in the AgentClient class to ensure it is executed in the finally block. This change guarantees that the ratio is captured even if the run is aborted, enhancing the reliability of the response message persistence. Removed redundant code and improved clarity in the handling of context metadata.
* refactor(agents): streamline bulk write logic in recordCollectedUsage function
Removed redundant bulk write operations and consolidated document handling in the recordCollectedUsage function. The logic now combines all documents into a single bulk write operation, improving efficiency and reducing error handling complexity. Updated logging to provide consistent error messages for bulk write failures.
* refactor(agents): enhance summarization configuration resolution in createRun function
Streamlined the summarization configuration logic by introducing a base configuration and allowing for overrides from agent-specific settings. This change improves clarity and maintainability, ensuring that the summarization configuration is consistently applied while retaining flexibility for customization. Updated the handling of summarization parameters to ensure proper integration with the agent's model and provider settings.
* refactor(agents): remove unused tokenCountMap and streamline calibration ratio handling
Eliminated the unused tokenCountMap variable from the AgentClient class to enhance code clarity. Additionally, streamlined the logic for capturing the calibration ratio by using optional chaining and a fallback value, ensuring that context metadata is consistently defined. This change improves maintainability and reduces potential confusion in the codebase.
* refactor(agents): extract agent log handler for improved clarity and reusability
Refactored the agent log handling logic by extracting it into a dedicated function, `agentLogHandler`, enhancing code clarity and reusability across different modules. Updated the event handlers in both the OpenAI and responses controllers to utilize the new handler, ensuring consistent logging behavior throughout the application.
* test: add summarization event tests for useStepHandler
Implemented a series of tests for the summarization events in the useStepHandler hook. The tests cover scenarios for ON_SUMMARIZE_START, ON_SUMMARIZE_DELTA, and ON_SUMMARIZE_COMPLETE events, ensuring proper handling of summarization logic, including message accumulation and finalization. This addition enhances test coverage and validates the correct behavior of the summarization process within the application.
* refactor(config): update summarizationTriggerSchema to use enum for type validation
Changed the type of the `type` field in the summarizationTriggerSchema from a string to an enum with a single value 'token_count'. This modification enhances type safety and ensures that only valid types are accepted in the configuration, improving overall clarity and maintainability of the schema.
* test(usage): add bulk write tests for message and summarization usage
Implemented tests for the bulk write functionality in the recordCollectedUsage function, covering scenarios for combined message and summarization usage, summarization-only usage, and message-only usage. These tests ensure correct document handling and token rollup calculations, enhancing test coverage and validating the behavior of the usage tracking logic.
* refactor(Chat): enhance clipboard copy functionality and type definitions in Summary component
Updated the Summary component to improve the clipboard copy functionality by handling clipboard permission errors. Refactored type definitions for SummaryProps to use a more specific type, enhancing type safety. Adjusted the SummaryButton and FloatingSummaryBar components to accept isCopied and onCopy props, promoting better separation of concerns and reusability.
* chore(translations): remove unused "Expand Summary" key from English translations
Deleted the "Expand Summary" key from the English translation file to streamline the localization resources and improve clarity in the user interface. This change helps maintain an organized and efficient translation structure.
* refactor: adjust token counting for Claude model to account for API discrepancies
Implemented a correction factor for token counting when using the Claude model, addressing discrepancies between Anthropic's API and local tokenizer results. This change ensures accurate token counts by applying a scaling factor, improving the reliability of token-related functionalities.
* refactor(agents): implement token count adjustment for Claude model messages
Added a method to adjust token counts for messages processed by the Claude model, applying a correction factor to align with API expectations. This enhancement improves the accuracy of token counting, ensuring reliable functionality when interacting with the Claude model.
* refactor(agents): token counting for media content in messages
Introduced a new method to estimate token costs for image and document blocks in messages, improving the accuracy of token counting. This enhancement ensures that media content is properly accounted for, particularly for the Claude model, by integrating additional token estimation logic for various content types. Updated the token counting function to utilize this new method, enhancing overall reliability and functionality.
* chore: fix missing import
* fix(agents): clamp baseContextTokens and document reserve ratio change
Prevent negative baseContextTokens when maxOutputTokens exceeds the
context window (misconfigured models). Document the 10%→5% default
reserve ratio reduction introduced alongside summarization.
* fix(agents): include media tokens in hydrated token counts
Add estimateMediaTokensForMessage to createTokenCounter so the hydration
path (used by hydrateMissingIndexTokenCounts) matches the precomputed
path in AgentClient.getTokenCountForMessage. Without this, messages
containing images or documents were systematically undercounted during
hydration, risking context window overflow.
Add 34 unit tests covering all block-type branches of
estimateMediaTokensForMessage.
* fix(agents): include summarization output tokens in usage return value
The returned output_tokens from recordCollectedUsage now reflects all
billed LLM calls (message + summarization). Previously, summarization
completions were billed but excluded from the returned metadata, causing
a discrepancy between what users were charged and what the response
message reported.
* fix(tests): replace process.exit with proper Redis cleanup in e2e test
The summarization E2E test used process.exit(0) to work around a Redis
connection opened at import time, which killed the Jest runner and
bypassed teardown. Use ioredisClient.quit() and keyvRedisClient.disconnect()
for graceful cleanup instead.
* fix(tests): update getConvo imports in OpenAI and response tests
Refactor test files to import getConvo from the main models module instead of the Conversation submodule. This change ensures consistency across tests and simplifies the import structure, enhancing maintainability.
* fix(clients): improve summary text validation in BaseClient
Refactor the summary extraction logic to ensure that only non-empty summary texts are considered valid. This change enhances the robustness of the message processing by utilizing a dedicated method for summary text retrieval, improving overall reliability.
* fix(config): replace z.any() with explicit union in summarization schema
Model parameters (temperature, top_p, etc.) are constrained to
primitive types rather than the policy-violating z.any().
* refactor(agents): deduplicate CLAUDE_TOKEN_CORRECTION constant
Export from the TS source in packages/api and import in the JS client,
eliminating the static class property that could drift out of sync.
* refactor(agents): eliminate duplicate selfProvider in buildAgentContext
selfProvider and provider were derived from the same expression with
different type casts. Consolidated to a single provider variable.
* refactor(agents): extract shared SSE handlers and restrict log levels
- buildSummarizationHandlers() factory replaces triplicated handler
blocks across responses.js and openai.js
- agentLogHandlerObj exported from callbacks.js for consistent reuse
- agentLogHandler restricted to an allowlist of safe log levels
(debug, info, warn, error) instead of accepting arbitrary strings
* fix(SSE): batch summarize deltas, add exhaustiveness check, conditional error announcement
- ON_SUMMARIZE_DELTA coalesces rapid-fire renders via requestAnimationFrame
instead of calling setMessages per chunk
- Exhaustive never-check on TStepEvent catches unhandled variants at
compile time when new StepEvents are added
- ON_SUMMARIZE_COMPLETE error announcement only fires when a summary
part was actually present and removed
* feat(agents): persist instruction overhead in contextMeta and seed across runs
Extend contextMeta with instructionOverhead and toolCount so the
provider-observed instruction overhead is persisted on the response message
and seeded into the pruner on subsequent runs. This enables the pruner to
use a calibrated budget from the first call instead of waiting for a
provider observation, preventing the ratio collapse caused by local
tokenizer overestimating tool schema tokens.
The seeded overhead is only used when encoding and tool count match
between runs, ensuring stale values from different configurations
are discarded.
* test(agents): enhance OpenAI test mocks for summarization handlers
Updated the OpenAI test suite to include additional mock implementations for summarization handlers, including buildSummarizationHandlers, markSummarizationUsage, and agentLogHandlerObj. This improves test coverage and ensures consistent behavior during testing.
* fix(agents): address review findings for summarization v2
Cancel rAF on unmount to prevent stale Recoil writes from dead
component context. Clear orphaned summarizing:true parts when
ON_SUMMARIZE_COMPLETE arrives without a summary payload. Add null
guard and safe spread to agentLogHandler. Handle Anthropic-format
base64 image/* documents in estimateMediaTokensForMessage. Use
role="region" for expandable summary content. Add .describe() to
contextMeta Zod fields. Extract duplicate usage loop into helper.
* refactor: simplify contextMeta to calibrationRatio + encoding only
Remove instructionOverhead and toolCount from cross-run persistence —
instruction tokens change too frequently between runs (prompt edits,
tool changes) for a persisted seed to be reliable. The intra-run
calibration in the pruner still self-corrects via provider observations.
contextMeta now stores only the tokenizer-bias ratio and encoding,
which are stable across instruction changes.
* test(SSE): enhance useStepHandler tests for ON_SUMMARIZE_COMPLETE behavior
Updated the test for ON_SUMMARIZE_COMPLETE to clarify that it finalizes the existing part with summarizing set to false when the summary is undefined. Added assertions to verify the correct behavior of message updates and the state of summary parts.
* refactor(BaseClient): remove handleContextStrategy and truncateToolCallOutputs functions
Eliminated the handleContextStrategy method from BaseClient to streamline message handling. Also removed the truncateToolCallOutputs function from the prompts module, simplifying the codebase and improving maintainability.
* refactor: add AGENT_DEBUG_LOGGING option and refactor token count handling in BaseClient
Introduced AGENT_DEBUG_LOGGING to .env.example for enhanced debugging capabilities. Refactored token count handling in BaseClient by removing the handleTokenCountMap method and simplifying token count updates. Updated AgentClient to log detailed token count recalculations and adjustments, improving traceability during message processing.
* chore: update dependencies in package-lock.json and package.json files
Bumped versions of several dependencies, including @librechat/agents to ^3.1.62 and various AWS SDK packages to their latest versions. This ensures compatibility and incorporates the latest features and fixes.
* chore: imports order
* refactor: extract summarization config resolution from buildAgentContext
* refactor: rename and simplify summarization configuration shaping function
* refactor: replace AgentClient token counting methods with single-pass pure utility
Extract getTokenCount() and getTokenCountForMessage() from AgentClient
into countFormattedMessageTokens(), a pure function in packages/api that
handles text, tool_call, image, and document content types in one loop.
- Decompose estimateMediaTokensForMessage into block-level helpers
(estimateImageDataTokens, estimateImageBlockTokens, estimateDocumentBlockTokens)
shared by both estimateMediaTokensForMessage and the new single-pass function
- Remove redundant per-call getEncoding() resolution (closure captures once)
- Remove deprecated gpt-3.5-turbo-0301 model branching
- Drop this.getTokenCount guard from BaseClient.sendMessage
* refactor: streamline token counting in createTokenCounter function
Simplified the createTokenCounter function by removing the media token estimation and directly calculating the token count. This change enhances clarity and performance by consolidating the token counting logic into a single pass, while maintaining compatibility with Claude's token correction.
* refactor: simplify summarization configuration types
Removed the AppSummarizationConfig type and directly used SummarizationConfig in the AppConfig interface. This change streamlines the type definitions and enhances consistency across the codebase.
* chore: import order
* fix: summarization event handling in useStepHandler
- Cancel pending summarizeDeltaRaf in clearStepMaps to prevent stale
frames firing after map reset or component unmount
- Move announcePolite('summarize_completed') inside the didFinalize
guard so screen readers only announce when finalization actually occurs
- Remove dead cleanup closure returned from stepHandler useCallback body
that was never invoked by any caller
* fix: estimate tokens for non-PDF/non-image base64 document blocks
Previously estimateDocumentBlockTokens returned 0 for unrecognized MIME
types (e.g. text/plain, application/json), silently underestimating
context budget. Fall back to character-based heuristic or countTokens.
* refactor: return cloned usage from markSummarizationUsage
Avoid mutating LangChain's internal usage_metadata object by returning
a shallow clone with the usage_type tag. Update all call sites in
callbacks, openai, and responses controllers to use the returned value.
* refactor: consolidate debug logging loops in buildMessages
Merge the two sequential O(n) debug-logging passes over orderedMessages
into a single pass inside the map callback where all data is available.
* refactor: narrow SummaryContentPart.content type
Replace broad Agents.MessageContentComplex[] with the specific
Array<{ type: ContentTypes.TEXT; text: string }> that all producers
and consumers already use, improving compile-time safety.
* refactor: use single output array in recordCollectedUsage
Have processUsageGroup append to a shared array instead of returning
separate arrays that are spread into a third, reducing allocations.
* refactor: use for...in in hydrateMissingIndexTokenCounts
Replace Object.entries with for...in to avoid allocating an
intermediate tuple array during token map hydration.
|
||
|
|
f9060fa25f
|
🔧 chore: Update ESLint Config & Run Linter (#10986) | ||
|
|
656e1abaea
|
🪦 refactor: Remove Legacy Code (#10533)
* 🗑️ chore: Remove unused Legacy Provider clients and related helpers * Deleted OpenAIClient and GoogleClient files along with their associated tests. * Removed references to these clients in the clients index file. * Cleaned up typedefs by removing the OpenAISpecClient export. * Updated chat controllers to use the OpenAI SDK directly instead of the removed client classes. * chore/remove-openapi-specs * 🗑️ chore: Remove unused mergeSort and misc utility functions * Deleted mergeSort.js and misc.js files as they are no longer needed. * Removed references to cleanUpPrimaryKeyValue in messages.js and adjusted related logic. * Updated mongoMeili.ts to eliminate local implementations of removed functions. * chore: remove legacy endpoints * chore: remove all plugins endpoint related code * chore: remove unused prompt handling code and clean up imports * Deleted handleInputs.js and instructions.js files as they are no longer needed. * Removed references to these files in the prompts index.js. * Updated docker-compose.yml to simplify reverse proxy configuration. * chore: remove unused LightningIcon import from Icons.tsx * chore: clean up translation.json by removing deprecated and unused keys * chore: update Jest configuration and remove unused mock file * Simplified the setupFiles array in jest.config.js by removing the fetchEventSource mock. * Deleted the fetchEventSource.js mock file as it is no longer needed. * fix: simplify endpoint type check in Landing and ConversationStarters components * Updated the endpoint type check to use strict equality for better clarity and performance. * Ensured consistency in the handling of the azureOpenAI endpoint across both components. * chore: remove unused dependencies from package.json and package-lock.json * chore: remove legacy EditController, associated routes and imports * chore: update banResponse logic to refine request handling for banned users * chore: remove unused validateEndpoint middleware and its references * chore: remove unused 'res' parameter from initializeClient in multiple endpoint files * chore: remove unused 'isSmallScreen' prop from BookmarkNav and NewChat components; clean up imports in ArchivedChatsTable and useSetIndexOptions hooks; enhance localization in PromptVersions * chore: remove unused import of Constants and TMessage from MobileNav; retain only necessary QueryKeys import * chore: remove unused TResPlugin type and related references; clean up imports in types and schemas |
||
|
|
c0cb48256e
|
🤖 refactor: Improve Agent Handoff Context Tracking (#10553)
* chore: update @librechat/agents dependency to version 3.0.18 * refactor: add optional metadata field to message schema and types * chore: update @librechat/agents to v3.0.19 * refactor: update return type of sendCompletion method to include metadata * chore: linting * chore: update @librechat/agents dependency to v3.0.20 * refactor: implement agent labeling for conversation history in multi-agent scenarios * refactor: improve error handling for capturing agent ID map in AgentClient * refactor: clear agentIdMap and related properties during client disposal to prevent memory leaks * chore: update sendCompletion method for FakeClient to return an object with completion and metadata fields |
||
|
|
667e78c51e
|
📦 chore: Remove @langchain/community & Related Legacy Code (#10375)
* chore: remove `@langchain/community` dependency * refactor: remove SerpAPI integration and update related imports * chore: remove legacy code with unnecessary dependencies * chore: cleanup packages * chore: cleanup packages * chore: update openai dependency version to 5.10.1 * chore: add back @librechat/agents dependency * chore: downgrade openai dependency from 5.10.1 to 5.8.2 * Remove dependency on @librechat/agents from the API package * chore: add @librechat/agents dependency to the API package * fix: add useLegacyContent property to RunAgent type and propagate it in createRun function * chore: remove openai dependency version 5.10.1 from package.json |
||
|
|
8a4a5a4790
|
🤖 feat: Agent Handoffs (Routing) (#10176)
* feat: Add support for agent handoffs with edges in agent forms and schemas chore: Mark `agent_ids` field as deprecated in favor of edges across various schemas and types chore: Update dependencies for @langchain/core and @librechat/agents to latest versions chore: Update peer dependency for @librechat/agents to version 3.0.0-rc2 in package.json chore: Update @librechat/agents dependency to version 3.0.0-rc3 in package.json and package-lock.json feat: first pass, multi-agent handoffs fix: update output type to ToolMessage in memory handling functions fix: improve type checking for graphConfig in createRun function refactor: remove unused content filtering logic in AgentClient chore: update @librechat/agents dependency to version 3.0.0-rc4 in package.json and package-lock.json fix: update @langchain/core peer dependency version to ^0.3.72 in package.json and package-lock.json fix: update @librechat/agents dependency to version 3.0.0-rc6 in package.json and package-lock.json; refactor stream rate handling in various endpoints feat: Agent handoff UI chore: update @librechat/agents dependency to version 3.0.0-rc8 in package.json and package-lock.json fix: improve hasInfo condition and adjust UI element classes in AgentHandoff component refactor: remove current fixed agent display from AgentHandoffs component due to redundancy feat: enhance AgentHandoffs UI with localized beta label and improved layout chore: update @librechat/agents dependency to version 3.0.0-rc10 in package.json and package-lock.json feat: add `createSequentialChainEdges` function to add back agent chaining via multi-agents feat: update `createSequentialChainEdges` call to only provide conversation context between agents feat: deprecate Agent Chain functionality and update related methods for improved clarity * chore: update @librechat/agents dependency to version 3.0.0-rc11 in package.json and package-lock.json * refactor: remove unused addCacheControl function and related imports and import from @librechat/agents * chore: remove unused i18n keys * refactor: remove unused format export from index.ts * chore: update @librechat/agents to v3.0.0-rc13 * chore: remove BEDROCK_LEGACY provider from Providers enum * chore: update @librechat/agents to version 3.0.2 in package.json |
||
|
|
7c9a868d34
|
📝 feat: Add Markdown Rendering Support for Artifacts (#10049)
* Add Markdown rendering support for artifacts * Add tests * Remove custom code for mermaid * Remove unnecessary dark mode hook * refactor: optimize mermaid dependencies - Added support for additional MIME types in artifact templates. - Updated mermaidDependencies to include new packages: class-variance-authority, clsx, tailwind-merge, and @radix-ui/react-slot. - Refactored zoom and refresh icons in MermaidDiagram component for improved clarity and maintainability. * fix: add Markdown support for artifacts rendering * feat: support 'text/md' as an additional MIME type for Markdown artifacts * refactor: simplify markdownDependencies structure in artifacts utility --------- Co-authored-by: Danny Avila <danny@librechat.ai> |
||
|
|
48f6f8f2f8
|
📎 feat: Upload as Text Support for Plaintext, STT, RAG, and Token Limits (#8868)
* 🪶 feat: Add Support for Uploading Plaintext Files feat: delineate between OCR and text handling in fileConfig field of config file - also adds support for passing in mimetypes as just plain file extensions feat: add showLabel bool to support future synthetic component DynamicDropdownInput feat: add new combination dropdown-input component in params panel to support file type token limits refactor: move hovercard to side to align with other hovercards chore: clean up autogenerated comments feat: add delineation to file upload path between text and ocr configured filetypes feat: add token limit checks during file upload refactor: move textParsing out of ocrEnabled logic refactor: clean up types for filetype config refactor: finish decoupling DynamicDropdownInput from fileTokenLimits fix: move image token cost function into file to fix circular dependency causing unittest to fail and remove unused var for linter chore: remove out of scope code following review refactor: make fileTokenLimit conform to existing styles chore: remove unused localization string chore: undo changes to DynamicInput and other strays feat: add fileTokenLimit to all provider config panels fix: move textParsing back into ocr tool_resource block for now so that it doesn't interfere with other upload types * 📤 feat: Add RAG API Endpoint Support for Text Parsing (#8849) * feat: implement RAG API integration for text parsing with fallback to native parsing * chore: remove TODO now that placeholder and fllback are implemented * ✈️ refactor: Migrate Text Parsing to TS (#8892) * refactor: move generateShortLivedToken to packages/api * refactor: move textParsing logic into packages/api * refactor: reduce nesting and dry code with createTextFile * fix: add proper source handling * fix: mock new parseText and parseTextNative functions in jest file * ci: add test coverage for textParser * 💬 feat: Add Audio File Support to Upload as Text (#8893) * feat: add STT support for Upload as Text * refactor: move processAudioFile to packages/api * refactor: move textParsing from utils to files * fix: remove audio/mp3 from unsupported mimetypes test since it is now supported * ✂️ feat: Configurable File Token Limits and Truncation (#8911) * feat: add configurable fileTokenLimit default value * fix: add stt to fileConfig merge logic * fix: add fileTokenLimit to mergeFileConfig logic so configurable value is actually respected from yaml * feat: add token limiting to parsed text files * fix: add extraction logic and update tests so fileTokenLimit isnt sent to LLM providers * fix: address comments * refactor: rename textTokenLimiter.ts to text.ts * chore: update form-data package to address CVE-2025-7783 and update package-lock * feat: use default supported mime types for ocr on frontend file validation * fix: should be using logger.debug not console.debug * fix: mock existsSync in text.spec.ts * fix: mock logger rather than every one of its function calls * fix: reorganize imports and streamline file upload processing logic * refactor: update createTextFile function to use destructured parameters and improve readability * chore: update file validation to use EToolResources for improved type safety * chore: update import path for types in audio processing module * fix: update file configuration access and replace console.debug with logger.debug for improved logging --------- Co-authored-by: Dustin Healy <dustinhealy1@gmail.com> Co-authored-by: Dustin Healy <54083382+dustinhealy@users.noreply.github.com> |
||
|
|
445e9eae85
|
🧩 fix: Human Message Content Handling for Legacy Content (#8525)
* wip: first pass content strings * 📦 chore: update @langchain/core to v0.3.62 for data-provider dev dep. * 📦 chore: bump @langchain/core to v0.3.62 for api dep. * 📦 chore: move @langchain/core to peerDependencies in package.json and package-lock.json * fix: update formatContentStrings to create HumanMessage directly from formatted content * chore: import order |
||
|
|
452151e408
|
🐛 fix: RAG API failing with OPENID_REUSE_TOKENS Enabled (#8090)
* feat: Implement Short-Lived JWT Token Generation for RAG API * fix: Update import paths * fix: Correct environment variable names for OpenID on behalf flow * fix: Remove unnecessary spaces in OpenID on behalf flow userinfo scope --------- Co-authored-by: Atef Bellaaj <slalom.bellaaj@external.daimlertruck.com> |
||
|
|
01e9b196bc
|
🤖 feat: Streamline Endpoints to Agent Framework (#8013)
* refactor(buildEndpointOption): Improve error logging in middleware, consolidate `isAgents` builder logic, remove adding `modelsConfig` to `endpointOption`
* refactor: parameter extraction and organization in agent services, minimize redundancy of shared fields across objects, make clear distinction of parameters processed uniquely by LibreChat vs LLM Provider Configs
* refactor(createPayload): streamline all endpoints to agent route
* fix: add `modelLabel` to response sender options for agent initialization
* chore: correct log message context in EditController abort controller cleanup
* chore: remove unused abortRequest hook
* chore: remove unused addToCache module and its dependencies
* refactor: remove AskController and related routes, update endpoint URLs (now all streamlined to agents route)
* chore: remove unused bedrock route and its related imports
* refactor: simplify response sender logic for Google endpoint
* chore: add `modelDisplayLabel` handling for agents endpoint
* feat: add file search capability to ephemeral agents, update code interpreter selection based of file upload, consolidate main upload menu for all endpoints
* feat: implement useToolToggle hook for managing tool toggle state, refactor CodeInterpreter and WebSearch components to utilize new hook
* feat: add ToolsDropdown component to BadgeRow for enhanced tool options
* feat: introduce BadgeRowContext and BadgeRowProvider for managing conversation state, refactor related components to utilize context
* feat: implement useMCPSelect hook for managing MCP selection state, refactor MCPSelect component to utilize new hook
* feat: enhance BadgeRowContext with MCPSelect and tool toggle functionality, refactor related components to utilize updated context and hooks
* refactor: streamline useToolToggle hook by integrating setEphemeralAgent directly into toggle logic and removing redundant setValue function
* refactor: consolidate codeApiKeyForm and searchApiKeyForm from CodeInterpreter and WebSearch to utilize new context properties
* refactor: update CheckboxButton to support controlled state and enhance ToolsDropdown with permission-based toggles for web search and code interpreter
* refactor: conditionally render CheckboxButton in CodeInterpreter and WebSearch components for improved UI responsiveness
* chore: add jotai dependency to package.json and package-lock.json
* chore: update brace-expansion package to version 2.0.2 in package-lock.json due to CVE-2025-5889
* Revert "chore: add jotai dependency to package.json and package-lock.json"
This reverts commit
|
||
|
|
d6a17784dc
|
🔗 feat: Agent Chain (Mixture-of-Agents) (#6374)
* wip: first pass, dropdown for selecting sequential agents * refactor: Improve agent selection logic and enhance performance in SequentialAgents component * wip: seq. agents working ideas * wip: sequential agents style change * refactor: move agent form options/submission outside of AgentConfig * refactor: prevent repeating code * refactor: simplify current agent display in SequentialAgents component * feat: persist form value handling in AgentSelect component for agent_ids * feat: first pass, sequential agnets agent update * feat: enhance message display with agent updates and empty text handling * chore: update Icon component to use EModelEndpoint for agent endpoints * feat: update content type checks in BaseClient to use constants for better readability * feat: adjust max context tokens calculation to use 90% of the model's max tokens * feat: first pass, agent run message pruning * chore: increase max listeners for abort controller to prevent memory leaks * feat: enhance runAgent function to include current index count map for improved token tracking * chore: update @librechat/agents dependency to version 2.2.5 * feat: update icons and style of SequentialAgents component for improved UI consistency * feat: add AdvancedButton and AdvancedPanel components for enhanced agent settings navigation, update styling for agent form * chore: adjust minimum height of AdvancedPanel component for better layout consistency * chore: update @librechat/agents dependency to version 2.2.6 * feat: enhance message formatting by incorporating tool set into agent message processing, in order to allow better mix/matching of agents (as tool calls for tools not found in set will be stringified) * refactor: reorder components in AgentConfig for improved readability and maintainability * refactor: enhance layout of AgentUpdate component for improved visual structure * feat: add DeepSeek provider to Bedrock settings and schemas * feat: enhance link styling in mobile.css for better visibility and accessibility * fix: update banner model import in update banner script; export Banner model * refactor: `duplicateAgentHandler` to include tool_resources only for OCR context files * feat: add 'qwen-vl' to visionModels for enhanced model support * fix: change image format from JPEG to PNG in DALLE3 response * feat: reorganize Advanced components and add localizations * refactor: simplify JSX structure in AgentChain component to defer container styling to parent * feat: add FormInput component for reusable input handling * feat: make agent recursion limit configurable from builder * feat: add support for agent capabilities chain in AdvancedPanel and update data-provider version * feat: add maxRecursionLimit configuration for agents and update related documentation * fix: update CONFIG_VERSION to 1.2.3 in data provider configuration * feat: replace recursion limit input with MaxAgentSteps component and enhance input handling * feat: enhance AgentChain component with hover card for additional information and update related labels * fix: pass request and response objects to `createActionTool` when using assistant actions to prevent auth error * feat: update AgentChain component layout to include agent count display * feat: increase default max listeners and implement capability check function for agent chain * fix: update link styles in mobile.css for better visibility in dark mode * chore: temp. remove agents package while bumping shared packages * chore: update @langchain/google-genai package to version 0.1.11 * chore: update @langchain/google-vertexai package to version 0.2.2 * chore: add @librechat/agents package at version 2.2.8 * feat: add deepseek.r1 model with token rate and context values for bedrock |
||
|
|
d6ab769b80
|
⚠️ refactor: Use Error Content Part Instead Of Throwing Error for Agents (#6262) | ||
|
|
e14df5956a
|
✨ feat: Anthropic Agents Prompt Caching & UI Accessibility Enhancements (#6045)
* chore: remove auto-focus for now * refactor: move react-hook-form Controller Logic to AgentSelect from AgentPanel * fix: a11y focus issue with AgentSelect by never replacing it in its component tree * fix: maintain ComboBox focus and force re-render on agent ID change in AgentPanel * chore: `gemini-2.0-flash-lite-preview-02-05` (deprecated) * refactor: extract cache control logic and headers configuration to helper functions in AnthropicClient * feat: anthropic agents prompt caching * chore: bump @librechat/agents and related dependencies * fix: typo |
||
|
|
350e72dede
|
🧠 feat: Reasoning UI for Agents (#5904)
* chore: bump https-proxy-agent and @librechat/agents * refactor: Improve error logging in OllamaClient for API fetch failures * feat: Add DeepSeek provider support and enhance provider name handling * refactor: Use Providers.OLLAMA constant for model name check in fetchModels function * feat: Enhance formatAgentMessages to handle reasoning content type * feat: OpenRouter Agent Reasoning * hard work and dedicationgit add .env.example :) * fix: Handle Google social login with missing last name Social login with Google was previously displaying 'undefined' when a user's last name was empty or not provided. Changes: - Conditionally render last name only if it exists - Prevent displaying 'undefined' when last name is missing * fix: add missing file endings for developers yml,yaml and log --------- Co-authored-by: Mohamed Al-Duraji <mbalduraji@college.harvard.edu> Co-authored-by: Deepak Kendole <deepakdpk101@gmail.com> Co-authored-by: Peter Rothlaender <peter.rothlaender@ginkgo.com> |
||
|
|
352565c9a6
|
🎥 feat: YouTube Tool (#5582)
* adding youtube tool * refactor: use short `url` param instead of `videoUrl` * refactor: move API key retrieval to a separate credentials module * refactor: remove unnecessary `isEdited` message property * refactor: remove unnecessary `isEdited` message property pt. 2 * refactor: YouTube Tool with new `tool()` generator, handle tools already created by new `tool` generator * fix: only reset request data for multi-convo messages * refactor: enhance YouTube tool by adding transcript parsing and returning structured JSON responses * refactor: update transcript parsing to handle raw response and clean up text output * feat: support toolkits and refactor YouTube tool as a toolkit for better LLM usage * refactor: remove unused OpenAPI specs and streamline tools transformation in loadAsyncEndpoints * refactor: implement manifestToolMap for better tool management and streamline authentication handling * feat: support toolkits for assistants * refactor: rename loadedTools to toolDefinitions for clarity in PluginController and assistant controllers * feat: complete support of toolkits for assistants --------- Co-authored-by: Danilo Pejakovic <danilo.pejakovic@leoninestudios.com> |
||
|
|
b35a8b78e2
|
🔧 refactor: Improve Agent Context & Minor Fixes (#5349)
* refactor: Improve Context for Agents
* 🔧 fix: Safeguard against undefined properties in OpenAIClient response handling
* refactor: log error before re-throwing for original stack trace
* refactor: remove toolResource state from useFileHandling, allow svg files
* refactor: prevent verbose logs from axios errors when using actions
* refactor: add silent method recordTokenUsage in AgentClient
* refactor: streamline token count assignment in BaseClient
* refactor: enhance safety settings handling for Gemini 2.0 model
* fix: capabilities structure in MCPConnection
* refactor: simplify civic integrity threshold handling in GoogleClient and llm
* refactor: update token count retrieval method in BaseClient tests
* ci: fix test for svg
|
||
|
|
1a815f5e19
|
🎉 feat: Code Interpreter API and Agents Release (#4860)
* feat: Code Interpreter API & File Search Agent Uploads chore: add back code files wip: first pass, abstract key dialog refactor: influence checkbox on key changes refactor: update localization keys for 'execute code' to 'run code' wip: run code button refactor: add throwError parameter to loadAuthValues and getUserPluginAuthValue functions feat: first pass, API tool calling fix: handle missing toolId in callTool function and return 404 for non-existent tools feat: show code outputs fix: improve error handling in callTool function and log errors fix: handle potential null value for filepath in attachment destructuring fix: normalize language before rendering and prevent null return fix: add loading indicator in RunCode component while executing code feat: add support for conditional code execution in Markdown components feat: attachments refactor: remove bash fix: pass abort signal to graph/run refactor: debounce and rate limit tool call refactor: increase debounce delay for execute function feat: set code output attachments feat: image attachments refactor: apply message context refactor: pass `partIndex` feat: toolCall schema/model/methods feat: block indexing feat: get tool calls chore: imports chore: typing chore: condense type imports feat: get tool calls fix: block indexing chore: typing refactor: update tool calls mapping to support multiple results fix: add unique key to nav link for rendering wip: first pass, tool call results refactor: update query cache from successful tool call mutation style: improve result switcher styling chore: note on using \`.toObject()\` feat: add agent_id field to conversation schema chore: typing refactor: rename agentMap to agentsMap for consistency feat: Agent Name as chat input placeholder chore: bump agents 📦 chore: update @langchain dependencies to latest versions to match agents package 📦 chore: update @librechat/agents dependency to version 1.8.0 fix: Aborting agent stream removes sender; fix(bedrock): completion removes preset name label refactor: remove direct file parameter to use req.file, add `processAgentFileUpload` for image uploads feat: upload menu feat: prime message_file resources feat: implement conversation access validation in chat route refactor: remove file parameter from processFileUpload and use req.file instead feat: add savedMessageIds set to track saved message IDs in BaseClient, to prevent unnecessary double-write to db feat: prevent duplicate message saves by checking savedMessageIds in AgentController refactor: skip legacy RAG API handling for agents feat: add files field to convoSchema refactor: update request type annotations from Express.Request to ServerRequest in file processing functions feat: track conversation files fix: resendFiles, addPreviousAttachments handling feat: add ID validation for session_id and file_id in download route feat: entity_id for code file uploads/downloads fix: code file edge cases feat: delete related tool calls feat: add stream rate handling for LLM configuration feat: enhance system content with attached file information fix: improve error logging in resource priming function * WIP: PoC, sequential agents WIP: PoC Sequential Agents, first pass content data + bump agents package fix: package-lock WIP: PoC, o1 support, refactor bufferString feat: convertJsonSchemaToZod fix: form issues and schema defining erroneous model fix: max length issue on agent form instructions, limit conversation messages to sequential agents feat: add abort signal support to createRun function and AgentClient feat: PoC, hide prior sequential agent steps fix: update parameter naming from config to metadata in event handlers for clarity, add model to usage data refactor: use only last contentData, track model for usage data chore: bump agents package fix: content parts issue refactor: filter contentParts to include tool calls and relevant indices feat: show function calls refactor: filter context messages to exclude tool calls when no tools are available to the agent fix: ensure tool call content is not undefined in formatMessages feat: add agent_id field to conversationPreset schema feat: hide sequential agents feat: increase upload toast duration to 10 seconds * refactor: tool context handling & update Code API Key Dialog feat: toolContextMap chore: skipSpecs -> useSpecs ci: fix handleTools tests feat: API Key Dialog * feat: Agent Permissions Admin Controls feat: replace label with button for prompt permission toggle feat: update agent permissions feat: enable experimental agents and streamline capability configuration feat: implement access control for agents and enhance endpoint menu items feat: add welcome message for agent selection in localization feat: add agents permission to access control and update version to 0.7.57 * fix: update types in useAssistantListMap and useMentions hooks for better null handling * feat: mention agents * fix: agent tool resource race conditions when deleting agent tool resource files * feat: add error handling for code execution with user feedback * refactor: rename AdminControls to AdminSettings for clarity * style: add gap to button in AdminSettings for improved layout * refactor: separate agent query hooks and check access to enable fetching * fix: remove unused provider from agent initialization options, creates issue with custom endpoints * refactor: remove redundant/deprecated modelOptions from AgentClient processes * chore: update @librechat/agents to version 1.8.5 in package.json and package-lock.json * fix: minor styling issues + agent panel uniformity * fix: agent edge cases when set endpoint is no longer defined * refactor: remove unused cleanup function call from AppService * fix: update link in ApiKeyDialog to point to pricing page * fix: improve type handling and layout calculations in SidePanel component * fix: add missing localization string for agent selection in SidePanel * chore: form styling and localizations for upload filesearch/code interpreter * fix: model selection placeholder logic in AgentConfig component * style: agent capabilities * fix: add localization for provider selection and improve dropdown styling in ModelPanel * refactor: use gpt-4o-mini > gpt-3.5-turbo * fix: agents configuration for loadDefaultInterface and update related tests * feat: DALLE Agents support |
||
|
|
95201908e9
|
📦 fix: npm warnings; chore: bump deprecated packages (#4707)
* chore: bump langchain deps to address vulnerability warnings * chore: bump community package and install textsplitters package * fix: update expected result in tokenSplit tests for accuracy * chore: remove CodeSherpa tools * chore: remove E2B tools and loadToolSuite * chore: remove CodeBrew tool and update related references * chore: remove HumanTool and ChatTool, update tool references * chore: remove Zapier tool from manifest.json and update SerpAPI * chore: remove basic tools * chore: update import path for RecursiveCharacterTextSplitter * chore: update import path for DynamicStructuredTool * chore: remove extractionChain.js and update tool filtering logic * chore: npm audit fix * chore: bump google packages * chore: update DALL-E tool to DALL-E-3 and adjust authentication logic * ci: update message classes * chore: elliptic npm audit fix * chore: update CallbackManager import and remove deprecated tool handling logic * chore: imports order * chore: remove unused code --------- Co-authored-by: Max Sanna <max@maxsanna.com> |
||
|
|
766643ea1c
|
🎯 feat: Enhance Title Parameter Parsing with new Anthropic Format | ||
|
|
0c2a583df8
|
🔧 refactor: Optimize Agent Tool Loading and Fix Bedrock Tool Handling (#4641)
* fix: bedrock tool name regex * fix: pass args as single input, attempt json first. * refactor: remove toolMap from agent tool load as is not used * fix: update formatAgentMessages test to use strictEqual for args comparison, testing new behavior |
||
|
|
2e519f9b57
|
🤖 feat: Custom Endpoint Agents (experimental) (#4627)
* wip: first pass, custom endpoint agents * chore: imports * chore: consolidate exports * fix: imports * feat: convert message.content array to strings for legacy format handling (deepseek/groq) * refactor: normalize ollama endpoint name * refactor: update mocking in isDomainAllowed.spec.js * refactor: update deepseekModels in tokens.js and tokens.spec.js |
||
|
|
ad74350036
|
🚧 chore: merge latest dev build (#4288)
* fix: agent initialization, add `collectedUsage` handling * style: improve side panel styling * refactor(loadAgent): Optimize order agent project ID retrieval * feat: code execution * fix: typing issues * feat: ExecuteCode content part * refactor: use local state for default collapsed state of analysis content parts * fix: code parsing in ExecuteCode component * chore: bump agents package, export loadAuthValues * refactor: Update handleTools.js to use EnvVar for code execution tool authentication * WIP * feat: download code outputs * fix(useEventHandlers): type issues * feat: backend handling for code outputs * Refactor: Remove console.log statement in Part.tsx * refactor: add attachments to TMessage/messageSchema * WIP: prelim handling for code outputs * feat: attachments rendering * refactor: improve attachments rendering * fix: attachments, nullish edge case, handle attachments from event stream, bump agents package * fix filename download * fix: tool assignment for 'run code' on agent creation * fix: image handling by adding attachments * refactor: prevent agent creation without provider/model * refactor: remove unnecessary space in agent creation success message * refactor: select first model if selecting provider from empty on form * fix: Agent avatar bug * fix: `defaultAgentFormValues` causing boolean typing issue and typeerror * fix: capabilities counting as tools, causing duplication of them * fix: formatted messages edge case where consecutive content text type parts with the latter having tool_call_ids would cause consecutive AI messages to be created. furthermore, content could not be an array for tool_use messages (anthropic limitation) * chore: bump @librechat/agents dependency to version 1.6.9 * feat: bedrock agents * feat: new Agents icon * feat: agent titling * feat: agent landing * refactor: allow sharing agent globally only if user is admin or author * feat: initial AgentPanelSkeleton * feat: AgentPanelSkeleton * feat: collaborative agents * chore: add potential authorName as part of schema * chore: Remove unnecessary console.log statement * WIP: agent model parameters * chore: ToolsDialog typing and tool related localization chnages * refactor: update tool instance type (latest langchain class), and rename google tool to 'google' proper * chore: add back tools * feat: Agent knowledge files upload * refactor: better verbiage for disabled knowledge * chore: debug logs for file deletions * chore: debug logs for file deletions * feat: upload/delete agent knowledge/file-search files * feat: file search UI for agents * feat: first pass, file search tool * chore: update default agent capabilities and info |
||
|
|
9a393be012
|
🪨 fix: Formatting Edge Case Handling for Bedrock Messages (#4016)
* refactor: Remove console.log statement in SelectDropDown component * fix(bedrock): edge case - message.content as string creating message formatting issue |
||
|
|
d59b62174f
|
🪨 feat: AWS Bedrock support (#3935)
* feat: Add BedrockIcon component to SVG library * feat: EModelEndpoint.bedrock * feat: first pass, bedrock chat. note: AgentClient is returning `agents` as conversation.endpoint * fix: declare endpoint in initialization step * chore: Update @librechat/agents dependency to version 1.4.5 * feat: backend content aggregation for agents/bedrock * feat: abort agent requests * feat: AWS Bedrock icons * WIP: agent provider schema parsing * chore: Update EditIcon props type * refactor(useGenerationsByLatest): make agents and bedrock editable * refactor: non-assistant message content, parts * fix: Bedrock response `sender` * fix: use endpointOption.model_parameters not endpointOption.modelOptions * fix: types for step handler * refactor: Update Agents.ToolCallDelta type * refactor: Remove unnecessary assignment of parentMessageId in AskController * refactor: remove unnecessary assignment of parentMessageId (agent request handler) * fix(bedrock/agents): message regeneration * refactor: dynamic form elements using react-hook-form Controllers * fix: agent icons/labels for messages * fix: agent actions * fix: use of new dynamic tags causing application crash * refactor: dynamic settings touch-ups * refactor: update Slider component to allow custom track class name * refactor: update DynamicSlider component styles * refactor: use Constants value for GLOBAL_PROJECT_NAME (enum) * feat: agent share global methods/controllers * fix: agents query * fix: `getResponseModel` * fix: share prompt a11y issue * refactor: update SharePrompt dialog theme styles * refactor: explicit typing for SharePrompt * feat: add agent roles/permissions * chore: update @librechat/agents dependency to version 1.4.7 for tool_call_ids edge case * fix(Anthropic): messages.X.content.Y.tool_use.input: Input should be a valid dictionary * fix: handle text parts with tool_call_ids and empty text * fix: role initialization * refactor: don't make instructions required * refactor: improve typing of Text part * fix: setShowStopButton for agents route * chore: remove params for now * fix: add streamBuffer and streamRate to help prevent 'Overloaded' errors from Anthropic API * refactor: remove console.log statement in ContentRender component * chore: typing, rename Context to Delete Button * chore(DeleteButton): logging * refactor(Action): make accessible * style(Action): improve a11y again * refactor: remove use/mention of mongoose sessions * feat: first pass, sharing agents * feat: visual indicator for global agent, remove author when serving to non-author * wip: params * chore: fix typing issues * fix(schemas): typing * refactor: improve accessibility of ListCard component and fix console React warning * wip: reset templates for non-legacy new convos * Revert "wip: params" This reverts commit |
||
|
|
a0291ed155
|
🚧 chore: merge latest dev build to main repo (#3844)
* agents - phase 1 (#30) * chore: copy assistant files * feat: frontend and data-provider * feat: backend get endpoint test * fix(MessageEndpointIcon): switched to AgentName and AgentAvatar * fix: small fixes * fix: agent endpoint config * fix: show Agent Builder * chore: install agentus * chore: initial scaffolding for agents * fix: updated Assistant logic to Agent Logic for some Agent components * WIP first pass, demo of agent package * WIP: initial backend infra for agents * fix: agent list error * wip: agents routing * chore: Refactor useSSE hook to handle different data events * wip: correctly emit events * chore: Update @librechat/agentus npm dependency to version 1.0.9 * remove comment * first pass: streaming agent text * chore: Remove @librechat/agentus root-level workspace npm dependency * feat: Agent Schema and Model * fix: content handling fixes * fix: content message save * WIP: new content data * fix: run step issue with tool calls * chore: Update @librechat/agentus npm dependency to version 1.1.5 * feat: update controller and agent routes * wip: initial backend tool and tool error handling support * wip: tool chunks * chore: Update @librechat/agentus npm dependency to version 1.1.7 * chore: update tool_call typing, add test conditions and logs * fix: create agent * fix: create agent * first pass: render completed content parts * fix: remove logging, fix step handler typing * chore: Update @librechat/agentus npm dependency to version 1.1.9 * refactor: cleanup maps on unmount * chore: Update BaseClient.js to safely count tokens for string, number, and boolean values * fix: support subsequent messages with tool_calls * chore: export order * fix: select agent * fix: tool call types and handling * chore: switch to anthropic for testing * fix: AgentSelect * refactor: experimental: OpenAIClient to use array for intermediateReply * fix(useSSE): revert old condition for streaming legacy client tokens * fix: lint * revert `agent_id` to `id` * chore: update localization keys for agent-related components * feat: zod schema handling for actions * refactor(actions): if no params, no zodSchema * chore: Update @librechat/agentus npm dependency to version 1.2.1 * feat: first pass, actions * refactor: empty schema for actions without params * feat: Update createRun function to accept additional options * fix: message payload formatting; feat: add more client options * fix: ToolCall component rendering when action has no args but has output * refactor(ToolCall): allow non-stringy args * WIP: first pass, correctly formatted tool_calls between providers * refactor: Remove duplicate import of 'roles' module * refactor: Exclude 'vite.config.ts' from TypeScript compilation * refactor: fix agent related types > - no need to use endpoint/model fields for identifying agent metadata > - add `provider` distinction for agent-configured 'endpoint' - no need for agent-endpoint map - reduce complexity of tools as functions into tools as string[] - fix types related to above changes - reduce unnecessary variables for queries/mutations and corresponding react-query keys * refactor: Add tools and tool_kwargs fields to agent schema * refactor: Remove unused code and update dependencies * refactor: Update updateAgentHandler to use req.body directly * refactor: Update AgentSelect component to use localized hooks * refactor: Update agent schema to include tools and provider fields * refactor(AgentPanel): add scrollbar gutter, add provider field to form, fix agent schema required values * refactor: Update AgentSwitcher component to use selectedAgentId instead of selectedAgent * refactor: Update AgentPanel component to include alternateName import and defaultAgentFormValues * refactor(SelectDropDown): allow setting value as option while still supporting legacy usage (string values only) * refactor: SelectDropdown changes - Only necessary when the available values are objects with label/value fields and the selected value is expected to be a string. * refactor: TypeError issues and handle provider as option * feat: Add placeholder for provider selection in AgentPanel component * refactor: Update agent schema to include author and provider fields * fix: show expected 'create agent' placeholder when creating agent * chore: fix localization strings, hide capabilities form for now * chore: typing * refactor: import order and use compact agents schema for now * chore: typing * refactor: Update AgentForm type to use AgentCapabilities * fix agent form agent selection issues * feat: responsive agent selection * fix: Handle cancelled fetch in useSelectAgent hook * fix: reset agent form on accordion close/open * feat: Add agent_id to default conversation for agents endpoint * feat: agents endpoint request handling * refactor: reset conversation model on agent select * refactor: add `additional_instructions` to conversation schema, organize other fields * chore: casing * chore: types * refactor(loadAgentTools): explicitly pass agent_id, do not pass `model` to loadAgentTools for now, load action sets by agent_id * WIP: initial draft of real agent client initialization * WIP: first pass, anthropic agent requests * feat: remember last selected agent * feat: openai and azure connected * fix: prioritize agent model for runs unless an explicit override model is passed from client * feat: Agent Actions * fix: save agent id to convo * feat: model panel (#29) * feat: model panel * bring back comments * fix: method still null * fix: AgentPanel FormContext * feat: add more parameters * fix: style issues; refactor: Agent Controller * fix: cherry-pick * fix: Update AgentAvatar component to use AssistantIcon instead of BrainCircuit * feat: OGDialog for delete agent; feat(assistant): update Agent types, introduced `model_parameters` * feat: icon and general `model_parameters` update * feat: use react-hook-form better * fix: agent builder form reset issue when switching panels * refactor: modularize agent builder form --------- Co-authored-by: Danny Avila <danny@librechat.ai> * fix: AgentPanel and ModelPanel type issues and use `useFormContext` and `watch` instead of `methods` directly and `useWatch`. * fix: tool call issues due to invalid input (anthropic) of empty string * fix: handle empty text in Part component --------- Co-authored-by: Marco Beretta <81851188+berry-13@users.noreply.github.com> * refactor: remove form ModelPanel and fixed nested ternary expressions in AgentConfig * fix: Model Parameters not saved correctly * refactor: remove console log * feat: avatar upload and get for Agents (#36) Co-authored-by: Marco Beretta <81851188+berry-13@users.noreply.github.com> * chore: update to public package * fix: typing, optional chaining * fix: cursor not showing for content parts * chore: conditionally enable agents * ci: fix azure test * ci: fix frontend tests, fix eslint api * refactor: Remove unused errorContentPart variable * continue of the agent message PR (#40) * last fixes * fix: agentMap * pr merge test (#41) * fix: model icon not fetching correctly * remove console logs * feat: agent name * refactor: pass documentsMap as a prop to allow re-render of assistant form * refactor: pass documentsMap as a prop to allow re-render of assistant form * chore: Bump version to 0.7.419 * fix: TypeError: Cannot read properties of undefined (reading 'id') * refactor: update AgentSwitcher component to use ControlCombobox instead of Combobox --------- Co-authored-by: Marco Beretta <81851188+berry-13@users.noreply.github.com> |
||
|
|
7c1ee242eb
|
🪄 feat: Code Artifacts (#3798)
* feat: Add CodeArtifacts component to Beta settings tab * chore: Update npm dependency to @codesandbox/sandpack-react@2.18.2 * WIP: artifacts first pass * WIP first pass remark-directive * chore: revert markdown to original component + new artifacts rendering * refactor: first pass rewrite * refactor: add throttling * first pass styling * style: Add Radix Tabs, more styling changes * feat: second pass * style: code styling * fix: package markdown fixes * feat: Add useEffect hook to Artifacts component for visibility control, slide in animation * fix: only set artifact if there is content * refactor: typing and make latest artifact active if the number of artifacts changed * feat: artifacts + shadcnui * feat: Add Copy Code button to Artifacts component * feat: first pass streaming updates * refactor: optimize ordering of artifacts in Artifacts component * refactor: optimize ordering of artifacts and add latest artifact activation in Artifacts component * refactor: add order prop to Artifact * feat: update to latest, use update time for ordering * refactor: optimize ordering of artifacts and activate latest artifact in Artifacts component * wip: remove thinking text and artifact formatting if empty * refactor: optimize Markdown rendering and add support for code artifacts * feat: global state for current artifact Id and set on artifact preview click * refactor: Rename CodePreview component to ArtifactButton * refactor: apply growth to artifact frame so artifact preview can take full space * refactor: remove artifactIdsState * refactor: nullify artifact state and reset on empty conversation * feat: reset artifact state on conversation change * feat: artifacts system prompt in backend * refactor: update UI artifact toggle label to match localization key * style: remove ArtifactButton inline-block styling * feat: memoize ArtifactPreview, add html support * refactor: abstract out components * chore: bump react-resizable-panel * refactor: resizable panel order props * fix: side panel resizing crashes * style: temporarily remove scrolling, add better styling * chore: remove thinking for now * chore: preprocess artifacts for now * feat: Add auto scrolling to CodeMarkdown (artifacts) * feat: autoswitch to preview * feat: auto switch to code, adjust prompt, remove unused code * feat: refresh button * feat: open/close artifacts * wip: mermaid * refactor: w-fit Artifact button * chore: organize code * feat: first pass mermaid * refactor: improve panning logic in MermaidDiagram component * feat: center/zoom on first render * refactor: add centering with reset button * style: mermaid styling * refactor: add back MermaidDiagram * fix: static/html template * fix: mermaid * add examples to artifacts prompt * refactor: fix CodeBar plugin prop logic * refactor: remove unnecessary mention of artifacts when not requested * fix: remove preprocessCodeArtifacts function and fix imports * feat: improve artifacts guidelines and remove unnecessary mentions * refactor: improve artifacts guidelines and remove unnecessary mentions * chore: uninstall unused packages * chore: bump vite * chore: update three dependency to version 0.167.1 * refactor: move beta settings, add additional artifacts toggles * feat: artifacts mode toggles * refactor: adjust prompt * feat: shadcnui instructions * feat: code artifacts custom prompt mode * chore: Update artifacts UI labels and instructions localizations * refactor: Remove unused code in Markdown component |
||
|
|
8ca1e4f94f
|
🐛 fix: Anthropic Prompt Caching Edge Case (#3690) | ||
|
|
a45b384bbc
|
💾 feat: Anthropic Prompt Caching (#3670)
* wip: initial cache control implementation, add typing for transactions handling * feat: first pass of Anthropic Prompt Caching * feat: standardize stream usage as pass in when calculating token counts * feat: Add getCacheMultiplier function to calculate cache multiplier for different valueKeys and cacheTypes * chore: imports order * refactor: token usage recording in AnthropicClient, no need to "correct" as we have the correct amount * feat: more accurate token counting using stream usage data * feat: Improve token counting accuracy with stream usage data * refactor: ensure more accurate than not token estimations if custom instructions or files are not being resent with every request * refactor: cleanup updateUserMessageTokenCount to allow transactions to be as accurate as possible even if we shouldn't update user message token counts * ci: fix tests |
||
|
|
084cf266a2
|
🗃️ fix: revise RAG prompt (#3006) | ||
|
|
795bb9c568
|
🌏 refactor: Improve Title Prompt for Multilingual Titles (#2832) | ||
|
|
2ec821ea4c
|
🌍 : Updated Translations & AI Generation Scripts (#2666)
* chore: bun scripts * feat: comparisons * refactor: move scripts to own folder * feat: generated prompts script and Es output * feat: generated prompts * created prompts * feat: Russian localization prompts * translation setup * additional ES translations * additional ES translations * translation services * feat: additional translations * fix regex for parseParamPrompt * RU translations * remove stores from git * update gitignore * update gitignore * ZH translations * move gen prompt output location * ZH traditional translations * AR translations * chore: rename * JP * cleanup scripts * add additional instruction prompts * fix translation prompt and add DE * FR translations (rate limited so not complete) * chore: update translation comparisons * chore: remove unused AnthropicClient changes * refactor: use compositional styling for archive/delete buttons, fix manage archive table styling |
||
|
|
9d854dac07
|
🤖 feat: Gemini 1.5 Support (+Vertex AI) (#2383)
* WIP: gemini-1.5 support * feat: extended vertex ai support * fix: handle possibly undefined modelName * fix: gpt-4-turbo-preview invalid vision model * feat: specify `fileConfig.imageOutputType` and make PNG default image conversion type * feat: better truncation for errors including base64 strings * fix: gemini inlineData formatting * feat: RAG augmented prompt for gemini-1.5 * feat: gemini-1.5 rates and token window * chore: adjust tokens, update docs, update vision Models * chore: add back `ChatGoogleVertexAI` for chat models via vertex ai * refactor: ask/edit controllers to not use `unfinished` field for google endpoint * chore: remove comment * chore(ci): fix AppService test * chore: remove comment * refactor(GoogleSearch): use `GOOGLE_SEARCH_API_KEY` instead, issue warning for old variable * chore: bump data-provider to 0.5.4 * chore: update docs * fix: condition for gemini-1.5 using generative ai lib * chore: update docs * ci: add additional AppService test for `imageOutputType` * refactor: optimize new config value `imageOutputType` * chore: bump CONFIG_VERSION * fix(assistants): avatar upload |
||
|
|
cd7f3a51e1
|
🧠 feat: Cohere support as Custom Endpoint (#2328)
* chore: bump cohere-ai, fix firebase vulnerabilities by going down versions * feat: cohere rates and context windows * feat(createCoherePayload): transform openai payload for cohere compatibility * feat: cohere backend support * refactor(UnknownIcon): optimize icon render and add cohere * docs: add cohere to Compatible AI Endpoints * Update ai_endpoints.md |
||
|
|
a00756c469
|
⬇️ feat: Assistant File Downloads (#2234)
* WIP: basic route for file downloads and file strategy for generating readablestream to pipe as res * chore(DALLE3): add typing for OpenAI client * chore: add `CONSOLE_JSON` notes to dotenv.md * WIP: first pass OpenAI Assistants File Output handling * feat: first pass assistants output file download from openai * chore: yml vs. yaml variation to .gitignore for `librechat.yml` * refactor(retrieveAndProcessFile): remove redundancies * fix(syncMessages): explicit sort of apiMessages to fix message order on abort * chore: add logs for warnings and errors, show toast on frontend * chore: add logger where console was still being used |
||
|
|
798e8763d0
|
👓 feat: Vision Support for Assistants (#2195)
* refactor(assistants/chat): use promises to speed up initialization, initialize shared variables, include `attachedFileIds` to streamRunManager * chore: additional typedefs * fix(OpenAIClient): handle edge case where attachments promise is resolved * feat: createVisionPrompt * feat: Vision Support for Assistants |
||
|
|
1f0fb497f8
|
🎉 feat: Optimizations and Anthropic Title Generation (#2184)
* feat: add claude-3-haiku-20240307 to default anthropic list * refactor: optimize `saveMessage` calls mid-stream via throttling * chore: remove addMetadata operations and consolidate in BaseClient * fix(listAssistantsForAzure): attempt to specify correct model mapping as accurately as possible (#2177) * refactor(client): update last conversation setup with current assistant model, call newConvo again when assistants load to allow fast initial load and ensure assistant model is always the default, not the last selected model * refactor(cache): explicitly add TTL of 2 minutes when setting titleCache and add default TTL of 10 minutes to abortKeys cache * feat(AnthropicClient): conversation titling using Anthropic Function Calling * chore: remove extraneous token usage logging * fix(convos): unhandled edge case for conversation grouping (undefined conversation) * style: Improved style of Search Bar after recent UI update * chore: remove unused code, content part helpers * feat: always show code option |
||
|
|
45a95acec2
|
📂 feat: RAG Improvements (#2169)
* feat: new vector file processing strategy * chore: remove unused client files * chore: remove more unused client files * chore: remove more unused client files and move used to new dir * chore(DataIcon): add className * WIP: Model Endpoint Settings Update, draft additional context settings * feat: improve parsing for augmented prompt, add full context option * chore: remove volume mounting from rag.yml as no longer necessary |