Commit graph

3 commits

Author SHA1 Message Date
Danny Avila
45cc53c40b
🛰️ chore: bump @librechat/agents to v3.4.0, cover streamed subagent results e2e (#14647)
* test: cover streamed subagent results end to end

* test: assert real e2e conversation id

* test: harden streamed subagent e2e

* test: stop incompatible subagent fixtures

* chore: update @librechat/agents to version 3.4.0 in package.json and package-lock.json
2026-08-05 22:25:03 -04:00
Danny Avila
cc813f430e
🎯 feat: Tool Intent Label Capability (tool_intents) (#14499)
* 🎯 feat: Tool Intent Label Capability (tool_intents)

Adds the fourth member of the per-tool capability family (defer_loading,
allowed_callers, run_in_background): an admin capability
AgentCapabilities.tool_intents plus a per-tool
tool_options[name].describe_intent flag. Opted-in tools get an optional
intent string injected as the FIRST property of their schema — one
model-authored sentence per call, streamed to the client as the call's
live status label (args already reach the client verbatim, so no new
event plumbing). Native host tools (web_search, create_file/edit_file,
set_memory/delete_memory, ask_user_question) default on while the
capability is enabled; explicit false opts out. SDK-native intent
schemas (@librechat/agents coding suite) are recognized and left alone.

- packages/api/src/agents/intent.ts: structural sibling of
  background.ts — first-key non-mutating injection with registry
  parity (covers deferred/tool_search discovery), eligibility and
  PTC-only skips, arg read/strip helpers, self-spawn strip for defs and
  registry, ephemeral/model-spec synthesis with a tool_options merge so
  the background and intent toggles compose.
- handlers.ts: intent runs BEFORE background injection so the label
  stays the first streamed key when a tool carries both (pinned by
  test); the arg is stripped before invocation unless the tool's own
  schema declares it, on both the foreground and background-dispatch
  paths; PTC target schemas are sanitized like background's.
- Capability plumbing through all four routes (endpoint initialize,
  openai + responses controllers, the exported OpenAI-compatible
  service) plus handoff discovery and added-convo agents, and the
  intentToolNames execution channel via configurable.
- describe_intent on toolOptionsSchema (all three written-out Zod
  annotations), ToolOptions, TEphemeralAgent, TModelSpec (+ zod), and
  data-schemas doc comments (tool_options is Mixed — no migration).
- intent.spec.ts: 28 tests cloned from background.spec.ts structure,
  including the intent+background key-order composition.

* 🧯 fix: Codex Review — Opt-Out Strips SDK-Native Intent, Skip mcp_all Placeholders

- An explicit describe_intent: false now REMOVES an SDK-native intent
  property from the definition and registry entry, so the per-tool
  opt-out actually disables the arg's token cost for tools like
  web_search that carry the schema natively (SDK bodies tolerate its
  absence). Previously the early return left the property in place.
- synthesizeIntentToolOptions skips lazily-expanded mcp_all
  placeholders instead of recording options under names that
  applyIntentLabels' exact-name matching can never match, and documents
  the limitation (parity with synthesizeBackgroundToolOptions).

The P1 about the client not rendering the label is the documented
slicing: the UI streaming-label PR follows once #14391's ToolCallGroup
changes merge — args already reach the client, so that slice is purely
rendering.

* 🧯 fix: Codex Re-Review — Label Marker Guard, Capability Kill Switch, Late Defs, Service Threading

- removeIntentParam is now marker-guarded (the label contract's opening
  instruction discriminates it), so an MCP/action tool's own business
  `intent` parameter is never stripped by an opt-out or the disabled
  path — previously an explicit false could remove a real, possibly
  required argument.
- New sanitizeIntentLabels pass runs AFTER every registration step
  (the skill catalog appends its SDK definition post-injection): with
  tool_intents disabled it strips SDK-native intent labels from all
  definitions and registry entries, making the capability a real kill
  switch over their token cost; with it enabled it enforces explicit
  per-tool opt-outs on late-registered definitions.
- ask_user_question removed from the native default-on set: its graph
  tool is rebuilt in run.ts from its own Zod schema (also the HITL
  card's wire shape), so definition-level injection never reached the
  model. Its intent support lands with the HITL slice, which threads
  the label into the interrupt payload deliberately.
- The exported OpenAI-compatible service now threads intentToolNames
  into the run configurable, so the executor's PTC path can strip
  host-injected intent schemas on that route like the in-repo
  controllers do.

* 🧯 fix: Codex Round 2 — Post-Skill Injection, PTC Native Strip, Service Boundary, Honest Docs

- Intent injection now runs LAST in initializeAgent, after the skill
  catalog — which both appends its own definition and REPLACES upgraded
  ones (skill-aware read_file), clobbering an earlier injection while
  intentToolNames still listed the tool. Injection PREPENDS while
  background APPENDS, so intent stays the first schema property under
  the new ordering (pinned by a reverse-order composition test).
- The PTC target-schema strip is now marker-guarded strip-ALL: SDK-
  native intent labels (which are deliberately never in intentToolNames)
  are removed from sandbox-advertised schemas alongside host-injected
  ones; business intent params survive.
- toolIntentsAvailable on the exported service documents the loader
  boundary: a custom LoadToolsFn returning only structured instances
  bypasses definition/registry injection and sanitize by construction.
- librechat.example.yaml describes tool_intents as backend groundwork
  with UI rendering in an upcoming release rather than promising a live
  label today.

* 📦 chore: bump `@librechat/agents` to v3.3.6

Brings in the SDK half of tool intent labels (danny-avila/agents#347,
#349): intent-first schemas on the coding suite across all three
engines, plus web_search / subagent / skill / tool_search, and the
outcome / outcome_patch result channel.

Activates three host paths that were inert while no SDK tool shipped an
`intent` property — verified against the real 3.3.6 schemas:
- capability OFF now strips SDK-native labels (a real admin kill switch)
- explicit `describe_intent: false` removes them per tool
- host injection stays idempotent against an SDK schema, keeping
  `intent` first and never double-injecting

* 🔬 test: Real-Provider Verification for Tool Intent Labels

Adds the live check the unit tests structurally cannot perform: whether a
real model actually authors the injected arg, places it FIRST, and gives
sibling calls to one tool distinct labels. Reuses the existing
real-provider harness (in-memory Mongo, seeded user, credential
neutralizer) and the existing stdio MCP fixture as a genuine tool, so no
external service is involved.

- e2e/config/librechat.real.yaml: adds the e2e-memory MCP server and the
  tool_intents capability, giving the real model something to call. The
  sibling spec asserts only relative token growth, so the extra schemas
  do not perturb it.
- e2e/playwright.config.real.ts: optional Langfuse passthrough. The
  LANGFUSE_* keys match the credential-neutralizer pattern and were being
  blanked before the server booted; they are preserved explicitly, read
  from the invoking environment only, and never written to the generated
  config.
- e2e/specs/real/tool-intents.spec.ts: two facts stored in one turn, both
  through the same tool, asserting intent is the first key of each call
  and that the two labels differ. Args are read from persistence rather
  than the DOM deliberately — no UI renders the label yet, and
  persistence is what a reloaded conversation and the trace both read.

First run against claude-haiku-4-5 produced 'Recording the location of
the OAuth callback router' and 'Recording the location of the MCP
connection pool configuration' — distinct, first-position, no tool name.

Also updates tool-intent-spec.md: records the 3.3.7 removal of the tense
verb map with the evidence that motivated it, the trimmed description and
the marker's role as an API, and a new mandatory requirement that
client-side label rendering be gated on a server-sent signal rather than
the presence of an intent key (a tool's own business 'intent' parameter
would otherwise render as a status label).

* 📦 chore: bump `@librechat/agents` to v3.3.7 and dedupe the intent contract

Picks up danny-avila/agents#353: the tense verb map is gone (a bare
intent now displays unchanged, with completion carried by UI state), the
model-facing description is trimmed 502 → 289 chars, and both the marker
and the description are exported.

Stops redeclaring the SDK contract here:
- INTENT_LABEL_MARKER is imported instead of duplicated as a string
  literal. Every removal path in this module keys on it, and a local copy
  that drifted from the SDK's would make them all stop recognizing
  SDK-native labels — failing OPEN, with labels left in schemas and
  per-tool opt-outs silently inert.
- INTENT_DESCRIPTION is imported too, so host-injected tools and
  SDK-native tools present the model with one identical instruction.
  Keeping the old local copy would also have meant host-injected tools
  still paying ~126 tokens per schema while SDK tools paid ~72.

Verified live against real Anthropic after the trim: two sibling calls to
one MCP tool produced 'Storing the OAuth callback router file location'
and 'Storing the MCP connection pool configuration file location' —
first-position and distinct, so the shorter description holds compliance.
2026-07-29 15:40:52 -04:00
Danny Avila
db7011d567
📊 feat: Real-Time Context Window & Token Usage Tracking (#13670)
Some checks are pending
Docker Dev Branch Images Build / build (Dockerfile, lc-dev, node) (push) Waiting to run
Docker Dev Branch Images Build / build (Dockerfile.multi, lc-dev-api, api-build) (push) Waiting to run
GitNexus Index / index (push) Waiting to run
GitNexus Index / post-index (push) Blocked by required conditions
* 📊 feat: Real-Time Context Window & Token Usage Tracking

* 🧪 fix: Align Pricing Spec Dep Signatures with TxDeps

* 🩹 fix: Resolve Codex Findings for Context Usage Tracking

* 📊 feat: Granular Tool Token Breakdown with Deferred Splits

* 🧪 test: Cover Session Cost in Mock E2E and Scope Usage Selectors

* 🧪 test: Live Host-Pipeline Usage Verification (Env-Gated)

* 🧪 test: Local Real-Provider Multi-Turn E2E Harness

* 🪙 fix: Keep Tagged Usage Buckets Out of the Live Context Estimate

* 🩹 fix: Scoped Token-Config Fallback and Sequential Visibility for Usage Events

* 🩹 fix: Address Usage Review Findings — Cost Timing, Scoped Caches, Finalized Output

- carry the post-snapshot output estimate into the context snapshot at
  finalize so the gauge keeps the last response after live resets
- accumulate per-rate billable units and price the session cost at
  render, so usage events arriving before the token-config load still
  count once it resolves
- pass user-scoped token-config cache keys through loadConfigModels
  fetches and drop the controller's unscoped fallback to prevent serving
  another user's resolved config
- tag emitted usage events with a per-run seq so resume dedupe never
  drops a distinct call with an identical payload
- admit the static tokenConfig override in the custom endpoint schema so
  it survives zod parsing into req.config

* 🩹 fix: Align Client Usage Accounting with Backend Cost Semantics

- classify cache tokens by provider (shared inputTokensIncludesCache from
  data-provider, consumed by both the backend billing path and the client)
  instead of a magnitude heuristic, so Anthropic/Bedrock turns where cache
  is smaller than uncached input no longer under-bill input
- mirror resolveCompletionTokens on the client so Vertex-style hidden
  thinking tokens are reflected in the Output row and session cost
- prefer endpoint pricing over adapter-provider pricing so a custom
  endpoint can price a known model name without built-in rates shadowing it
- carry static cacheRead/cacheWrite overrides through the tokenConfig
  schema and buildTokenConfigMap

* 🩹 fix: Honor Static Token Config in Billing; Tighten Usage Freshness

- initializeCustom now uses a static endpoint tokenConfig as the agent's
  endpointTokenConfig (billing + balance checks), not just the advertised
  UI config — previously the gauge showed admin rates while the agent
  billed against built-in tables
- invalidate the token-config query alongside models on user-key add/
  revoke so context windows and pricing refresh without a reload
- include maxContextTokens in ChatForm's stabilized conversation memo so
  the gauge reflects a changed context-window setting immediately
- feed the live output estimate from the legacy content path (direct and
  assistants streams), setting from cumulative part text rather than
  accumulating deltas

* 🩹 fix: Resume Usage Dedup, Agent Pricing, and Partial Override Billing

- fold usage events idempotently by (runId, seq) so resume backfill no
  longer resets the conversation totals — a mid-stream reconnect keeps the
  usage of prompts already completed earlier in the session
- tap replayed pending message/reasoning/content events so output streamed
  past the resume snapshot reaches the live estimate, not just the message
- resolve cost against the agent's backing endpoint (Agents conversations
  report endpoint `agents` / provider `openAI`, neither of which keys a
  custom endpoint's tokenConfig)
- getMultiplier/getCacheMultiplier fall back to the standard tables for
  models absent from a partial endpointTokenConfig, so a partial static
  override no longer bills non-listed models at defaultRate while the UI
  shows the correct pattern rate

* 🩹 fix: Repaired Output in Gauge, Cache-Rate Keys, Config Gate, Usage Cleanup

- live/completed gauge counts the repaired completion (normalized output),
  so under-reporting providers don't drop the response from used context
- translate static tokenConfig cacheWrite/cacheRead onto the write/read
  keys getCacheMultiplier reads, so cache tokens bill at the configured
  rate instead of the prompt-rate fallback
- clear the token index and usage atoms when leaving a conversation, so
  visited histories don't accumulate in memory for the tab's lifetime
- wait for startupConfig before mounting the gauge, so a deployment with
  contextUsage disabled never briefly mounts it or fires the token-config
  query on first load

* 🩹 fix: Move Token-Config Resolution to TS; Key Live Usage by Created Convo

- extract the token-config resolution (override gathering + cache lookup +
  buildTokenConfigMap) into resolveTokenConfigMap in packages/api, leaving
  the /api controller a thin request-scoped wrapper (CLAUDE.md TS rule)
- getConvoKey prefers the user message's real conversationId once the
  `created` event stamps it, so a new chat's first-response live gauge and
  totals land under the id TokenUsage subscribes to instead of NEW_CONVO

* 🩹 fix: Clear Stale Redis Job Usage; Live-Tap Legacy Streams; Share Fetched Config

- DEL the Redis job hash before re-creating it so a reused streamId can't
  inherit a prior run's contextUsage/tokenUsage and backfill stale usage
- tap the legacy {message,text} stream branch (non-agent OpenAI/Anthropic
  streams) into the live estimate, not just the content path
- copy a deduped fetch's token config to every sibling endpoint sharing the
  baseURL/key/headers, so /token-config resolves each by its own name

* ⏪ revert: Don't DEL Redis job hash in createJob (breaks cross-replica resume)

createJob is an idempotent join — a second replica calls it for the same
streamId to share an in-flight stream's state. DELeting the hash wiped the
prior replica's persisted created/usage state, so a joining replica missed
the created event (GenerationJobManager cross-replica integration test).
Reverts the F1 change from 2bfce0c34b; the stale-usage concern doesn't
arise in practice (streamId is unique per generation).

* 🩹 fix: Best-Effort Usage Emit; Tag Hidden Sequential-Agent Usage

- wrap the ModelEndHandler usage emit in try/catch so a failed telemetry
  delivery (closed SSE / Redis publish error) can't abort the handler
  before thought-signature capture, which would break resumed tool calls
- tag hidden sequential-agent usage as 'sequential' (non-primary) so the
  client folds it into session cost/totals but not the live context gauge,
  instead of letting an undefined usage_type inflate the visible gauge

* 🩹 fix: Refetch Stale Token Config on Mount; Normalize Vertex for Lookup

- useTokenConfigQuery refetches on mount when stale, so a user-key change
  that invalidates tokenConfig while the gauge is unmounted takes effect on
  return instead of serving the prior key's resolved config
- normalize a Vertex-backed agent's provider (vertexai) to the google
  token-config key, so Gemini context windows and rates resolve instead of
  showing unknown context / $0 cost

* ✨ feat: Server-Side Per-Event Cost (Authoritative Pricing for the Gauge)

Move usage-cost pricing to the single source of truth. The backend prices
each model call with the same billing functions (premium tiers via
getMultiplier(inputTokenCount), cache rates) and emits the USD cost on
on_token_usage when interface.contextCost is enabled; the client sums
emitted costs instead of re-deriving from base token-config rates.

- computeUsageCostUSD reuses prepareTokenSpend/prepareStructuredTokenSpend
  so the emitted cost matches what is billed (incl. premium thresholds)
- getDefaultHandlers gains a usageCost pricing context; initialize.js wires
  db.getMultiplier/getCacheMultiplier gated on contextCost (agents path)
- client UsageTotals carries a summed costUSD; retire the client-side rate
  lookups (costFromUnits/calcUsageCost) that drifted from backend pricing
  and produced the provider-keying / cache-key / Vertex / premium findings
- keep normalizeUsageUnits for the displayed token counts; token-config is
  still used for the context-window meter

Fixes the premium-tier session-cost under-report (gpt-5.x / gemini-3.1
above their input thresholds).

* 🩹 fix: Branch-Accurate Usage Snapshot + Clearer Gauge Track Contrast

- re-anchor the context snapshot from the user message to the response
  message at finalize. Regenerating a response branches off a shared user
  message, so anchoring on it made the snapshot read as "active" on both
  branches — switching to the sibling branch showed the wrong (other
  branch's) context. The response message is branch-unique, so sibling
  branches now correctly fall back to their own per-branch totals.
- raise the gauge ring's track/fill contrast (muted track, prominent fill)
  so the used portion reads clearly as a fill-level indicator

* 🩹 fix: Tag Sequential Usage in Billing; Emit Subagent Cost; Reset Live on Resume Errors

- tag hidden sequential-agent usage `usage_type: 'sequential'` on the
  COLLECTED usage (not just the emit), and treat it as non-primary in
  recordCollectedUsage (billed, excluded from the reported output total) so
  hidden intermediate output stops inflating the parent's tokenCount/pruning
- emit on_token_usage from the subagent usage sink (tagged `subagent`, with
  authoritative cost when contextCost is on) so the gauge's session
  cost/totals include billed subagent usage; it stays out of the live meter
- call resetLive on the resumable 404 and max-retry terminal branches so the
  gauge doesn't keep counting stale in-flight tokens after the stream ends

* 🎨 fix: Contrast the Popup Context Bar; Revert Ring Restyle

- raise the popup breakdown's context progressbar contrast (muted
  surface-tertiary track, prominent text-primary fill) — that's the bar the
  contrast feedback was about
- revert the gauge ring restyle (kept its original border-heavy track /
  text-secondary fill); the ring wasn't the element in question

* 🩹 fix: Stop Snapshot Granularity Leaking Across Branches; Revert Tree Memo

- a null-anchor context snapshot was treated as active on every branch,
  leaking one generation's granular breakdown onto sibling branches. Require
  a non-null (response-message) anchor on the viewed branch instead, so
  siblings without a matching snapshot fall back to their own totals.
- revert the buildTree WeakMap memo in messages.ts. buildTree is pure (builds
  from shallow copies) so the memo was behaviorally identical, but it was the
  feature's only change to core branch-navigation selectors — removing it
  matches upstream and rules it out of branch-navigation debugging.

* 🪙 fix: Thread Endpoint Token Config to Agent Billing, Cost, and Context Limits

Custom-endpoint agents resolve an endpointTokenConfig during agent init but
it never reached the AgentClient, so spending, emitted cost, and runtime
max-token resolution all fell back to default rates for those agents.

- Surface options.endpointTokenConfig on the returned InitializedAgent.
- Pass it to the AgentClient (this.options.endpointTokenConfig) so the
  spending path bills at configured rates.
- Thread it through usageCost to computeUsageCostUSD so emitted per-event
  cost matches billing.
- getModelMaxTokens/getModelMaxOutputTokens fall back to the built-in map
  for models absent from a partial override (matches buildTokenConfigMap);
  consolidates the duplicated fallback in pricing.ts.

* 🪙 fix: Preserve Granular Breakdown Across Branch Switches

The granular context breakdown lives only in the live on_context_usage
snapshot — a single per-conversation slot, anchored to the latest response
and overwritten by each generation. Switching to a branch generated earlier
this session lost its tool/skill/system rows and fell back to coarse totals.

Retain each generation's finalized snapshot in a per-conversation map keyed
by its branch-unique response id (snapshotsByAnchorFamily). When the live
snapshot is off the viewed branch, walk the branch tail for its deepest
stored anchor and render that breakdown. Bounded by generation count and
cleared on conversation switch; the live/just-generated path is unchanged.

* 🪙 fix: Harden Resume Seeding and Subagent Usage Emission

- useResumableSSE: skip the trailing-output live seed when the resume
  carries a context snapshot; the snapshot's messageTokens already counts
  produced output, so seeding it again inflated usage until the next reset.
- AgentClient subagent emitter: await GenerationJobManager.emitChunk like
  every other caller (it persists before publishing), so a floating promise
  can't race job cleanup and a Redis/publish failure is caught by the
  emitter's try/catch instead of surfacing as an unhandled rejection.

* 🧪 test: Playwright Coverage for Context Breakdown Granularity

Add a test-only data-testid distinguishing the granular snapshot breakdown
(context-breakdown) from the coarse message-history estimate
(context-estimate), then assert granularity in the mock e2e harness:

- renders the granular breakdown from the live on_context_usage snapshot
  (guards that the snapshot event actually reaches the popover, not just the
  usage totals).
- preserves the granular breakdown after switching branches — regenerate to
  overwrite the single live snapshot, switch back, and confirm the rows
  survive via the per-anchor snapshot history map.

Branch regenerate/sibling selectors mirror the existing chat.spec branch test.
All three usage specs pass against the mock pipeline.

* 🪙 fix: Correct Resume Live-Seed, Fallback Re-index, and Subagent Emit Flush

Codex round on the prior commit:

- countTrailingOutputChars now counts only output at the very END of the
  aggregated content (0 when the model paused at a tool call), and the resume
  path always seeds it. The earlier skip-trailing-tool-parts behavior plus the
  skip-seed-when-snapshot gate together over- or under-counted in-flight
  output on resume; one rule fixes both — pre-invoke snapshot budget is never
  double-counted, and genuine in-flight output is no longer dropped.
- useTokenUsage re-indexes from the messages cache on tail change while
  submitting. The cache subscriber is muted during streaming, so without a
  context snapshot (non-agent streams) sumBranch missed the created tail and
  dropped history + prompt until finalize. Bounded — tailId only shifts on
  created/finalize/branch-switch.
- AgentClient tracks subagent usage emit promises and flushes them in
  chatCompletion's finally. The sink fires the emitter without awaiting, and
  resume reads the usage emitChunk persists (HSET), so cleanup must not race
  it or resumed clients miss billed subagent usage.
2026-06-13 19:38:28 -04:00