LibreChat/api/server/routes/agents/__tests__/streamTenant.spec.js
Danny Avila 08f9d7f7e0
🧵 feat: Stream a Background Subagent Child Through Its Own Generation Job (#18169)
* 🫀 feat: Let a Job Owner Keep a Silent Run Alive

A generation job's idle clock advances only with emitted chunks, so a run that
works without output for 20 minutes is reaped while it is still alive.
touchJob refreshes the idle clock and the storage lifetimes an append would
have extended, fenced on the generation epoch and limited to the replica that
owns the runtime.

* 🧶 refactor: Share One Fold From Child Frames to Content Parts

readSubagentContent folded public child frames into content parts inline.
The fold is now a synchronous function a second caller can use, so a frame
can be folded and published without yielding in between.

* 🏷️ feat: Mark the Generation Job of a Background Subagent Task

Job metadata gains subagentTaskId and subagentParentConversationId, set only
by the subagent task store. A marked job is a child-thread job, so it stays
out of the active chat list, and isBackgroundChildJob tells it apart from an
event-bound child, whose job the chat controller owns.

* 🧵 feat: Run a Background Child as a Resumable Generation Job

createChildGenerationHost opens a generation job keyed by the child thread id
and feeds it the child's public step frames. Text deltas that queue behind a
slow write merge instead of dropping, the terminal state is claimed before the
caller persists and the final event is published only afterwards, and no
stream failure can fail the child: the run and its result carry on without a
live view.

* 🪢 feat: Stream a Background Child Through Its Own Generation Job

The subagent task store takes a childGenerations dependency. Once a child's
thread and input row are durable it opens a job keyed by the thread id, feeds
it each public frame, refreshes its idle clock on every lease renewal, and
settles it around its own persistence, so a reader sees the final event only
after the result row exists. A cancel, a failure and a job aborted from
outside all end both the child and its stream.

endpoints.agents.subagentActivity.generationJobs turns it off. A stream that
cannot be opened or written never fails the child.

* 🔌 feat: Give the Server's Task Store Its Child Stream

The host wiring builds the child generation host from the generation job
manager and hands it to the subagent task store, so a background child's
stream is reachable in the running server.

* 🚧 fix: Refuse Chat Abort and Steer on a Background Child's Job

The chat abort route saves a user row and a partial response and upserts the
conversation, and the steer route queues text only the chat controller drains.
Both authorized on the job's owner alone, so either could act on a background
child's job and write a chat-shaped history into a view-only thread, or accept
a steer nothing would ever apply. Both now answer 409 for a job the subagent
task store owns; the child is stopped and steered through its control
endpoint.

* 🧹 fix: Evict a Finished Job Before a Live One at Capacity

The in-memory job store made room by deleting its oldest job whatever its
status, so a full store could end a run somebody was still waiting on. It now
takes the oldest finished job first. Also covers a job manager that shuts down
while a background child runs: the child still persists its cancellation.

* 🛰️ test: Read a Background Child's Stream From Another Replica

Against real Redis: a second replica resumes with everything the child has
streamed, receives only newer deltas live, then the final event; a child that
streams past the manager's 5,000-event early buffer before anyone attaches
still resumes whole on both replicas; and the owner's touch keeps a silent
child's job alive.

* 🧽 chore: Sort Imports in the Child Stream Files

* 🧪 test: Expect the Child Stream Lever in the Activity Defaults

* 📜 feat: Store What a Background Child Streamed as Standard Content (#18173)

A child's result row carried its text and a bounded activity projection, not
the content parts every agent message stores, so a reader had to rebuild the
child's output from the projection. The row now also carries what the child
streamed, on completion, failure and cancellation, in the standard shape.

endpoints.agents.subagentActivity.generationContentBytes bounds it (4 MiB by
default, 0 stores none). Content over the bound is left off, so the row always
fits and keeps its text and projection as before.

* 🛟 fix: Keep a Stalled or Failing Child Stream From Touching Its Child

Four ways the stream could still reach the run it only observes. A frame that
could not be copied threw into the child's own progress report. A job store
that never answered held the child's start behind job creation and its result
behind the terminal claim. And a stream that had stopped being written was
never ended, so its job read as a live child for twenty minutes after the
child finished.

Frames are now copied defensively, opening and settling are bounded, a job
that arrives after its child stopped waiting is ended, and a detached stream
is still touched and still settled with its full final event.

* 🧯 fix: Let a Child Stream Observe Its Run Without Ever Stopping It

Review round 1.

A failed Redis append retires the job's runtime by aborting its controller,
and the child stream forwarded every abort of that controller to the task
store as a cancellation. A stream-store outage therefore stopped the child's
model and tool work. The stream no longer signals the child at all: deletion,
shutdown and the control endpoint already stop a child through the task
store, and an aborted, reaped or retired job now only ends the live view.

The in-memory store's capacity eviction preferred any terminal job, including
one that still owed persistence, a host action, a schedule reconciliation or
a provider drain. Those are no longer treated as finished.

The chat abort route's refusal for a background child's job moves out of the
CJS route into createBackgroundChildAbortGuard, registered ahead of it.

* ♻️ fix: Replace a Dead Background Child's Job on Its Thread

A background child's job outlives an owner that crashed, or a settlement that
could not reach the store, for up to twenty minutes. A continuation of the
thread in that window was refused its own stream, because the dead job still
read as live. The caller of open holds the thread's lease, so a background
child's job still marked live there has lost its owner: it is replaced and its
readers are told to refetch. A live job that is not a background child's, such
as an event-bound child paused for approval, is still never replaced.

* 🧱 fix: Fail the Abort Guard Closed and Budget Content Against the Whole Row

Review round 2.

The abort guard passed the request on when its job lookup failed, so a
retried lookup in the route could run the chat-shaped abort on a child the
guard never saw. A failed lookup now ends the request at the error boundary.

A child's streamed content was checked against its own limit only. A result
row also carries the text and the transcript, so content that passed the
limit could push the document past MongoDB's 16 MiB and turn a successful
child into a failed one. The content is now the optional field: it is left
off whenever the row's large fields together would exceed the budget.

Also accounts for the guard's job read in the two abort route specs that
sequence it, and gives the limiter spec's api mock the guard.

* 🪶 refactor: Settle a Child Stream Once and Fold a Frame It Cannot Copy

The task store settles a child's stream on its outcome path and again in its
cleanup. The second call made another terminal claim that could only lose;
the stream now remembers that it claimed and only persists. A frame that
cannot be copied is still folded from the original, so the stored content
stays whole when the stream detaches over it.

* 🧪 test: Cover the Child Stream's Failure Branches and the Content Fold

The change-risk report showed the stream's fail-open branches untested: a
terminal claim that lands after its wait, a claim or a final publish that
throws, the settle timeout, the backlog limit, and a rejected touch. Each now
has a case. The shared content fold gets direct cases for a closed step, a
commentary phase and an applied steer.
2026-11-15 08:35:08 -05:00

1379 lines
48 KiB
JavaScript

const express = require('express');
const request = require('supertest');
const mockGenerationJobManager = {
getJob: jest.fn(),
subscribe: jest.fn(),
subscribeWithResume: jest.fn(),
getResumeState: jest.fn(),
markSyncSent: jest.fn(),
abortJob: jest.fn(),
getActiveJobIdsForUser: jest.fn().mockResolvedValue([]),
steering: {
claim: jest.fn().mockResolvedValue([]),
claimDetailed: jest.fn().mockResolvedValue({ generationProtocolVersion: 1, steers: [] }),
},
};
const mockCaptureAgentCheckpointGeneration = jest.fn();
const mockDeleteAgentCheckpoint = jest.fn();
const mockSaveMessage = jest.fn();
const mockLoadPlainAppConfig = jest.fn().mockResolvedValue({});
jest.mock('@librechat/data-schemas', () => ({
...jest.requireActual('@librechat/data-schemas'),
logger: {
debug: jest.fn(),
warn: jest.fn(),
error: jest.fn(),
info: jest.fn(),
},
}));
jest.mock('@librechat/api', () => ({
...jest.requireActual('@librechat/api'),
isEnabled: jest.fn().mockReturnValue(false),
GenerationJobManager: mockGenerationJobManager,
captureAgentCheckpointGeneration: (...args) => mockCaptureAgentCheckpointGeneration(...args),
deleteAgentCheckpoint: (...args) => mockDeleteAgentCheckpoint(...args),
}));
jest.mock('~/models', () => ({
initializeMessageBudget: jest.fn(),
initializeChannelStorageConfig: jest.fn(),
saveMessage: (...args) => mockSaveMessage(...args),
}));
let mockUserId = 'user-123';
let mockTenantId;
jest.mock('~/server/middleware', () => ({
uaParser: (req, res, next) => next(),
checkBan: (req, res, next) => next(),
requireJwtAuth: (req, res, next) => {
req.user = { id: mockUserId, tenantId: mockTenantId };
next();
},
moderateText: (req, res, next) => next(),
messageIpLimiter: (req, res, next) => next(),
configMiddleware: (req, res, next) => next(),
requireTermsAcceptance: (req, res, next) => next(),
loadStreamKeepaliveMs: (req) =>
jest
.requireActual('@librechat/api')
.createStreamKeepaliveLoader((...args) => mockLoadPlainAppConfig(...args))(req),
messageUserLimiter: (req, res, next) => next(),
}));
jest.mock('~/server/routes/agents/chat', () => require('express').Router());
jest.mock('~/server/routes/agents/v1', () => {
const router = require('express').Router();
router.use((req, res) => res.status(418).json({ error: 'v1 caught stream route' }));
return { v1: router };
});
jest.mock('~/server/routes/agents/openai', () => require('express').Router());
jest.mock('~/server/routes/agents/responses', () => require('express').Router());
jest.mock('~/server/routes/agents/management', () => require('express').Router());
const agentsRouter = require('../index');
const app = express();
app.use(express.json());
app.use('/agents', agentsRouter);
app.use((error, _req, res, _next) => res.status(500).json({ error: error.message }));
function mockSubscribeSuccess() {
mockGenerationJobManager.subscribe.mockImplementation((_streamId, _writeEvent, onDone) => {
process.nextTick(() => onDone({ done: true }));
return { unsubscribe: jest.fn() };
});
}
/** A real HTTP socket must reach subscribe before fake time can exercise its keepalive. */
function openSilentStream() {
let resolveAttached;
const subscribed = new Promise((resolve) => {
resolveAttached = resolve;
});
mockGenerationJobManager.subscribe.mockImplementation(async (_streamId, _writeEvent, onDone) => {
resolveAttached(onDone);
return { unsubscribe: jest.fn() };
});
const pending = request(app)
.get('/agents/chat/stream/stream-123')
.then((res) => res);
const attached = Promise.race([
subscribed,
pending.then((res) => {
throw new Error(`Stream ended before subscribing: HTTP ${res.status}`);
}),
]);
return { pending, attached };
}
describe('SSE stream tenant isolation', () => {
beforeEach(() => {
jest.clearAllMocks();
mockUserId = 'user-123';
mockTenantId = undefined;
mockCaptureAgentCheckpointGeneration.mockResolvedValue({
threadId: 'stream-123',
checkpointIds: ['checkpoint-a'],
});
mockDeleteAgentCheckpoint.mockResolvedValue(undefined);
mockSaveMessage.mockImplementation(async (_context, message) => ({
...message,
persisted: true,
}));
mockGenerationJobManager.getActiveJobIdsForUser.mockResolvedValue([]);
mockGenerationJobManager.steering.claim.mockResolvedValue([]);
mockGenerationJobManager.steering.claimDetailed.mockResolvedValue({
generationProtocolVersion: 1,
steers: [],
});
});
describe('GET /chat/stream/:streamId', () => {
it('returns 403 when a user from a different tenant accesses a stream', async () => {
mockUserId = 'user-456';
mockTenantId = 'tenant-b';
mockGenerationJobManager.getJob.mockResolvedValue({
metadata: { userId: 'user-456', tenantId: 'tenant-a' },
status: 'running',
});
const res = await request(app).get('/agents/chat/stream/stream-123');
expect(res.status).toBe(403);
expect(res.body.error).toBe('Unauthorized');
});
it('returns 404 when stream does not exist', async () => {
mockGenerationJobManager.getJob.mockResolvedValue(null);
const res = await request(app).get('/agents/chat/stream/nonexistent');
expect(res.status).toBe(404);
});
it('returns GENERATION_REPLACED before SSE headers when the requested epoch is stale', async () => {
mockGenerationJobManager.getJob.mockResolvedValue({
metadata: { userId: 'user-123' },
status: 'running',
createdAt: 2000,
});
const res = await request(app).get('/agents/chat/stream/stream-123?generationCreatedAt=1000');
expect(res.status).toBe(409);
expect(res.body).toEqual(expect.objectContaining({ code: 'GENERATION_REPLACED' }));
expect(res.headers['content-type']).not.toContain('text/event-stream');
expect(mockGenerationJobManager.subscribe).not.toHaveBeenCalled();
expect(mockGenerationJobManager.subscribeWithResume).not.toHaveBeenCalled();
});
it('passes the expected epoch into the manager subscription fence', async () => {
mockSubscribeSuccess();
mockGenerationJobManager.getJob.mockResolvedValue({
metadata: { userId: 'user-123' },
status: 'running',
createdAt: 1000,
});
const res = await request(app).get('/agents/chat/stream/stream-123?generationCreatedAt=1000');
expect(res.status).toBe(200);
expect(mockGenerationJobManager.subscribe).toHaveBeenCalledWith(
'stream-123',
expect.any(Function),
expect.any(Function),
expect.any(Function),
expect.objectContaining({ expectedCreatedAt: 1000 }),
);
});
it('pins an unfenced legacy subscription to the generation that passed authorization', async () => {
mockSubscribeSuccess();
mockGenerationJobManager.getJob.mockResolvedValue({
metadata: { userId: 'user-123', tenantId: 'tenant-a' },
status: 'running',
createdAt: 1000,
});
mockTenantId = 'tenant-a';
const res = await request(app).get('/agents/chat/stream/stream-123');
expect(res.status).toBe(200);
expect(mockGenerationJobManager.subscribe).toHaveBeenCalledWith(
'stream-123',
expect.any(Function),
expect.any(Function),
expect.any(Function),
expect.objectContaining({ expectedCreatedAt: 1000 }),
);
});
it('never attaches a cross-owner replacement after authorizing the predecessor', async () => {
mockGenerationJobManager.getJob
.mockResolvedValueOnce({
metadata: { userId: 'user-123', tenantId: 'tenant-a' },
status: 'running',
createdAt: 1000,
conversationId: 'stream-123',
})
.mockResolvedValueOnce({
metadata: { userId: 'other-user', tenantId: 'tenant-b' },
status: 'running',
createdAt: 2000,
conversationId: 'stream-123',
});
mockTenantId = 'tenant-a';
mockGenerationJobManager.subscribe.mockResolvedValue(null);
const res = await request(app).get('/agents/chat/stream/stream-123');
expect(res.status).toBe(200);
expect(mockGenerationJobManager.subscribe).toHaveBeenCalledWith(
'stream-123',
expect.any(Function),
expect.any(Function),
expect.any(Function),
expect.objectContaining({ expectedCreatedAt: 1000 }),
);
expect(res.text).toContain('event: error');
expect(res.text).not.toContain('generation_replaced');
});
it('fails closed when the job owner is missing', async () => {
mockGenerationJobManager.getJob.mockResolvedValue({
metadata: { tenantId: 'tenant-a' },
status: 'running',
createdAt: 1000,
});
mockTenantId = 'tenant-a';
const res = await request(app).get('/agents/chat/stream/stream-123');
expect(res.status).toBe(403);
expect(mockGenerationJobManager.subscribe).not.toHaveBeenCalled();
});
it('reconciles a replacement that wins after headers but before manager attachment', async () => {
mockGenerationJobManager.getJob
.mockResolvedValueOnce({
metadata: { userId: 'user-123', generationProtocolVersion: 2 },
status: 'running',
createdAt: 1000,
conversationId: 'stream-123',
})
.mockResolvedValueOnce({
metadata: { userId: 'user-123' },
status: 'running',
createdAt: 2000,
conversationId: 'stream-123',
});
mockGenerationJobManager.subscribe.mockResolvedValue(null);
const res = await request(app)
.get('/agents/chat/stream/stream-123?generationCreatedAt=1000&generationProtocolVersion=2')
.set('X-LibreChat-Generation-Protocol', '2');
const payload = JSON.parse(res.text.trim().split('\ndata: ')[1]);
expect(res.status).toBe(200);
expect(payload).toEqual({
final: true,
reconcile: true,
reconcileReason: 'generation_replaced',
generationCreatedAt: 1000,
conversation: { conversationId: 'stream-123' },
generationProtocolVersion: 2,
});
});
it('reconciles a generation deleted in the snapshot-to-attach window', async () => {
mockGenerationJobManager.getJob
.mockResolvedValueOnce({
metadata: { userId: 'user-123', generationProtocolVersion: 2 },
status: 'running',
createdAt: 1000,
conversationId: 'stream-123',
})
.mockResolvedValueOnce(null);
mockGenerationJobManager.subscribe.mockResolvedValue(null);
const res = await request(app)
.get('/agents/chat/stream/stream-123?generationCreatedAt=1000&generationProtocolVersion=2')
.set('X-LibreChat-Generation-Protocol', '2');
const payload = JSON.parse(res.text.trim().split('\ndata: ')[1]);
expect(res.status).toBe(200);
expect(payload).toEqual({
final: true,
reconcile: true,
reconcileReason: 'terminal_payload_missing',
generationCreatedAt: 1000,
conversation: { conversationId: 'stream-123' },
generationProtocolVersion: 2,
});
});
it('translates reconciliation into an error for a v1 subscriber', async () => {
mockGenerationJobManager.getJob
.mockResolvedValueOnce({
metadata: { userId: 'user-123', generationProtocolVersion: 1 },
status: 'running',
createdAt: 1000,
conversationId: 'stream-123',
})
.mockResolvedValueOnce(null);
mockGenerationJobManager.subscribe.mockResolvedValue(null);
const res = await request(app).get('/agents/chat/stream/stream-123?generationCreatedAt=1000');
expect(res.status).toBe(200);
expect(res.headers['x-librechat-generation-protocol']).toBe('1');
expect(res.text).toContain('event: error');
expect(res.text).toContain('Generation state changed');
expect(res.text).not.toContain('"reconcile":true');
});
it('does not misclassify a failed reconciliation read as a missing terminal job', async () => {
mockGenerationJobManager.getJob
.mockResolvedValueOnce({
metadata: { userId: 'user-123' },
status: 'running',
createdAt: 1000,
conversationId: 'stream-123',
})
.mockRejectedValueOnce(new Error('store unavailable'));
mockGenerationJobManager.subscribe.mockResolvedValue(null);
const res = await request(app).get('/agents/chat/stream/stream-123?generationCreatedAt=1000');
expect(res.status).toBe(200);
expect(res.text).toContain('event: error');
expect(res.text).toContain('Failed to subscribe to stream');
expect(res.text).not.toContain('terminal_payload_missing');
});
it('proceeds past tenant guard when tenant matches', async () => {
mockUserId = 'user-123';
mockTenantId = 'tenant-a';
mockSubscribeSuccess();
mockGenerationJobManager.getJob.mockResolvedValue({
metadata: { userId: 'user-123', tenantId: 'tenant-a' },
status: 'running',
createdAt: 1000,
});
const res = await request(app).get('/agents/chat/stream/stream-123');
expect(res.status).toBe(200);
expect(mockGenerationJobManager.subscribe).toHaveBeenCalledTimes(1);
});
it('proceeds past tenant guard when job has no tenantId (single-tenant mode)', async () => {
mockUserId = 'user-123';
mockTenantId = undefined;
mockSubscribeSuccess();
mockGenerationJobManager.getJob.mockResolvedValue({
metadata: { userId: 'user-123' },
status: 'running',
createdAt: 1000,
});
const res = await request(app).get('/agents/chat/stream/stream-123');
expect(res.status).toBe(200);
expect(mockGenerationJobManager.subscribe).toHaveBeenCalledTimes(1);
});
it('writes the resume sync frame before activating live delivery', async () => {
mockGenerationJobManager.getJob.mockResolvedValue({
metadata: { userId: 'user-123' },
status: 'running',
createdAt: 1000,
});
const activate = jest.fn();
mockGenerationJobManager.subscribeWithResume.mockImplementation(
async (_streamId, writeEvent, onDone) => {
activate.mockImplementation(() => {
writeEvent({ event: 'on_message_delta', data: { text: 'live' } });
onDone({ final: true });
});
return {
subscription: { unsubscribe: jest.fn(), activate },
resumeState: { runSteps: [], aggregatedContent: [] },
pendingEvents: [],
};
},
);
const res = await request(app).get('/agents/chat/stream/stream-123?resume=true');
const payloads = res.text
.trim()
.split('\n\n')
.map((frame) => JSON.parse(frame.split('\ndata: ')[1]));
expect(res.status).toBe(200);
expect(payloads).toEqual([
{
sync: true,
resumeState: { runSteps: [], aggregatedContent: [] },
pendingEvents: [],
},
{ event: 'on_message_delta', data: { text: 'live' } },
{ final: true, generationProtocolVersion: 1 },
]);
expect(activate).toHaveBeenCalledTimes(1);
expect(mockGenerationJobManager.markSyncSent).toHaveBeenCalledWith('stream-123', 1000);
});
it('keeps a silent stream alive with SSE comment frames', async () => {
jest.useFakeTimers({ doNotFake: ['nextTick', 'setImmediate'] });
try {
mockGenerationJobManager.getJob.mockResolvedValue({
metadata: { userId: 'user-123' },
status: 'running',
createdAt: 1000,
});
const { pending, attached } = openSilentStream();
const finish = await attached;
await jest.advanceTimersByTimeAsync(60_000);
finish({ final: true });
const res = await pending;
expect(res.status).toBe(200);
expect(res.text.startsWith(':\n\n')).toBe(true);
} finally {
jest.useRealTimers();
}
});
it('does not load the app config for a stream it refuses', async () => {
mockGenerationJobManager.getJob.mockResolvedValue(null);
const res = await request(app).get('/agents/chat/stream/stream-123');
expect(res.status).toBe(404);
expect(mockLoadPlainAppConfig).not.toHaveBeenCalled();
});
it('reads the keepalive interval without runtime config augmentation', async () => {
mockSubscribeSuccess();
mockGenerationJobManager.getJob.mockResolvedValue({
metadata: { userId: 'user-123' },
status: 'running',
createdAt: 1000,
});
const res = await request(app).get('/agents/chat/stream/stream-123');
expect(res.status).toBe(200);
expect(mockLoadPlainAppConfig).toHaveBeenCalledTimes(1);
expect(mockLoadPlainAppConfig).toHaveBeenCalledWith(
expect.objectContaining({ userId: 'user-123', skipRuntimeAugmentation: true }),
);
});
it('keeps the default keepalive when the config read fails', async () => {
jest.useFakeTimers({ doNotFake: ['nextTick', 'setImmediate'] });
mockLoadPlainAppConfig.mockRejectedValueOnce(new Error('config store unavailable'));
try {
mockGenerationJobManager.getJob.mockResolvedValue({
metadata: { userId: 'user-123' },
status: 'running',
createdAt: 1000,
});
const { pending, attached } = openSilentStream();
const finish = await attached;
await jest.advanceTimersByTimeAsync(60_000);
finish({ final: true });
const res = await pending;
expect(res.text.startsWith(':\n\n')).toBe(true);
} finally {
jest.useRealTimers();
}
});
it('subscribes and keeps the stream alive while the app config never loads', async () => {
jest.useFakeTimers({ doNotFake: ['nextTick', 'setImmediate'] });
mockLoadPlainAppConfig.mockReturnValueOnce(new Promise(() => undefined));
try {
mockGenerationJobManager.getJob.mockResolvedValue({
metadata: { userId: 'user-123' },
status: 'running',
createdAt: 1000,
});
const { pending, attached } = openSilentStream();
const finish = await attached;
await jest.advanceTimersByTimeAsync(60_000);
expect(mockGenerationJobManager.subscribe).toHaveBeenCalled();
finish({ final: true });
const res = await pending;
expect(res.text.startsWith(':\n\n')).toBe(true);
} finally {
jest.useRealTimers();
}
});
it('detaches a paused resume subscription when the response ends before activation', async () => {
mockGenerationJobManager.getJob.mockResolvedValue({
metadata: { userId: 'user-123' },
status: 'running',
createdAt: 1000,
});
const subscription = {
unsubscribe: jest.fn(),
activate: jest.fn(),
};
mockGenerationJobManager.subscribeWithResume.mockImplementation(
async (_streamId, _writeEvent, onDone) => {
onDone({ final: true });
return {
subscription,
resumeState: { runSteps: [], aggregatedContent: [] },
pendingEvents: [],
};
},
);
const res = await request(app).get('/agents/chat/stream/stream-123?resume=true');
expect(res.status).toBe(200);
expect(subscription.unsubscribe).toHaveBeenCalled();
expect(subscription.activate).not.toHaveBeenCalled();
});
it('returns 403 when job has tenantId but user has no tenantId', async () => {
mockUserId = 'user-123';
mockTenantId = undefined;
mockGenerationJobManager.getJob.mockResolvedValue({
metadata: { userId: 'user-123', tenantId: 'some-tenant' },
status: 'running',
});
const res = await request(app).get('/agents/chat/stream/stream-123');
expect(res.status).toBe(403);
});
});
describe('GET /chat/status/:conversationId', () => {
it('echoes v2 for a clean jobless status after the server rollout gate', async () => {
mockGenerationJobManager.getJob.mockResolvedValue(null);
mockGenerationJobManager.steering.claimDetailed.mockResolvedValue({
generationProtocolVersion: 2,
steers: [],
});
const res = await request(app)
.get('/agents/chat/status/conv-123?generationProtocolVersion=2')
.set('X-LibreChat-Generation-Protocol', '2');
expect(res.status).toBe(200);
expect(res.headers['x-librechat-generation-protocol']).toBe('2');
expect(res.body).toEqual({ active: false, generationProtocolVersion: 2 });
expect(mockGenerationJobManager.steering.claimDetailed).toHaveBeenCalledWith(
'conv-123',
{ userId: 'user-123', tenantId: undefined },
2,
);
});
it('downgrades a jobless v2 request when its parked payload is legacy', async () => {
mockGenerationJobManager.getJob.mockResolvedValue(null);
mockGenerationJobManager.steering.claimDetailed.mockResolvedValue({
generationProtocolVersion: 1,
steers: [{ steerId: 'legacy-steer', text: 'legacy words' }],
});
const res = await request(app)
.get('/agents/chat/status/conv-123?generationProtocolVersion=2')
.set('X-LibreChat-Generation-Protocol', '2');
expect(res.status).toBe(200);
expect(res.headers['x-librechat-generation-protocol']).toBe('1');
expect(res.body).toEqual({
active: false,
generationProtocolVersion: 1,
unrecoveredSteers: [{ steerId: 'legacy-steer', text: 'legacy words' }],
});
});
it('returns 403 when tenant does not match', async () => {
mockUserId = 'user-123';
mockTenantId = 'tenant-b';
mockGenerationJobManager.getJob.mockResolvedValue({
metadata: { userId: 'user-123', tenantId: 'tenant-a' },
status: 'running',
});
const res = await request(app).get('/agents/chat/status/conv-123');
expect(res.status).toBe(403);
expect(res.body.error).toBe('Unauthorized');
});
it('returns status when tenant matches', async () => {
mockUserId = 'user-123';
mockTenantId = 'tenant-a';
mockGenerationJobManager.getJob.mockResolvedValue({
metadata: { userId: 'user-123', tenantId: 'tenant-a' },
status: 'running',
createdAt: Date.now(),
});
mockGenerationJobManager.getResumeState.mockResolvedValue(null);
const res = await request(app).get('/agents/chat/status/conv-123');
expect(res.status).toBe(200);
expect(res.body.active).toBe(true);
});
it.each([true, false])(
'reports the run temporary state %s recorded at admission',
async (isTemporary) => {
mockUserId = 'user-123';
mockTenantId = 'tenant-a';
mockGenerationJobManager.getJob.mockResolvedValue({
metadata: { userId: 'user-123', tenantId: 'tenant-a', isTemporary },
status: 'running',
createdAt: Date.now(),
});
mockGenerationJobManager.getResumeState.mockResolvedValue(null);
const res = await request(app).get('/agents/chat/status/conv-123');
expect(res.status).toBe(200);
expect(res.body.isTemporary).toBe(isTemporary);
},
);
it('preserves the immutable v2 marker on an active status response', async () => {
mockGenerationJobManager.getJob.mockResolvedValue({
metadata: { userId: 'user-123', generationProtocolVersion: 2 },
status: 'running',
createdAt: 1000,
});
mockGenerationJobManager.getResumeState.mockResolvedValue(null);
const res = await request(app)
.get('/agents/chat/status/conv-123?generationProtocolVersion=2')
.set('X-LibreChat-Generation-Protocol', '2');
expect(res.status).toBe(200);
expect(res.headers['x-librechat-generation-protocol']).toBe('2');
expect(res.body).toEqual(
expect.objectContaining({
active: true,
streamId: 'conv-123',
generationProtocolVersion: 2,
}),
);
});
it('discards resume content when a cross-owner replacement wins after authorization', async () => {
mockUserId = 'user-123';
mockTenantId = 'tenant-a';
mockGenerationJobManager.getJob
.mockResolvedValueOnce({
metadata: { userId: 'user-123', tenantId: 'tenant-a' },
status: 'running',
createdAt: 1000,
})
.mockResolvedValueOnce({
metadata: { userId: 'other-user', tenantId: 'tenant-b' },
status: 'running',
createdAt: 2000,
});
mockGenerationJobManager.getResumeState.mockResolvedValue({
aggregatedContent: [{ text: 'replacement secret' }],
});
const res = await request(app).get('/agents/chat/status/conv-123');
expect(res.status).toBe(403);
expect(res.text).not.toContain('replacement secret');
});
it('retries a same-owner replacement snapshot and returns only its verified state', async () => {
mockUserId = 'user-123';
mockTenantId = 'tenant-a';
const first = {
metadata: { userId: 'user-123', tenantId: 'tenant-a' },
status: 'running',
createdAt: 1000,
};
const replacement = {
metadata: { userId: 'user-123', tenantId: 'tenant-a' },
status: 'running',
createdAt: 2000,
};
mockGenerationJobManager.getJob
.mockResolvedValueOnce(first)
.mockResolvedValueOnce(replacement)
.mockResolvedValueOnce(replacement);
mockGenerationJobManager.getResumeState
.mockResolvedValueOnce({ aggregatedContent: [{ text: 'discarded mixed snapshot' }] })
.mockResolvedValueOnce({ aggregatedContent: [{ text: 'verified replacement' }] });
const res = await request(app).get('/agents/chat/status/conv-123');
expect(res.status).toBe(200);
expect(res.body.createdAt).toBe(2000);
expect(res.body.aggregatedContent).toEqual([{ text: 'verified replacement' }]);
expect(res.text).not.toContain('discarded mixed snapshot');
expect(mockGenerationJobManager.getResumeState).toHaveBeenCalledTimes(2);
});
it('fails closed when status snapshots keep changing', async () => {
mockUserId = 'user-123';
mockGenerationJobManager.getJob
.mockResolvedValueOnce({
metadata: { userId: 'user-123' },
status: 'running',
createdAt: 1,
})
.mockResolvedValueOnce({
metadata: { userId: 'user-123' },
status: 'running',
createdAt: 2,
})
.mockResolvedValueOnce({
metadata: { userId: 'user-123' },
status: 'running',
createdAt: 3,
})
.mockResolvedValueOnce({
metadata: { userId: 'user-123' },
status: 'running',
createdAt: 4,
});
mockGenerationJobManager.getResumeState.mockResolvedValue({
aggregatedContent: [{ text: 'never verified' }],
});
const res = await request(app).get('/agents/chat/status/conv-123');
expect(res.status).toBe(503);
expect(res.body).toEqual({ code: 'SERVER_NOT_READY', generationProtocolVersion: 1 });
expect(res.text).not.toContain('never verified');
});
it('keeps an abort persistence-pending terminal snapshot on the readiness path', async () => {
const pendingAbort = {
metadata: {
userId: 'user-123',
terminalPersistencePending: true,
},
status: 'aborted',
createdAt: 1000,
};
mockGenerationJobManager.getJob.mockResolvedValue(pendingAbort);
mockGenerationJobManager.getResumeState.mockResolvedValue({
aggregatedContent: [{ text: 'not authoritative yet' }],
});
const res = await request(app).get('/agents/chat/status/conv-123');
expect(res.status).toBe(503);
expect(res.headers['retry-after']).toBe('1');
expect(res.body).toEqual({ code: 'SERVER_NOT_READY', generationProtocolVersion: 1 });
expect(res.text).not.toContain('not authoritative yet');
expect(mockGenerationJobManager.steering.claim).not.toHaveBeenCalled();
expect(mockGenerationJobManager.steering.claimDetailed).not.toHaveBeenCalled();
});
it('fails closed when the status job owner is missing', async () => {
mockGenerationJobManager.getJob.mockResolvedValue({
metadata: {},
status: 'running',
createdAt: 1000,
});
const res = await request(app).get('/agents/chat/status/conv-123');
expect(res.status).toBe(403);
expect(mockGenerationJobManager.getResumeState).not.toHaveBeenCalled();
});
});
describe('POST /chat/abort', () => {
it.each([
[{ streamId: { malicious: true } }],
[{ conversationId: 123 }],
[{ abortKey: { split: 'not a function' } }],
[{ streamId: '' }],
])('rejects malformed abort target fields without touching the store', async (body) => {
const res = await request(app).post('/agents/chat/abort').send(body);
expect(res.status).toBe(400);
expect(res.body).toEqual({ code: 'INVALID_ABORT_TARGET', generationProtocolVersion: 1 });
expect(mockGenerationJobManager.getJob).not.toHaveBeenCalled();
expect(mockGenerationJobManager.abortJob).not.toHaveBeenCalled();
});
it('returns 403 when tenant does not match', async () => {
mockUserId = 'user-123';
mockTenantId = 'tenant-b';
mockGenerationJobManager.getJob.mockResolvedValue({
metadata: { userId: 'user-123', tenantId: 'tenant-a' },
status: 'running',
});
const res = await request(app).post('/agents/chat/abort').send({ streamId: 'stream-123' });
expect(res.status).toBe(403);
expect(res.body.error).toBe('Unauthorized');
});
it('fails closed when the stored job owner is missing', async () => {
mockGenerationJobManager.getJob.mockResolvedValue({
metadata: {},
status: 'running',
createdAt: 1000,
});
const res = await request(app).post('/agents/chat/abort').send({
streamId: 'stream-123',
generationCreatedAt: 1000,
});
expect(res.status).toBe(403);
expect(mockGenerationJobManager.abortJob).not.toHaveBeenCalled();
});
it('does not fall back from an unknown concrete id to an unrelated active run', async () => {
mockGenerationJobManager.getJob.mockResolvedValue(null);
mockGenerationJobManager.getActiveJobIdsForUser.mockResolvedValue(['other-active']);
const res = await request(app).post('/agents/chat/abort').send({
streamId: 'stale-or-typo-id',
});
expect(res.status).toBe(404);
expect(mockGenerationJobManager.getActiveJobIdsForUser).not.toHaveBeenCalled();
expect(mockGenerationJobManager.abortJob).not.toHaveBeenCalled();
});
it('uses the generation epoch to resolve the exact new-placeholder run', async () => {
mockGenerationJobManager.getActiveJobIdsForUser.mockResolvedValue(['run-a', 'run-b']);
mockGenerationJobManager.getJob.mockImplementation(async (streamId) => ({
metadata: { userId: 'user-123' },
status: 'running',
createdAt: streamId === 'run-a' ? 1000 : 2000,
}));
mockGenerationJobManager.abortJob.mockResolvedValue({ success: true });
const res = await request(app).post('/agents/chat/abort').send({
conversationId: 'new',
generationCreatedAt: 2000,
});
expect(res.status).toBe(200);
expect(res.body.aborted).toBe('run-b');
expect(mockGenerationJobManager.abortJob).toHaveBeenCalledWith(
'run-b',
expect.objectContaining({ expectedCreatedAt: 2000 }),
);
});
it('refuses an ambiguous unfenced new-placeholder abort', async () => {
mockGenerationJobManager.getActiveJobIdsForUser.mockResolvedValue(['run-a', 'run-b']);
mockGenerationJobManager.getJob.mockImplementation(async (streamId) => ({
metadata: { userId: 'user-123' },
status: 'running',
createdAt: streamId === 'run-a' ? 1000 : 2000,
}));
const res = await request(app).post('/agents/chat/abort').send({ conversationId: 'new' });
expect(res.status).toBe(409);
expect(res.body).toEqual({
code: 'AMBIGUOUS_ACTIVE_RUN',
generationProtocolVersion: 1,
});
expect(mockGenerationJobManager.abortJob).not.toHaveBeenCalled();
});
it('409s a stale generation identity without aborting the replacement', async () => {
mockGenerationJobManager.getJob.mockResolvedValue({
metadata: { userId: 'user-123' },
status: 'running',
createdAt: 2000,
});
const res = await request(app).post('/agents/chat/abort').send({
streamId: 'stream-123',
generationCreatedAt: 1000,
});
expect(res.status).toBe(409);
expect(res.body).toEqual({ code: 'RUN_REPLACED', generationProtocolVersion: 1 });
expect(mockGenerationJobManager.abortJob).not.toHaveBeenCalled();
});
it('threads the authenticated epoch into abortJob and reports a manager-side replacement', async () => {
mockGenerationJobManager.getJob.mockResolvedValue({
metadata: { userId: 'user-123' },
status: 'running',
createdAt: 1000,
});
mockGenerationJobManager.abortJob.mockResolvedValue({
success: false,
failureReason: 'generation_replaced',
});
const res = await request(app).post('/agents/chat/abort').send({
streamId: 'stream-123',
generationCreatedAt: 1000,
});
expect(res.status).toBe(409);
expect(res.body).toEqual({ code: 'RUN_REPLACED', generationProtocolVersion: 1 });
expect(mockGenerationJobManager.abortJob).toHaveBeenCalledWith(
'stream-123',
expect.objectContaining({ expectedCreatedAt: 1000 }),
);
});
it('does not report success when the same generation remains active after an abort race', async () => {
mockGenerationJobManager.getJob.mockResolvedValue({
metadata: { userId: 'user-123' },
status: 'running',
createdAt: 1000,
});
mockGenerationJobManager.abortJob.mockResolvedValue({
success: false,
failureReason: 'job_still_active',
});
const res = await request(app).post('/agents/chat/abort').send({
streamId: 'stream-123',
generationCreatedAt: 1000,
});
expect(res.status).toBe(409);
expect(res.headers['retry-after']).toBe('1');
expect(res.body).toEqual({ code: 'RUN_STILL_ACTIVE', generationProtocolVersion: 1 });
});
it('reports an already-settled generation when natural completion wins the abort CAS', async () => {
/** Read by the child-job guard, by the route, and again after the abort lost its CAS. */
mockGenerationJobManager.getJob
.mockResolvedValueOnce({
metadata: { userId: 'user-123', generationProtocolVersion: 2 },
status: 'running',
createdAt: 1000,
})
.mockResolvedValueOnce({
metadata: { userId: 'user-123', generationProtocolVersion: 2 },
status: 'running',
createdAt: 1000,
})
.mockResolvedValueOnce({
metadata: { userId: 'user-123', generationProtocolVersion: 2 },
status: 'complete',
createdAt: 1000,
});
mockGenerationJobManager.abortJob.mockResolvedValue({
success: false,
jobData: { status: 'running', createdAt: 1000 },
});
const res = await request(app)
.post('/agents/chat/abort')
.set('X-LibreChat-Generation-Protocol', '2')
.send({
streamId: 'stream-123',
generationCreatedAt: 1000,
generationProtocolVersion: 2,
});
expect(res.status).toBe(200);
expect(res.body).toEqual({
success: false,
settled: true,
code: 'RUN_ALREADY_SETTLED',
streamId: 'stream-123',
terminalStatus: 'complete',
generationProtocolVersion: 2,
});
});
it('keeps the legacy abort ACK shape when natural completion wins for protocol v1', async () => {
/** Read by the child-job guard, by the route, and again after the abort lost its CAS. */
mockGenerationJobManager.getJob
.mockResolvedValueOnce({
metadata: { userId: 'user-123', generationProtocolVersion: 1 },
status: 'running',
createdAt: 1000,
})
.mockResolvedValueOnce({
metadata: { userId: 'user-123', generationProtocolVersion: 1 },
status: 'running',
createdAt: 1000,
})
.mockResolvedValueOnce({
metadata: { userId: 'user-123', generationProtocolVersion: 1 },
status: 'complete',
createdAt: 1000,
});
mockGenerationJobManager.abortJob.mockResolvedValue({
success: false,
jobData: { status: 'running', createdAt: 1000 },
});
const res = await request(app).post('/agents/chat/abort').send({
streamId: 'stream-123',
generationCreatedAt: 1000,
});
expect(res.status).toBe(200);
expect(res.body).toEqual({
success: true,
aborted: 'stream-123',
generationProtocolVersion: 1,
});
});
it('returns 404 when the job disappears before the manager can observe it', async () => {
mockGenerationJobManager.getJob.mockResolvedValue({
metadata: { userId: 'user-123' },
status: 'running',
createdAt: 1000,
});
mockGenerationJobManager.abortJob.mockResolvedValue({
success: false,
jobData: null,
});
const res = await request(app).post('/agents/chat/abort').send({
streamId: 'stream-123',
generationCreatedAt: 1000,
});
expect(res.status).toBe(404);
expect(res.body).toEqual({
success: false,
error: 'Job not found',
streamId: 'stream-123',
generationProtocolVersion: 1,
});
});
it('leaves the run untouched when the required checkpoint snapshot cannot be captured', async () => {
mockGenerationJobManager.getJob.mockResolvedValue({
metadata: {
userId: 'user-123',
pendingAction: { payload: { type: 'tool_approval' } },
},
status: 'requires_action',
createdAt: 1000,
});
mockCaptureAgentCheckpointGeneration.mockRejectedValue(
new Error('checkpoint snapshot unavailable'),
);
const res = await request(app).post('/agents/chat/abort').send({
streamId: 'stream-123',
generationCreatedAt: 1000,
});
expect(res.status).toBe(500);
expect(res.body).toEqual({
code: 'ABORT_FAILED',
error: 'Failed to abort generation',
generationProtocolVersion: 1,
});
expect(mockCaptureAgentCheckpointGeneration).toHaveBeenCalledWith('stream-123', undefined, {
throwOnError: true,
});
expect(mockGenerationJobManager.abortJob).not.toHaveBeenCalled();
});
it('persists the aborted parent and prunes the captured checkpoint before normal FINAL', async () => {
const publishFinal = jest.fn();
mockGenerationJobManager.getJob.mockResolvedValue({
metadata: {
userId: 'user-123',
pendingAction: { payload: { type: 'tool_approval' } },
},
status: 'requires_action',
createdAt: 1000,
});
mockGenerationJobManager.abortJob.mockImplementation(async (_streamId, options) => {
const abortResult = {
success: true,
jobData: {
createdAt: 1000,
conversationId: 'stream-123',
responseMessageId: 'response-1',
userMessage: { messageId: 'user-1' },
sender: 'Agent',
endpoint: 'agents',
},
content: [{ type: 'text', text: 'partial answer' }],
text: 'partial answer',
pendingSteers: [{ steerId: 'steer-1', text: 'next' }],
};
await options.beforePublish(abortResult);
publishFinal();
return abortResult;
});
const res = await request(app).post('/agents/chat/abort').send({
streamId: 'stream-123',
generationCreatedAt: 1000,
});
expect(res.status).toBe(200);
expect(mockCaptureAgentCheckpointGeneration).toHaveBeenCalledWith('stream-123', undefined, {
throwOnError: true,
});
expect(mockSaveMessage).toHaveBeenCalledTimes(2);
expect(mockSaveMessage).toHaveBeenNthCalledWith(
1,
expect.any(Object),
expect.objectContaining({ messageId: 'user-1', isCreatedByUser: true }),
expect.objectContaining({ insertOnly: true }),
);
expect(mockSaveMessage).toHaveBeenNthCalledWith(
2,
expect.any(Object),
expect.objectContaining({ messageId: 'response-1', isCreatedByUser: false }),
expect.any(Object),
);
expect(mockDeleteAgentCheckpoint).toHaveBeenCalledWith(
'stream-123',
undefined,
{ threadId: 'stream-123', checkpointIds: ['checkpoint-a'] },
{ throwOnError: true },
);
expect(mockSaveMessage.mock.invocationCallOrder[0]).toBeLessThan(
publishFinal.mock.invocationCallOrder[0],
);
expect(mockDeleteAgentCheckpoint.mock.invocationCallOrder[0]).toBeLessThan(
publishFinal.mock.invocationCallOrder[0],
);
expect(mockSaveMessage.mock.invocationCallOrder[1]).toBeLessThan(
mockDeleteAgentCheckpoint.mock.invocationCallOrder[0],
);
expect(mockSaveMessage.mock.invocationCallOrder[0]).toBeLessThan(
mockSaveMessage.mock.invocationCallOrder[1],
);
expect(res.body.pendingSteers).toEqual([{ steerId: 'steer-1', text: 'next' }]);
});
it('prunes the whole immutable generation namespace without a racy id snapshot', async () => {
mockGenerationJobManager.getJob.mockResolvedValue({
metadata: {
userId: 'user-123',
checkpointNamespace: 'generation-1000',
pendingAction: { payload: { type: 'tool_approval' } },
},
status: 'requires_action',
createdAt: 1000,
});
mockGenerationJobManager.abortJob.mockImplementation(async (_streamId, options) => {
const abortResult = {
success: true,
jobData: { createdAt: 1000, conversationId: 'stream-123' },
content: [],
text: '',
pendingSteers: [],
};
await options.beforePublish(abortResult);
return abortResult;
});
const res = await request(app).post('/agents/chat/abort').send({
streamId: 'stream-123',
generationCreatedAt: 1000,
});
expect(res.status).toBe(200);
expect(mockCaptureAgentCheckpointGeneration).not.toHaveBeenCalled();
expect(mockDeleteAgentCheckpoint).toHaveBeenCalledWith('stream-123', undefined, undefined, {
throwOnError: true,
checkpointNamespace: 'generation-1000',
});
});
it('suppresses interrupt-drain payloads when required abort persistence fails', async () => {
mockGenerationJobManager.getJob.mockResolvedValue({
metadata: { userId: 'user-123', generationProtocolVersion: 2 },
status: 'running',
createdAt: 1000,
});
mockGenerationJobManager.abortJob.mockResolvedValue({
success: true,
persistenceFailed: true,
pendingSteers: [{ steerId: 'steer-1', text: 'do not auto-send' }],
});
const res = await request(app)
.post('/agents/chat/abort')
.set('X-LibreChat-Generation-Protocol', '2')
.send({
streamId: 'stream-123',
generationCreatedAt: 1000,
generationProtocolVersion: 2,
});
expect(res.status).toBe(200);
expect(res.body).toEqual({
success: true,
aborted: 'stream-123',
persistenceFailed: true,
generationProtocolVersion: 2,
});
});
it('fails closed instead of acknowledging abort persistence failure to a v1 client', async () => {
mockGenerationJobManager.getJob.mockResolvedValue({
metadata: { userId: 'user-123', generationProtocolVersion: 1 },
status: 'running',
createdAt: 1000,
});
mockGenerationJobManager.abortJob.mockResolvedValue({
success: true,
persistenceFailed: true,
pendingSteers: [{ steerId: 'steer-1', text: 'do not expose' }],
});
const res = await request(app).post('/agents/chat/abort').send({
streamId: 'stream-123',
generationCreatedAt: 1000,
});
expect(res.status).toBe(409);
expect(res.headers['retry-after']).toBe('1');
expect(res.body).toEqual({
code: 'ABORT_PERSISTENCE_FAILED',
generationProtocolVersion: 1,
});
expect(res.text).not.toContain('steer-1');
});
it('still prunes the captured checkpoint when the aborted-message save fails', async () => {
mockGenerationJobManager.getJob.mockResolvedValue({
metadata: {
userId: 'user-123',
generationProtocolVersion: 2,
pendingAction: { payload: { type: 'tool_approval' } },
},
status: 'requires_action',
createdAt: 1000,
});
mockSaveMessage
.mockResolvedValueOnce({ messageId: 'user-1' })
.mockRejectedValueOnce(new Error('message store unavailable'));
mockGenerationJobManager.abortJob.mockImplementation(async (_streamId, options) => {
const abortResult = {
success: true,
jobData: {
createdAt: 1000,
conversationId: 'stream-123',
responseMessageId: 'response-1',
userMessage: { messageId: 'user-1' },
},
content: [{ type: 'text', text: 'partial answer' }],
text: 'partial answer',
pendingSteers: [{ steerId: 'steer-1', text: 'do not drain' }],
};
try {
await options.beforePublish(abortResult);
return abortResult;
} catch (_error) {
return { ...abortResult, persistenceFailed: true };
}
});
const res = await request(app)
.post('/agents/chat/abort')
.set('X-LibreChat-Generation-Protocol', '2')
.send({
streamId: 'stream-123',
generationCreatedAt: 1000,
generationProtocolVersion: 2,
});
expect(res.status).toBe(200);
expect(mockSaveMessage).toHaveBeenCalledTimes(2);
expect(mockDeleteAgentCheckpoint).toHaveBeenCalledWith(
'stream-123',
undefined,
{ threadId: 'stream-123', checkpointIds: ['checkpoint-a'] },
{ throwOnError: true },
);
expect(res.body).toEqual({
success: true,
aborted: 'stream-123',
persistenceFailed: true,
generationProtocolVersion: 2,
});
});
it('keeps checkpoint-prune failure on the conservative reconciliation path', async () => {
const publishNormalFinal = jest.fn();
const publishReconcile = jest.fn();
mockGenerationJobManager.getJob.mockResolvedValue({
metadata: {
userId: 'user-123',
generationProtocolVersion: 2,
pendingAction: { payload: { type: 'tool_approval' } },
},
status: 'requires_action',
createdAt: 1000,
});
mockDeleteAgentCheckpoint.mockRejectedValue(new Error('checkpoint store unavailable'));
mockGenerationJobManager.abortJob.mockImplementation(async (_streamId, options) => {
const abortResult = {
success: true,
jobData: {
createdAt: 1000,
conversationId: 'stream-123',
userMessage: { messageId: 'user-1' },
},
content: [],
text: '',
pendingSteers: [{ steerId: 'steer-1', text: 'do not drain' }],
};
try {
await options.beforePublish(abortResult);
publishNormalFinal();
return abortResult;
} catch (_error) {
publishReconcile();
return { ...abortResult, persistenceFailed: true };
}
});
const res = await request(app)
.post('/agents/chat/abort')
.set('X-LibreChat-Generation-Protocol', '2')
.send({
streamId: 'stream-123',
generationCreatedAt: 1000,
generationProtocolVersion: 2,
});
expect(res.status).toBe(200);
expect(publishNormalFinal).not.toHaveBeenCalled();
expect(publishReconcile).toHaveBeenCalledTimes(1);
expect(mockDeleteAgentCheckpoint).toHaveBeenCalledWith(
'stream-123',
undefined,
{ threadId: 'stream-123', checkpointIds: ['checkpoint-a'] },
{ throwOnError: true },
);
expect(res.body).toEqual({
success: true,
aborted: 'stream-123',
persistenceFailed: true,
generationProtocolVersion: 2,
});
});
});
});