mirror of
https://github.com/danny-avila/LibreChat.git
synced 2026-11-18 14:08:21 +00:00
Some checks failed
Backend Unit Tests / Build packages (push) Waiting to run
Backend Unit Tests / Codegraph select (push) Waiting to run
Backend Unit Tests / TypeScript type checks (push) Blocked by required conditions
Backend Unit Tests / Circular dependency checks (push) Waiting to run
Backend Unit Tests / Tests: api (shard 1/3) (push) Blocked by required conditions
Backend Unit Tests / Tests: api (shard 2/3) (push) Blocked by required conditions
Backend Unit Tests / Tests: api (shard 3/3) (push) Blocked by required conditions
Backend Unit Tests / Tests: data-provider (push) Blocked by required conditions
Backend Unit Tests / Tests: data-schemas (push) Blocked by required conditions
Backend Unit Tests / Tests: @librechat/api (shard 1/4) (push) Blocked by required conditions
Backend Unit Tests / Tests: @librechat/api (shard 2/4) (push) Blocked by required conditions
Backend Unit Tests / Tests: @librechat/api (shard 3/4) (push) Blocked by required conditions
Backend Unit Tests / Tests: @librechat/api (shard 4/4) (push) Blocked by required conditions
Channel Gateway / gateway (push) Waiting to run
Codegraph E2E Vote Gate / Gate (push) Waiting to run
Codegraph E2E Votes / vote (shard 1/10) (push) Waiting to run
Codegraph E2E Votes / vote (shard 10/10) (push) Waiting to run
Codegraph E2E Votes / vote (shard 2/10) (push) Waiting to run
Codegraph E2E Votes / vote (shard 3/10) (push) Waiting to run
Codegraph E2E Votes / vote (shard 4/10) (push) Waiting to run
Codegraph E2E Votes / vote (shard 5/10) (push) Waiting to run
Codegraph E2E Votes / vote (shard 6/10) (push) Waiting to run
Codegraph E2E Votes / vote (shard 7/10) (push) Waiting to run
Codegraph E2E Votes / vote (shard 8/10) (push) Waiting to run
Codegraph E2E Votes / vote (shard 9/10) (push) Waiting to run
Docker Dev Branch Images Build / publish (push) Waiting to run
Frontend Unit Tests / Client build recovery regression (push) Waiting to run
Frontend Unit Tests / Codegraph select (push) Waiting to run
Frontend Unit Tests / Build packages (push) Waiting to run
Frontend Unit Tests / TypeScript type checks (client) (push) Blocked by required conditions
Frontend Unit Tests / Tests: @librechat/chat (push) Blocked by required conditions
Frontend Unit Tests / Tests: @librechat/client (push) Blocked by required conditions
Frontend Unit Tests / Tests: Ubuntu (shard 1/2) (push) Blocked by required conditions
Frontend Unit Tests / Tests: Ubuntu (shard 2/2) (push) Blocked by required conditions
Frontend Unit Tests / Vite build verification (push) Blocked by required conditions
Playwright E2E Tests / Codegraph select (push) Waiting to run
Playwright E2E Tests / e2e () (push) Blocked by required conditions
Playwright E2E Tests / MCP list_changed (replica count 1) (push) Blocked by required conditions
Playwright E2E Tests / MCP list_changed (replica count 2) (push) Blocked by required conditions
Playwright E2E Tests / MCP Apps (browser and server) (push) Waiting to run
Playwright E2E Tests / e2e (push) Blocked by required conditions
Main Promotion Policy Tests / Main promotion denial and fast-forward tests (push) Waiting to run
Sync Locize Translations & Create Translation PR / Sync Translation Keys with Locize (push) Has been cancelled
Sync Locize Translations & Create Translation PR / Create Translation PR on Version Published (push) Has been cancelled
* 🍃 ci: Provision the mongod Binary Before Tests Start mongodb-memory-server downloads its ~122 MB mongod the first time a test starts a server, inside a 15 s beforeAll. Only backend-review.yml cached ~/.cache/mongodb-binaries (the same step pasted four times), and a cache alone still leaves the download inside the first hook on a miss. Integration Tests downloaded on every run: 6 failures in 274 runs with "Exceeded timeout of 15000 ms for a hook" in secrets.integration.spec.ts, followed by "spawn ETXTBSY" in the suites that started mongod while the timed-out download was still writing it. Add .github/actions/mongodb-binary: restore the cache with the existing key scheme, ask the installed mongodb-memory-server for the binary it wants (MongoBinary.getPath), make sure that binary sits in the home cache every working directory resolves, run it once, and save the cache right away. The step fails the job when no runnable binary can be put in place. Use it in every job that starts mongodb-memory-server, after dependencies are installed and before tests, and add the action to the path filters of the workflows that list their own file. * 🧪 test: Cover the mongod Provisioning Script Without the Network Run the real script against a stand-in resolver that keeps the properties it relies on: home cache first, download when missing, and a memoised path until the cache is cleared. Integration Tests runs the suite before it uses the action. * 🧪 test: Run mongod Provisioning Against the Real Resolver The suite replaced mongodb-memory-server with a hand-written stand-in, so it checked the script against a description of the resolver rather than the installed one. It now runs the script against the real library and replaces only the network: MONGOMS_DOWNLOAD_MIRROR points at a local server that serves an archive whose bin/mongod is a shell script, with the .md5 the library verifies it against. Recovery also no longer charges a broken copy that was already on disk to the download: with a broken binary in both the home cache and the root node_modules/.cache, the first attempt removed one, the retry met the other, and the step failed without ever downloading. Existing copies are now discarded one at a time and the download keeps its own two attempts. * 🧪 test: Reach the Exit-Status Check Behind the Library's System Binary Probe A system binary that prints nothing is rejected by the library's own version probe, so the case never reached this script's check or the rule that a file the job named is not deleted. Add one that answers the probe and exits non-zero.
225 lines
7.5 KiB
JavaScript
225 lines
7.5 KiB
JavaScript
#!/usr/bin/env node
|
|
/**
|
|
* Puts the mongod build the installed mongodb-memory-server asks for into
|
|
* `~/.cache/mongodb-binaries` before any test starts, and proves it runs.
|
|
*
|
|
* Left to itself the library downloads the ~120 MB archive the first time a
|
|
* test calls `MongoMemoryServer.create()`: inside a `beforeAll` with a 15 s
|
|
* budget. On a slow mirror that hook times out while the download carries on,
|
|
* and whatever starts next spawns a file that is still being written
|
|
* (`spawn ETXTBSY`).
|
|
*
|
|
* The home cache is the one location every consumer resolves no matter where
|
|
* it runs. The copy `npm ci` leaves in the root `node_modules/.cache` is only
|
|
* seen from the repository root: the library derives that directory from
|
|
* INIT_CWD or the working directory, so suites run from `api`,
|
|
* `packages/api` or `packages/data-schemas` look in their own, find nothing
|
|
* and download. A binary found anywhere else is therefore copied into the
|
|
* shared directory rather than trusted.
|
|
*
|
|
* The version, platform and archive are the library's own answer
|
|
* (`MongoBinary.getPath()`), read from the working directory's install, so a
|
|
* dependency bump moves the binary with it. Exits non-zero when no runnable
|
|
* binary could be put in place.
|
|
*/
|
|
import os from 'node:os';
|
|
import path from 'node:path';
|
|
import { createRequire } from 'node:module';
|
|
import { spawnSync } from 'node:child_process';
|
|
import {
|
|
chmodSync,
|
|
copyFileSync,
|
|
existsSync,
|
|
mkdirSync,
|
|
readdirSync,
|
|
rmSync,
|
|
statSync,
|
|
} from 'node:fs';
|
|
|
|
const HOME_CACHE = path.join(os.homedir(), '.cache', 'mongodb-binaries');
|
|
const CHECKSUM = '.md5';
|
|
const DOWNLOAD_ATTEMPTS = 2;
|
|
|
|
/**
|
|
* Where the library looks for an existing binary: MONGOMS_DOWNLOAD_DIR, the
|
|
* home cache, `node_modules/.cache` and `./mongodb-binaries`.
|
|
*/
|
|
const LOOKUP_LOCATIONS = 4;
|
|
|
|
/**
|
|
* The directory the library checks first from any working directory: the home
|
|
* cache, unless the job points MONGOMS_DOWNLOAD_DIR somewhere else for its tests too.
|
|
*/
|
|
const SHARED = process.env.MONGOMS_DOWNLOAD_DIR
|
|
? path.resolve(process.env.MONGOMS_DOWNLOAD_DIR)
|
|
: HOME_CACHE;
|
|
|
|
/** A job that names its own mongod owns that file: it is checked, never copied or replaced. */
|
|
const SYSTEM_BINARY = process.env.MONGOMS_SYSTEM_BINARY;
|
|
|
|
/** A workflow command ends at the first newline, so a multi-line message is folded into one. */
|
|
const headline = (error) =>
|
|
String(error?.message ?? error)
|
|
.split(/\s*\n\s*/)
|
|
.filter(Boolean)
|
|
.join(' ');
|
|
|
|
function loadLibrary() {
|
|
const require = createRequire(path.join(process.cwd(), 'package.json'));
|
|
try {
|
|
return require('mongodb-memory-server');
|
|
} catch (error) {
|
|
throw new Error(
|
|
`mongodb-memory-server is not installed under ${process.cwd()}; run this action after dependencies are installed (${headline(error)})`,
|
|
);
|
|
}
|
|
}
|
|
|
|
function entries(directory) {
|
|
try {
|
|
return readdirSync(directory);
|
|
} catch {
|
|
return [];
|
|
}
|
|
}
|
|
|
|
/** Modification time per file, so a binary the library replaced is not mistaken for the cached one. */
|
|
function snapshot(directory) {
|
|
return new Map(
|
|
entries(directory).map((name) => [name, statSync(path.join(directory, name)).mtimeMs]),
|
|
);
|
|
}
|
|
|
|
function remove(binary) {
|
|
rmSync(binary, { force: true });
|
|
rmSync(binary + CHECKSUM, { force: true });
|
|
}
|
|
|
|
/** The checksum sidecar travels with the binary: the library discards a binary that lacks one. */
|
|
function copy(source, target) {
|
|
mkdirSync(path.dirname(target), { recursive: true });
|
|
copyFileSync(source, target);
|
|
chmodSync(target, 0o755);
|
|
if (existsSync(source + CHECKSUM)) {
|
|
copyFileSync(source + CHECKSUM, target + CHECKSUM);
|
|
}
|
|
}
|
|
|
|
/** The first line of `mongod --version`; throws with the reason when the binary cannot run. */
|
|
function version(binary) {
|
|
const result = spawnSync(binary, ['--version'], { encoding: 'utf8', timeout: 60_000 });
|
|
if (result.status === 0) {
|
|
return result.stdout.split('\n', 1)[0].trim();
|
|
}
|
|
const reason =
|
|
result.error?.message ||
|
|
result.stderr?.trim() ||
|
|
`exit ${result.status ?? result.signal ?? 'unknown'}`;
|
|
throw new Error(`${binary} does not run: ${reason}`);
|
|
}
|
|
|
|
function describe(located, binary, cached) {
|
|
if (SYSTEM_BINARY) {
|
|
return 'MONGOMS_SYSTEM_BINARY';
|
|
}
|
|
if (located !== binary) {
|
|
return `copied from ${located}`;
|
|
}
|
|
return cached.get(path.basename(binary)) === statSync(binary).mtimeMs
|
|
? 'already present'
|
|
: 'downloaded';
|
|
}
|
|
|
|
/**
|
|
* Checks whatever the library resolves. A binary that does not run is deleted
|
|
* wherever it was found, so the next call resolves something else: another
|
|
* copy that was already on disk, or a fresh download.
|
|
*/
|
|
async function provision({ MongoBinary, DryMongoBinary }) {
|
|
const cached = snapshot(SHARED);
|
|
const located = await MongoBinary.getPath();
|
|
const inPlace = Boolean(SYSTEM_BINARY) || path.dirname(located) === SHARED;
|
|
const binary = inPlace ? located : path.join(SHARED, path.basename(located));
|
|
if (!inPlace) {
|
|
copy(located, binary);
|
|
}
|
|
const source = describe(located, binary, cached);
|
|
try {
|
|
return { binary, source, version: version(binary) };
|
|
} catch (error) {
|
|
if (!SYSTEM_BINARY) {
|
|
remove(binary);
|
|
remove(located);
|
|
DryMongoBinary.binaryCache.clear();
|
|
}
|
|
throw Object.assign(error, { preexisting: source !== 'downloaded' });
|
|
}
|
|
}
|
|
|
|
/**
|
|
* Copies that were already on disk and do not run are discarded one at a time
|
|
* without spending a download, so a broken file in one location cannot hide a
|
|
* good one, or the download, behind it. The download itself gets two attempts:
|
|
* a second failure is the mirror or the archive, not chance.
|
|
*/
|
|
async function provisionWithRecovery(library) {
|
|
let discarded = 0;
|
|
let downloads = 0;
|
|
for (;;) {
|
|
try {
|
|
return await provision(library);
|
|
} catch (error) {
|
|
if (error.preexisting) {
|
|
discarded += 1;
|
|
} else {
|
|
downloads += 1;
|
|
}
|
|
if (SYSTEM_BINARY || downloads >= DOWNLOAD_ATTEMPTS || discarded > LOOKUP_LOCATIONS) {
|
|
throw error;
|
|
}
|
|
const next = error.preexisting ? 'Discarded it; resolving again' : 'Retrying the download';
|
|
console.log(`::warning::${headline(error)}. ${next}.`);
|
|
}
|
|
}
|
|
}
|
|
|
|
/** A newer binary supersedes the ones an older lockfile's cache carried over. */
|
|
function prune(binary) {
|
|
const keep = new Set([path.basename(binary), path.basename(binary) + CHECKSUM]);
|
|
return entries(SHARED).filter((name) => {
|
|
if (!name.startsWith('mongod-') || keep.has(name)) {
|
|
return false;
|
|
}
|
|
rmSync(path.join(SHARED, name), { force: true });
|
|
return true;
|
|
});
|
|
}
|
|
|
|
async function main() {
|
|
const library = loadLibrary();
|
|
const result = await provisionWithRecovery(library);
|
|
console.log(`mongod ready: ${result.binary}`);
|
|
console.log(` ${result.version} (${result.source})`);
|
|
|
|
if (SYSTEM_BINARY || SHARED !== HOME_CACHE) {
|
|
console.log(` outside ${HOME_CACHE}, so the binary cache does not cover it`);
|
|
}
|
|
if (SYSTEM_BINARY || result.source === 'already present') {
|
|
return;
|
|
}
|
|
const stale = prune(result.binary);
|
|
if (stale.length > 0) {
|
|
console.log(` removed superseded binaries: ${stale.join(', ')}`);
|
|
}
|
|
if (process.env.MONGODB_BINARY_CACHE_HIT === 'true') {
|
|
console.log(
|
|
'::warning::The restored MongoDB binary cache had no runnable mongod for this install, and an exact cache hit is never re-saved: every job repeats this until the cache entry is deleted or package-lock.json changes.',
|
|
);
|
|
}
|
|
}
|
|
|
|
main().catch((error) => {
|
|
console.error(error);
|
|
console.error(`::error title=MongoDB binary::${headline(error)}`);
|
|
process.exit(1);
|
|
});
|