LibreChat/.github/workflows/gitnexus-index.yml
Danny Avila 76e9543f99
🧹 chore: Cap PR Indexes at 3 and Add Delete-Before-Sync (#12672)
* fix: add docker system prune before image pull to prevent disk exhaustion

The 60GB droplet filled up after ~40 deploys because each
docker compose pull leaves the previous image's layers as
dangling/unused. The gitnexus image is ~700MB, so ~40 stale
copies ≈ 28GB of dead layers. Combined with indexes, OS, and
Docker's build cache, the disk hits 100% and the next pull fails
with 'no space left on device'.

Add a docker system prune -af --volumes BEFORE pulling the new
image on every deploy. This removes stopped containers, unused
networks, all images not referenced by a running container, and
build cache. Running containers are never touched. Typically
frees 1-2GB per deploy (the previous image's layers).

Also add a hard 2GB free-space guard after prune so the deploy
fails with a clear error instead of letting docker pull attempt
a 700MB extract onto a near-full disk.

* fix: cap PR indexes at 3 + delete-before-sync for 10GB disk

The 10GB droplet has ~2GB free. Each index is ~130MB, so 7 PR indexes
(~900MB) plus main+dev (~260MB) plus the ~700MB Docker image leaves
almost nothing for image pulls. The deploy failed with 'no space left
on device' during docker compose pull.

Three changes:

1. Cap PR indexes at MAX_PR_INDEXES=3. The resolve step now sorts
   PR artifacts by created_at descending and only keeps the 3 most
   recent. Older PR indexes are logged as evicted and their droplet
   folders get cleaned by the prune step.

2. Prune BEFORE sync (was after). Freeing disk space from evicted
   indexes before rsyncing new data is critical on a tight disk. The
   old order (sync then prune) could briefly hold both old evicted
   indexes and newly-uploaded ones simultaneously.

3. Delete-before-sync for every index, including main/dev. Instead
   of rsync --delete (which transfers new files then removes extras),
   rm -rf the target folder before rsync so the disk never holds both
   old and new copies of the same index (~260MB saved per index).
   Main/dev are only deleted when a fresh artifact is about to replace
   them — never evicted between deploys.

Budget on 10GB disk:
  OS + Docker engine:    ~4.0 GB
  Docker image (running): ~0.7 GB
  main + dev indexes:    ~0.26 GB
  3 PR indexes:          ~0.39 GB
  Docker prune headroom: ~0.7 GB (for image pull)
  Free:                  ~3.9 GB

* refine: restrict automatic PR indexing to danny-avila authored PRs

With 200+ open PRs and a 10GB disk capped at 3 served PR indexes,
auto-indexing every contributor PR burns CI minutes for artifacts
that will mostly be evicted before anyone queries them.

Narrow the pull_request auto-trigger to PRs authored by danny-avila
only. Other contributors' PRs can still be indexed on demand via
/gitnexus index (contributor-gated comment command) or manual
workflow_dispatch — both arrive as workflow_dispatch events and
bypass the pull_request filter entirely.

* fix: drop --volumes from docker system prune to preserve Caddy TLS state

The deploy workflow explicitly handles a caddy-not-running state later
in the same step. If Caddy is stopped when the prune runs, --volumes
deletes the caddy-data and caddy-config volumes (TLS certs + ACME
account keys), forcing a Let's Encrypt re-issuance on next start.
LE rate-limits to 5 certs per domain per week, so repeated wipes
could brick HTTPS for days.

docker system prune -af (without --volumes) still removes stopped
containers, unused networks, all dangling/unreferenced images, and
build cache — which is where the disk savings come from. Named
volumes are left untouched.

* fix: rsync-then-swap instead of delete-before-sync

The delete-before-sync pattern removed the live index BEFORE rsync
ran. If rsync failed (SSH timeout, disk pressure, network error),
the index was already gone — production served nothing for that
repo until a later deploy succeeded.

Replace with rsync-then-swap: upload to a .new temp directory, and
only rm + mv into place after rsync succeeds. On rsync failure,
the .new temp is cleaned up and the old index stays live. The cost
is ~130MB of extra disk while both old and new coexist, but the
prune step runs first and frees evicted PR indexes, so this fits
comfortably on the 10GB disk.

* fix: fail deploy on main/dev rsync failure, soft-fail PRs only

The rsync-then-swap pattern downgraded ALL failures to a warning,
so the deploy continued even when LibreChat or LibreChat-dev failed
to sync. The job would pull the new image, restart the container,
and report success while serving stale or missing core indexes.

Split by criticality: main/dev rsync failures now exit 1 (aborting
the deploy before the container restart). PR index failures remain
soft-fail with a warning — a missing PR index is inconvenient but
shouldn't take the whole server down.
2026-04-15 09:46:48 -04:00

217 lines
8.9 KiB
YAML

name: GitNexus Index
on:
push:
branches: [main, dev]
paths-ignore: ['**.md', 'docs/**', 'LICENSE', '.github/**']
pull_request:
branches: [main, dev]
paths-ignore: ['**.md', 'docs/**', 'LICENSE', '.github/**']
workflow_dispatch:
inputs:
embeddings:
description: 'Enable embedding generation (slow, increases index size)'
type: boolean
default: false
force:
description: 'Force full re-index'
type: boolean
default: false
# When invoked from the /gitnexus index PR command, the command
# workflow fills these so the index is built from the PR's head
# ref and uploaded under the PR-numbered artifact name.
pr_number:
description: 'PR number to index (set by /gitnexus command)'
type: string
default: ''
pr_ref:
description: 'PR head SHA or ref to check out (set by /gitnexus command)'
type: string
default: ''
permissions:
contents: read
actions: write # dispatch gitnexus-deploy.yml on bot-triggered runs
pull-requests: write # post completion comments for /gitnexus command runs
concurrency:
# When triggered by the /gitnexus command, group by PR number so rapid
# re-runs coalesce. Otherwise group by git ref as before.
group: gitnexus-${{ inputs.pr_number != '' && format('pr-{0}', inputs.pr_number) || github.ref }}
cancel-in-progress: true
env:
GITNEXUS_VERSION: '1.5.3'
jobs:
index:
# Push + dispatch run unconditionally. Native pull_request events
# are restricted to PRs authored by danny-avila only — this keeps
# automatic CI spend low on a repo with 200+ open PRs.
#
# Other contributors' PRs can still be indexed on demand:
# - /gitnexus index (PR comment command, contributor-gated)
# - workflow_dispatch (manual dispatch from Actions UI)
# Both bypass this filter because they arrive as workflow_dispatch,
# not pull_request.
if: |
github.event_name != 'pull_request' ||
github.event.pull_request.user.login == 'danny-avila'
runs-on: ubuntu-latest
timeout-minutes: 25
steps:
- name: Checkout repository
uses: actions/checkout@v4
with:
# When the /gitnexus command dispatches us with a pr_ref, it's
# a refs/pull/<N>/head ref that GitHub mirrors into the base
# repo for every PR, so checkout works for fork PRs too. When
# pr_ref is empty (native push/pull_request), fall back to the
# default ref actions/checkout would use.
ref: ${{ inputs.pr_ref || '' }}
fetch-depth: 1
- name: Setup Node.js
uses: actions/setup-node@v4
with:
node-version: 24
- name: Cache npm store
uses: actions/cache@v4
with:
path: ~/.npm
key: gitnexus-npm-${{ runner.os }}-${{ env.GITNEXUS_VERSION }}
restore-keys: gitnexus-npm-${{ runner.os }}-
- name: Run GitNexus Analyze
env:
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
run: |
FLAGS="--skip-agents-md --verbose"
# Decide whether to generate embeddings. Rules:
# push (main/dev) -> always embed
# pull_request -> embed ONLY when the PR changes files
# under paths that also trigger backend
# or frontend unit tests (api/, client/,
# packages/). Docs/config-only PRs skip
# embeddings to save ~3-5 min of CI.
# workflow_dispatch -> respect the explicit `embeddings` input
# (default false). This also covers the
# /gitnexus index [embeddings] command.
ENABLE_EMBEDDINGS=false
case "${{ github.event_name }}" in
workflow_dispatch)
[ "${{ inputs.embeddings }}" = "true" ] && ENABLE_EMBEDDINGS=true
;;
push)
ENABLE_EMBEDDINGS=true
;;
pull_request)
PR_NUM="${{ github.event.pull_request.number }}"
CHANGED=$(gh api "repos/${{ github.repository }}/pulls/$PR_NUM/files" \
--paginate --jq '.[].filename' 2>/dev/null || echo "")
if printf '%s\n' "$CHANGED" | grep -qE '^(api/|client/|packages/)'; then
echo "PR #$PR_NUM touches unit-test paths (api|client|packages) — enabling embeddings"
ENABLE_EMBEDDINGS=true
else
echo "PR #$PR_NUM does not touch unit-test paths — graph-only index"
fi
;;
esac
if [ "$ENABLE_EMBEDDINGS" = "true" ]; then
FLAGS="$FLAGS --embeddings"
fi
if [ "${{ inputs.force }}" = "true" ]; then
FLAGS="$FLAGS --force"
fi
npx --yes gitnexus@${{ env.GITNEXUS_VERSION }} analyze . $FLAGS
- name: Verify index
run: |
if [ ! -d ".gitnexus" ] || [ ! -f ".gitnexus/meta.json" ]; then
echo "::error::GitNexus index was not created"
exit 1
fi
echo "::group::Index metadata"
cat .gitnexus/meta.json
echo ""
echo "::endgroup::"
- name: Upload GitNexus index
uses: actions/upload-artifact@v4
with:
# Artifact naming order of precedence:
# 1. /gitnexus command dispatch: inputs.pr_number -> pr-<N>
# 2. Native pull_request event: github.event.pull_request.number
# 3. Push or manual dispatch without pr_number: github.ref_name
name: >-
gitnexus-index-${{
inputs.pr_number != ''
&& format('pr-{0}', inputs.pr_number)
|| (github.event_name == 'pull_request'
&& format('pr-{0}', github.event.pull_request.number)
|| github.ref_name)
}}
path: .gitnexus/
include-hidden-files: true
retention-days: 30
# GitHub suppresses workflow_run events for workflow runs whose
# triggering actor is GITHUB_TOKEN (to prevent recursive chaining).
# That means when this workflow is dispatched by gitnexus-pr-command
# via `gh api workflow_dispatch`, the deploy workflow's workflow_run
# trigger never fires. Manually dispatch the deploy here in that
# specific case — user-triggered runs continue to rely on the
# existing workflow_run trigger, so we don't double-deploy.
- name: Trigger deploy workflow for bot-triggered runs
if: github.triggering_actor == 'github-actions[bot]'
uses: actions/github-script@v7
with:
script: |
core.info('Triggering actor is github-actions[bot]; workflow_run would not fire. Dispatching gitnexus-deploy.yml manually.');
// Pass pr_number through so the deploy workflow knows which
// PR to post its completion comment on (for /gitnexus
// command runs this will be set; for other bot dispatches
// it's empty and the deploy step falls back to matrix match).
await github.rest.actions.createWorkflowDispatch({
owner: context.repo.owner,
repo: context.repo.repo,
workflow_id: 'gitnexus-deploy.yml',
ref: 'main',
inputs: {
pr_number: '${{ inputs.pr_number }}',
},
});
# Reply on the PR when the /gitnexus command path runs so the
# requester knows the index step finished. This only fires when
# inputs.pr_number is set (command-triggered) AND the rest of the
# job succeeded. A separate comment posts from the deploy workflow
# when the live server has the fresh index.
- name: Comment on PR — index complete
if: always() && inputs.pr_number != ''
uses: actions/github-script@v7
with:
script: |
const outcome = '${{ job.status }}' === 'success' ? '✅ indexed' : '❌ index failed';
const prNum = parseInt('${{ inputs.pr_number }}', 10);
const runUrl = `${context.serverUrl}/${context.repo.owner}/${context.repo.repo}/actions/runs/${context.runId}`;
const embeddingsFlag = '${{ inputs.embeddings }}' === 'true' ? 'with embeddings' : 'graph-only';
const body = [
`### GitNexus: ${outcome}`,
``,
`PR #${prNum} was indexed ${embeddingsFlag}.`,
`[Index run](${runUrl})`,
'',
'${{ job.status }}' === 'success'
? '⏳ Waiting for deploy to serve the fresh index…'
: '_Index run failed — the previous index (if any) continues to be served._',
].join('\n');
await github.rest.issues.createComment({
owner: context.repo.owner,
repo: context.repo.repo,
issue_number: prNum,
body,
});