LibreChat/.github/workflows/frontend-review.yml
Danny Avila bd9108a87a
🧯 ci: Stop Diagnostic Artifact Uploads From Failing or Stalling Test Jobs (#18068)
On 2026-11-11 the artifact service answered every request with an HTML
page. In Playwright run 43092661470 eight memory shards failed in
"Upload Playwright HTML report" after retrying for 9 to 10.5 minutes;
in six of them every test had passed.

Uploads that no job downloads (Playwright reports, traces and
screenshots, the Lighthouse report, the advisory quality reports) are
now `continue-on-error` with a three minute step timeout, so a failed
upload stays out of the job result and a stalled one cannot run the job
into its own timeout. The Jest result uploads were already
`continue-on-error` and gain the same bound. Build outputs and mutation
shards that a later job downloads stay fatal.

merge-gate.test.mjs holds the gated workflows to that split, and
workflow.test.mts holds Change Quality to it.
2026-11-11 19:17:15 -05:00

691 lines
26 KiB
YAML

name: Frontend Unit Tests
on:
pull_request:
paths:
- 'e2e/client-build.test.mjs'
- 'client/**'
- 'packages/client/**'
- 'packages/chat/**'
- 'packages/data-provider/**'
- 'scripts/typecheck.mts'
- 'package.json'
- 'package-lock.json'
- '.bun-version'
- '.github/workflows/frontend-review.yml'
# The guard spec list, its reader and their test, then every path a frontend guard spec
# reads outside the trees above; guard-specs.test.mjs fails when one is missing here.
- '.github/guard-specs.json'
- '.github/scripts/guard-specs*.mjs'
- 'eslint-suppressions.json'
- '!**.md'
# Post-merge safety net and full-run baseline for gated selection (same rationale as
# backend-review.yml stage 1): every dev merge touching frontend paths runs the full suite.
push:
branches:
- dev
paths:
- 'e2e/client-build.test.mjs'
- 'client/**'
- 'packages/client/**'
- 'packages/chat/**'
- 'packages/data-provider/**'
- 'scripts/typecheck.mts'
- 'package.json'
- 'package-lock.json'
- '.bun-version'
- '.github/workflows/frontend-review.yml'
# The guard spec list, its reader and their test, then every path a frontend guard spec
# reads outside the trees above; guard-specs.test.mjs fails when one is missing here.
- '.github/guard-specs.json'
- '.github/scripts/guard-specs*.mjs'
- 'eslint-suppressions.json'
# The merge queue tests the commit about to land on dev. It has no path filter and no
# selection: every job below runs in full.
merge_group:
types: [checks_requested]
# Promotion requires full CI at the exact dev SHA, including path-filtered commits.
workflow_dispatch:
permissions:
contents: read
pull-requests: read
concurrency:
# PR pushes supersede each other (per-PR canceling group). Push events get a PER-COMMIT group:
# dev-push runs are the post-merge safety net and the full-run baseline, and with a shared
# canceling group closely spaced merges cancel each other's runs — observed live on 2026-08-23,
# when three consecutive dev merges cancelled the runs that would have caught #15142's red
# (Codex P2 on #15145).
group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.sha }}
cancel-in-progress: true
env:
NODE_OPTIONS: '--max-old-space-size=${{ secrets.NODE_MAX_OLD_SPACE_SIZE || 6144 }}'
jobs:
client-build-regression:
name: Client build recovery regression
runs-on: ubuntu-latest
timeout-minutes: 5
steps:
- uses: actions/checkout@v5
- uses: actions/setup-node@v5
with:
node-version: '24.22.3'
cache: npm
# The full workspace tree, under the E2E workflows' exact key and path
# list. This job also runs on dev pushes, so it seeds the dev-scoped
# entry that every pull request against dev can restore.
- name: Restore node_modules cache
id: cache-node-modules
uses: actions/cache/restore@v5
with:
path: |
node_modules
client/node_modules
packages/client/node_modules
packages/chat/node_modules
packages/data-provider/node_modules
packages/data-schemas/node_modules
packages/api/node_modules
packages/channel-gateway/node_modules
api/node_modules
key: node-modules-e2e-${{ runner.os }}-24.22.3-${{ hashFiles('package-lock.json') }}
- name: Install dependencies
id: install-dependencies
if: steps.cache-node-modules.outputs.cache-hit != 'true'
run: npm ci
- name: Save node_modules cache
if: steps.install-dependencies.outcome == 'success'
continue-on-error: true
uses: actions/cache/save@v5
with:
path: |
node_modules
client/node_modules
packages/client/node_modules
packages/chat/node_modules
packages/data-provider/node_modules
packages/data-schemas/node_modules
packages/api/node_modules
packages/channel-gateway/node_modules
api/node_modules
key: ${{ steps.cache-node-modules.outputs.cache-primary-key }}
- run: google-chrome --version
- run: npm run test:client-build
env:
PLAYWRIGHT_CHANNEL: chrome
# Here because this job has node_modules, always runs and gates no other job.
- name: Test guard spec selection
if: ${{ !cancelled() }}
run: node --test .github/scripts/guard-specs.test.mjs
# Stage 1.5 of codegraph gating (stage 1 = backend jest, #15132; stage 2 = matrix lanes,
# #15136). Same mechanism, same record: across 604 finalized shadow receipts the frontend
# selection has zero structural misses (its one raw MISSED was a chronic flake), over 500
# narrowed decisions. Selected (safe mode) on synchronize only; full on PR open/reopen, on
# every dev push, and on any doubt. A workspace skips ONLY on an explicit NONE. Kill switch:
# repo variable CODEGRAPH_GATING=off. Neither frontend workspace defines
# testPathIgnorePatterns, so --runTestsByPath needs no exclude mirroring here.
#
# The graph follows imports, so a spec that reads source with fs is invisible to it. Those
# guard specs are listed in .github/guard-specs.json and run with every narrowed decision.
codegraph_select:
name: Codegraph select
runs-on: ubuntu-latest
timeout-minutes: 5
if: >-
github.event_name == 'pull_request' &&
github.event.action == 'synchronize' &&
vars.CODEGRAPH_GATING != 'off'
outputs:
decided: ${{ steps.sel.outputs.decided }}
client_run: ${{ steps.sel.outputs.client_run }}
client_files: ${{ steps.sel.outputs.client_files }}
clientpkg_run: ${{ steps.sel.outputs.clientpkg_run }}
clientpkg_files: ${{ steps.sel.outputs.clientpkg_files }}
steps:
# The commit under test, for the guard spec list, its reader and the reader's check that
# a selection is not stale. Best effort: without it the select step runs every narrowed
# workspace FULL, so a failed checkout never fails this job.
- uses: actions/checkout@v5
continue-on-error: true
with:
persist-credentials: false
- name: Select tests, fail open on any doubt
id: sel
env:
URL: ${{ secrets.CODEGRAPH_URL }}
TOKEN: ${{ secrets.CODEGRAPH_TOKEN }}
GH_TOKEN: ${{ github.token }}
REPO: ${{ github.repository }}
PR: ${{ github.event.pull_request.number }}
BASE_SHA: ${{ github.event.pull_request.base.sha }}
HEAD_SHA: ${{ github.event.pull_request.head.sha }}
CHANGED: ${{ github.event.pull_request.changed_files }}
run: |
set +e
note() { echo "$1" >> "$GITHUB_STEP_SUMMARY"; }
note "### Codegraph select — GATING (frontend jest)"
if [ -z "$URL" ] || [ -z "$TOKEN" ]; then note "_no codegraph config; running FULL_"; exit 0; fi
if ! gh api "repos/$REPO/pulls/$PR/files" --paginate \
--jq '.[] | {path: .filename, status, patch}' > files.ndjson; then
note "_could not fetch changed files; running FULL_"; exit 0
fi
jq -s . files.ndjson > files.json
N=$(jq 'length' files.json)
if [ "$N" -eq 0 ] || { [ -n "$CHANGED" ] && [ "$N" -ne "$CHANGED" ]; }; then
note "_changed-file list incomplete ($N of ${CHANGED:-?}); running FULL_"; exit 0
fi
jq -c --arg b "$BASE_SHA" --arg h "$HEAD_SHA" \
'{files: ., mode: "safe", lockBaseSha: $b, lockHeadSha: $h}' files.json > body.json
RESP=$(curl -sS --fail-with-body -m 45 -H "Authorization: Bearer $TOKEN" \
-H 'content-type: application/json' --data-binary @body.json "$URL/v1/select"); RC=$?
if [ "$RC" -ne 0 ] || [ -z "$RESP" ] || ! echo "$RESP" | jq -e '.selected.client.mode' >/dev/null 2>&1; then
note "_codegraph unavailable (curl exit $RC: ${RESP:0:120}); running FULL_"
exit 0
fi
emit() {
key="$1"; ws="$2"
mode=$(echo "$RESP" | jq -r --arg w "$ws" '.selected[$w].mode')
files=""
if [ "$mode" = "FILES" ]; then
# Only an explicit NONE may skip. FILES with a missing/empty list is a malformed
# decision (service/schema skew) and must run FULL (Codex P1 on #15145).
raw_n=$(echo "$RESP" | jq -r --arg w "$ws" '.selected[$w].files // [] | length')
# Every selected path must live under the workspace: a wrong-prefixed path would
# survive ltrimstr, match nothing in the workspace cwd, and --passWithNoTests would
# turn "ran nothing" into green — a silent fail-closed (Codex P1 on #15145).
misplaced=$(echo "$RESP" | jq -r --arg w "$ws" --arg p "$ws/" '[.selected[$w].files // [] | .[] | select(startswith($p) | not)] | length')
if [ "$raw_n" = "0" ] || [ "$misplaced" != "0" ]; then
mode="FULL"
note "| $ws | malformed FILES decision ($raw_n files, $misplaced outside $ws/); running FULL |"
else
files=$(echo "$RESP" | jq -r --arg w "$ws" --arg p "$ws/" \
'.selected[$w].files // [] | map(select(test(" ") | not)) | map(ltrimstr($p)) | join(" ")')
spaced=$(echo "$RESP" | jq -r --arg w "$ws" '[.selected[$w].files // [] | .[] | select(test(" "))] | length')
if [ "$spaced" != "0" ]; then mode="FULL"; files=""; fi
fi
fi
# Guard specs read source with fs, so no import edge selects them. A narrowed run
# carries its workspace's guards, and a skip becomes a guards-only run when the
# change touches what they read (.github/guard-specs.json; the rules are in
# .github/scripts/guard-specs.mjs). A list that cannot be read runs FULL. A selection
# with no file left in this checkout gets no guards: the test job runs FULL on a
# stale selection, and guards that exist would turn that into a guards-only run.
guard_note=""
if [ "$mode" = "FILES" ] || [ "$mode" = "NONE" ]; then
guarded=$(SELECTED="$files" node .github/scripts/guard-specs.mjs "$mode" "$ws" files.json ""); grc=$?
if [ "$grc" -ne 0 ]; then
mode="FULL"; files=""
note "| $ws | guard specs unreadable (exit $grc); running FULL |"
elif [ -n "$guarded" ]; then
if [ "$mode" = "NONE" ]; then guard_note=", guard specs only"; fi
mode="FILES"; files="$guarded"
fi
fi
if [ "$mode" = "NONE" ]; then
echo "${key}_run=false" >> "$GITHUB_OUTPUT"
note "| $ws | skip (no reachable tests) |"
elif [ "$mode" = "FILES" ]; then
n=$(echo "$files" | wc -w | tr -d ' ')
echo "${key}_run=true" >> "$GITHUB_OUTPUT"
echo "${key}_files=$files" >> "$GITHUB_OUTPUT"
note "| $ws | $n selected files$guard_note |"
else
echo "${key}_run=true" >> "$GITHUB_OUTPUT"
note "| $ws | FULL |"
fi
}
note "| workspace | decision |"
note "|---|---|"
emit client client
emit clientpkg packages/client
echo "decided=true" >> "$GITHUB_OUTPUT"
note ""
note "kill switch: repo variable \`CODEGRAPH_GATING=off\`; full runs remain on PR open and on every dev push"
exit 0
build:
name: Build packages
runs-on: ubuntu-latest
timeout-minutes: 15
steps:
- uses: actions/checkout@v5
- name: Use Node.js 24.22.3
uses: actions/setup-node@v5
with:
node-version: '24.22.3'
- name: Restore node_modules cache
id: cache-node-modules
uses: actions/cache@v5
with:
path: |
node_modules
client/node_modules
packages/client/node_modules
packages/chat/node_modules
packages/data-provider/node_modules
key: node-modules-frontend-${{ runner.os }}-24.22.3-${{ hashFiles('package-lock.json') }}
- name: Install dependencies
if: steps.cache-node-modules.outputs.cache-hit != 'true'
run: npm ci
- name: Restore data-provider build cache
id: cache-data-provider
uses: actions/cache@v5
with:
path: packages/data-provider/dist
key: build-data-provider-${{ runner.os }}-${{ hashFiles('package.json', 'package-lock.json', 'packages/data-provider/src/**', 'packages/data-provider/tsconfig*.json', 'packages/data-provider/tsdown.config.mjs', 'packages/data-provider/package.json') }}
- name: Build data-provider
if: steps.cache-data-provider.outputs.cache-hit != 'true'
run: npm run build:data-provider
- name: Restore client-package build cache
id: cache-client-package
uses: actions/cache@v5
with:
path: packages/client/dist
key: build-client-package-${{ runner.os }}-${{ hashFiles('package.json', 'package-lock.json', 'packages/client/src/**', 'packages/client/tsconfig*.json', 'packages/client/tsdown.config.mjs', 'packages/client/package.json', 'packages/data-provider/src/**', 'packages/data-provider/tsconfig*.json', 'packages/data-provider/tsdown.config.mjs', 'packages/data-provider/package.json') }}
- name: Build client-package
if: steps.cache-client-package.outputs.cache-hit != 'true'
run: npm run build:client-package
# About a second, so it is not cached. The client imports the package's built entries, so
# the type check, unit tests and Vite build below download this output.
- name: Build chat package
run: npm run build:chat
- name: Upload data-provider build
uses: actions/upload-artifact@v6
with:
name: build-data-provider
path: packages/data-provider/dist
retention-days: 2
- name: Upload client-package build
uses: actions/upload-artifact@v6
with:
name: build-client-package
path: packages/client/dist
retention-days: 2
- name: Upload chat-package build
uses: actions/upload-artifact@v6
with:
name: build-chat-package
path: packages/chat/dist
retention-days: 2
typecheck:
name: TypeScript type checks (client)
needs: build
runs-on: ubuntu-latest
timeout-minutes: 10
steps:
- uses: actions/checkout@v5
- name: Use Node.js 24.22.3
uses: actions/setup-node@v5
with:
node-version: '24.22.3'
- name: Restore node_modules cache
id: cache-node-modules
uses: actions/cache@v5
with:
path: |
node_modules
client/node_modules
packages/client/node_modules
packages/chat/node_modules
packages/data-provider/node_modules
key: node-modules-frontend-${{ runner.os }}-24.22.3-${{ hashFiles('package-lock.json') }}
- name: Install dependencies
if: steps.cache-node-modules.outputs.cache-hit != 'true'
run: npm ci
- name: Download data-provider build
uses: actions/download-artifact@v7
with:
name: build-data-provider
path: packages/data-provider/dist
- name: Download client-package build
uses: actions/download-artifact@v7
with:
name: build-client-package
path: packages/client/dist
- name: Download chat-package build
uses: actions/download-artifact@v7
with:
name: build-chat-package
path: packages/chat/dist
- name: Use pinned Bun
uses: oven-sh/setup-bun@v2
with:
bun-version-file: .bun-version
- name: Type check client
run: npm run typecheck
working-directory: client
test-packages-chat:
name: 'Tests: @librechat/chat'
needs: build
if: ${{ !cancelled() && needs.build.result == 'success' }}
runs-on: ubuntu-latest
timeout-minutes: 10
steps:
- uses: actions/checkout@v5
- name: Use Node.js 24.22.3
uses: actions/setup-node@v5
with:
node-version: '24.22.3'
- name: Restore node_modules cache
id: cache-node-modules
uses: actions/cache@v5
with:
path: |
node_modules
client/node_modules
packages/client/node_modules
packages/chat/node_modules
packages/data-provider/node_modules
key: node-modules-frontend-${{ runner.os }}-24.22.3-${{ hashFiles('package-lock.json') }}
- name: Install dependencies
if: steps.cache-node-modules.outputs.cache-hit != 'true'
run: npm ci
- name: Download data-provider build
uses: actions/download-artifact@v7
with:
name: build-data-provider
path: packages/data-provider/dist
- name: Download client-package build
uses: actions/download-artifact@v7
with:
name: build-client-package
path: packages/client/dist
- name: Use pinned Bun
uses: oven-sh/setup-bun@v2
with:
bun-version-file: .bun-version
- name: Type check
run: npm run typecheck
working-directory: packages/chat
- name: Run unit tests
run: npm run test:ci
working-directory: packages/chat
test-packages-client:
name: 'Tests: @librechat/client'
needs: [build, codegraph_select]
if: >-
!cancelled() && needs.build.result == 'success' &&
needs.codegraph_select.outputs.clientpkg_run != 'false'
runs-on: ubuntu-latest
timeout-minutes: 10
steps:
- uses: actions/checkout@v5
- name: Use Node.js 24.22.3
uses: actions/setup-node@v5
with:
node-version: '24.22.3'
- name: Restore node_modules cache
id: cache-node-modules
uses: actions/cache@v5
with:
path: |
node_modules
client/node_modules
packages/client/node_modules
packages/chat/node_modules
packages/data-provider/node_modules
key: node-modules-frontend-${{ runner.os }}-24.22.3-${{ hashFiles('package-lock.json') }}
- name: Install dependencies
if: steps.cache-node-modules.outputs.cache-hit != 'true'
run: npm ci
- name: Download data-provider build
uses: actions/download-artifact@v7
with:
name: build-data-provider
path: packages/data-provider/dist
- name: Run unit tests
env:
SELECTED: ${{ needs.codegraph_select.outputs.clientpkg_files }}
JEST_JSON: --json --outputFile=${{ github.workspace }}/jest-results/jest-results-packages-client.json
run: |
mkdir -p "$GITHUB_WORKSPACE/jest-results"
# A selected path can be stale in exactly two ways at this checkout (Codex P2, #15145 r6):
# deleted on the branch — dropped, which matches full CI (the file runs nowhere) — or
# renamed, where the NEW path is a changed test file and is selected independently. If
# NOTHING selected exists, the selection is stale wholesale and the suite runs FULL;
# --passWithNoTests must never turn "ran nothing" into green.
if [ -n "$SELECTED" ]; then
KEEP=""
for f in $SELECTED; do
if [ -f "$f" ]; then KEEP="$KEEP $f"; else echo "dropping selected path absent at HEAD (deleted or renamed): $f"; fi
done
KEEP="${KEEP# }"
if [ -z "$KEEP" ]; then
echo "no selected test file exists at HEAD (stale selection); running FULL"
npm run test:ci -- $JEST_JSON
else
echo "codegraph: $(echo $KEEP | wc -w) selected test files (safe mode)"
npm run test:ci -- --passWithNoTests --runTestsByPath $KEEP $JEST_JSON
fi
else
npm run test:ci -- $JEST_JSON
fi
working-directory: packages/client
# Per-test results keyed by head SHA (run.head_sha), for the codegraph test-evidence feed.
# Never part of the gate: it cannot fail the job or stall it into its timeout, and a re-run
# overwrites its own artifact.
- name: Upload Jest results
if: ${{ !cancelled() }}
timeout-minutes: 3
continue-on-error: true
uses: actions/upload-artifact@v6
with:
name: jest-results-packages-client
path: jest-results/
retention-days: 7
if-no-files-found: ignore
overwrite: true
test-ubuntu:
name: 'Tests: Ubuntu (shard ${{ matrix.shard }}/2)'
needs: [build, codegraph_select]
if: >-
!cancelled() && needs.build.result == 'success' &&
needs.codegraph_select.outputs.client_run != 'false'
runs-on: ubuntu-latest
timeout-minutes: 15
strategy:
fail-fast: false
matrix:
shard: [1, 2]
steps:
- uses: actions/checkout@v5
- name: Use Node.js 24.22.3
uses: actions/setup-node@v5
with:
node-version: '24.22.3'
- name: Restore node_modules cache
id: cache-node-modules
uses: actions/cache@v5
with:
path: |
node_modules
client/node_modules
packages/client/node_modules
packages/chat/node_modules
packages/data-provider/node_modules
key: node-modules-frontend-${{ runner.os }}-24.22.3-${{ hashFiles('package-lock.json') }}
- name: Install dependencies
if: steps.cache-node-modules.outputs.cache-hit != 'true'
run: npm ci
- name: Download data-provider build
uses: actions/download-artifact@v7
with:
name: build-data-provider
path: packages/data-provider/dist
- name: Download client-package build
uses: actions/download-artifact@v7
with:
name: build-client-package
path: packages/client/dist
- name: Download chat-package build
uses: actions/download-artifact@v7
with:
name: build-chat-package
path: packages/chat/dist
- name: Run unit tests (shard ${{ matrix.shard }}/2)
env:
SELECTED: ${{ needs.codegraph_select.outputs.client_files }}
JEST_JSON: --json --outputFile=${{ github.workspace }}/jest-results/jest-results-client-${{ matrix.shard }}.json
run: |
mkdir -p "$GITHUB_WORKSPACE/jest-results"
if [ -n "$SELECTED" ]; then
KEEP=""
for f in $SELECTED; do
if [ -f "$f" ]; then KEEP="$KEEP $f"; else echo "dropping selected path absent at HEAD (deleted or renamed): $f"; fi
done
KEEP="${KEEP# }"
if [ -z "$KEEP" ]; then
echo "no selected test file exists at HEAD (stale selection); running FULL"
npm run test:ci -- --shard=${{ matrix.shard }}/2 $JEST_JSON
else
echo "codegraph: $(echo $KEEP | wc -w) selected test files (safe mode)"
npm run test:ci -- --shard=${{ matrix.shard }}/2 --passWithNoTests --runTestsByPath $KEEP $JEST_JSON
fi
else
npm run test:ci -- --shard=${{ matrix.shard }}/2 $JEST_JSON
fi
working-directory: client
# Per-test results keyed by head SHA (run.head_sha), for the codegraph test-evidence feed.
# Never part of the gate: it cannot fail the job or stall it into its timeout, and a re-run
# overwrites its own artifact.
- name: Upload Jest results
if: ${{ !cancelled() }}
timeout-minutes: 3
continue-on-error: true
uses: actions/upload-artifact@v6
with:
name: jest-results-client-${{ matrix.shard }}
path: jest-results/
retention-days: 7
if-no-files-found: ignore
overwrite: true
build-verify:
name: Vite build verification
needs: build
runs-on: ubuntu-latest
timeout-minutes: 15
steps:
- uses: actions/checkout@v5
- name: Use Node.js 24.22.3
uses: actions/setup-node@v5
with:
node-version: '24.22.3'
- name: Restore node_modules cache
id: cache-node-modules
uses: actions/cache@v5
with:
path: |
node_modules
client/node_modules
packages/client/node_modules
packages/chat/node_modules
packages/data-provider/node_modules
key: node-modules-frontend-${{ runner.os }}-24.22.3-${{ hashFiles('package-lock.json') }}
- name: Install dependencies
if: steps.cache-node-modules.outputs.cache-hit != 'true'
run: npm ci
- name: Download data-provider build
uses: actions/download-artifact@v7
with:
name: build-data-provider
path: packages/data-provider/dist
- name: Download client-package build
uses: actions/download-artifact@v7
with:
name: build-client-package
path: packages/client/dist
- name: Download chat-package build
uses: actions/download-artifact@v7
with:
name: build-chat-package
path: packages/chat/dist
- name: Build client
run: cd client && npm run build:ci
# dev's merge queue requires this job by name (see merge-gate.yml). It depends on every
# other job here, so a new job has to be added to `needs`; merge-gate.test.mjs fails
# until it is. A skip passes only where ALLOW_SKIPPED names it.
gate:
name: frontend
needs:
- client-build-regression
- codegraph_select
- build
- typecheck
- test-packages-chat
- test-packages-client
- test-ubuntu
- build-verify
if: ${{ !cancelled() }}
runs-on: ubuntu-latest
timeout-minutes: 5
steps:
- uses: actions/checkout@v5
with:
persist-credentials: false
sparse-checkout: .github/scripts
- name: Verify every frontend job passed
env:
NEEDS: ${{ toJSON(needs) }}
ALLOW_SKIPPED: >-
codegraph_select
${{ needs.codegraph_select.outputs.clientpkg_run == 'false' && 'test-packages-client' || '' }}
${{ needs.codegraph_select.outputs.client_run == 'false' && 'test-ubuntu' || '' }}
run: node .github/scripts/merge-gate.mjs verify