The Clef head read its weights with std::ifstream::seekg. The Windows
llama-server ships with libc++ 18.1.8, where that seek truncates the
offset to 32 bits. clef-flash stores the clef.* tensors at about 9.5GB,
so the head loaded backbone bytes as its weights and every
/v1/systemone request failed with "Clef: non-finite logit". clef:27b
keeps the head and output.weight below 2GB and was unaffected.
Read with ggml_fopen and an explicit 64-bit seek instead, as read_at
in llama/compat already does.
Fixes#18769
Forward GET /api/usage and GET /api/balance through the local server
using the signed-in account.
Add API documentation and tests for proxying, signing, and
disabled cloud access.
Implements the EmbeddingGemma2Model architecture on the MLX runner: 24-layer
bidirectional text encoder with PLE, shared gemma4 vision/audio towers,
mean-pool + L2 output. /api/embed accepts per-item media via input dicts.
* Reduce model lookup and MLX decision request overhead
Avoid decoding unrelated manifests when resolving existing model names.
Reuse small Metal scratch buffers between decision requests, releasing
larger working sets and clearing scratch after scoring errors.
* address comments
The create tests upload blobs which can cause problems when run in parallel on multiple systems. This change splits them out with an explicit tag to make it easier to control when they are run.
* mlxrunner: restore System One scoring
Carry the MLX implementation and its tests from the original System One
commit. The API/compiler and llama-server implementation already exist on
the release branch, so retain those versions. Adapt the optional model test
to the renamed decision package and its explicit rendering/context setup.
(cherry picked from commit 26e546ddb3)
* Enable MLX System One and add regression coverage
* mlxrunner: add Laya support for System One
Import Laya checkpoints and implement bidirectional decision scoring.
* mlxrunner: release cached buffers after System One scoring
* decision: match Tev1 and Nimble publisher prompt formats
Select Tev1 prompts using renderer metadata and fix JSON formatting, with regression coverage for GGUF and MLX.
* mlx: support Clef and Clef Flash decision models
Add model import, quantization safeguards, and text/image scoring with the Clef joint decision head.
* support vision capabilities for decision models
* Make Clef scoring cancellable between prefill chunks
Preserve joint-head inputs and test cancellation, cleanup, and recovery.
* mlx: separate decision preparation and readout from forward
Let serial and batched scoring share model-owned preparation and readout
without duplicating architecture-specific logic in the runner.
* mlx: add Strands Decider support
* implement caching
---------
Co-authored-by: ParthSareen <parth.sareen@ollama.com>
The [img] placeholders mixed internal markers into tool text: the native
llama-server path sends them as literal text, so an image-only result
gained a text part, and tool output containing [img] or [img-N] moved or
dropped image tags. api.Message cannot carry image positions without
in-band markers, so stop merging tool results that include images.
Text-only tool results, the only array form the Chat Completions spec
allows for tool messages, are still merged into one message (now with a
strings.Builder). Results with images keep the per-part split in order,
which preserves image positions, with the tool call id and name set on
every part.
Merging a tool result's parts collected its images separately from the
text, so renderers placed them all before the text: "Before: <A> After:
<B>" rendered as "<A><B>Before: After:". Write an [img] placeholder at
each image's position so renderContentWithImageTags and chatPrompt put
each image back where it was.
Show and list now return just "decision" for models declaring it, so clients don't offer them for general chat, tools, or thinking. The show response also omits the thinking field for these models. Scheduling and serving still check the full capability set.
A tool message with "content": [] produced no message at all, so the
tool call it answers had no result in the rendered prompt. Emit an empty
tool message that keeps its ToolCallID and ToolName, and skip it only
when it has no parts, call id, or name.
FromChatRequest splits array content into one api.Message per part. That
is intended for user content, but a tool message is a single result: the
split also dropped its tool_call_id and tool name, since those were only
copied when the message had tool_calls. Renderers that resolve tool
results by ID then see extra, unnamed results; for example, Kimi K3 fails
with "tool message 2 has no resolvable tool name" on a tool result with
two text parts, which the OpenAI spec allows.
Merge the parts of a tool message into one message with the text joined,
the images kept, and ToolCallID and ToolName set, matching how
function_call_output items are converted for /v1/responses.
Add CAPABILITY declarations to Modelfiles and an additive capabilities
field to create requests. Preserve declarations across GGUF and safetensors
creation, inheritance, and Modelfile export.
Require decision capability before scheduling System One requests instead
of matching Qwen architecture/renderer metadata. Retain main's GGUF-only
scoring restriction until the separate MLX runtime work lands.
Extracted from the capability foundation in 36d46a0c3 on system_one_mlx;
MLX scoring and manifest-list changes are intentionally separate.
* transfer: bound stall retries and let the watchdog interrupt them
A download that stalls could never recover. copy derived its own
cancellable context, but the request was built from the parent, so
cancelling on a stall never reached the in-flight body and the read
stayed parked. The stall watchdog could detect a stall it had no way
to stop.
Stall retries were also unbounded: they never incremented attempt, so
the retry loop had no limit, and because backoff was gated on
attempt > 0 they retried with no delay at all.
The request now owns the cancellable context and passes its cancel
down to copy, so a stall aborts the read. Stalls get their own retry
budget before counting against the limit, and backoff is keyed to
every retry rather than only the counted ones.
* move out of x/
* bench: use code prompts for prompt-token runs
Replace the synthetic word-list prompt with packed HumanEval Python prompts so speculative draft models see code-like continuations.
Keep the benchmark on the chat path, and calibrate -prompt-tokens with untimed chat probes that read PromptEvalCount instead of relying on a separate token-count API.
Each request still gets a nonce prefix to avoid prefix-cache hits while timed epochs measure the chat-rendered workload.
* review comments
* use different windows for each epoch
* add OpenAI API support
* address comments
Add manifest-list storage so runner-specific manifests can coexist under one tag while preserving existing v1 tags as best-effort downgrade anchors. Show/list/copy/remove/pull/push now understand runner and digest selection and transfer referenced child manifests and layers.
Add lazy local compatibility migration for legacy Ollama GGUFs into llama.cpp-compatible children, covering the patched model families and preserving parser/renderer, templates, projectors, media metadata, split GGUFs, and MTP/draft tensors where applicable.
Extend create/import/conversion paths for the same compatibility rules, harden manifest-list combine validation, and add focused manifest, migration, transfer, and integration coverage for first-load conversion plus chat/tools/vision/audio/embedding behavior.
Switch from rejecting API requests to logging a warning. Model creation still rejects the deprecated param.
Follow up from #18448 to remove the hard API failure.
