The MLX pull path ignored errors from writing a blob, so a full disk
still counted the blob as downloaded. "no space left" only appeared
later, while writing the manifest, after the payload had been read.
Return the write error immediately and do not retry it.
Co-authored-by: Ganesh <ganeshmaheshwaram2005@gmail.com>
* Reduce model lookup and MLX decision request overhead
Avoid decoding unrelated manifests when resolving existing model names.
Reuse small Metal scratch buffers between decision requests, releasing
larger working sets and clearing scratch after scoring errors.
* address comments
The create tests upload blobs which can cause problems when run in parallel on multiple systems. This change splits them out with an explicit tag to make it easier to control when they are run.
* mlxrunner: restore System One scoring
Carry the MLX implementation and its tests from the original System One
commit. The API/compiler and llama-server implementation already exist on
the release branch, so retain those versions. Adapt the optional model test
to the renamed decision package and its explicit rendering/context setup.
(cherry picked from commit 26e546ddb3)
* Enable MLX System One and add regression coverage
* mlxrunner: add Laya support for System One
Import Laya checkpoints and implement bidirectional decision scoring.
* mlxrunner: release cached buffers after System One scoring
* decision: match Tev1 and Nimble publisher prompt formats
Select Tev1 prompts using renderer metadata and fix JSON formatting, with regression coverage for GGUF and MLX.
* mlx: support Clef and Clef Flash decision models
Add model import, quantization safeguards, and text/image scoring with the Clef joint decision head.
* support vision capabilities for decision models
* Make Clef scoring cancellable between prefill chunks
Preserve joint-head inputs and test cancellation, cleanup, and recovery.
* mlx: separate decision preparation and readout from forward
Let serial and batched scoring share model-owned preparation and readout
without duplicating architecture-specific logic in the runner.
* mlx: add Strands Decider support
* implement caching
---------
Co-authored-by: ParthSareen <parth.sareen@ollama.com>
The [img] placeholders mixed internal markers into tool text: the native
llama-server path sends them as literal text, so an image-only result
gained a text part, and tool output containing [img] or [img-N] moved or
dropped image tags. api.Message cannot carry image positions without
in-band markers, so stop merging tool results that include images.
Text-only tool results, the only array form the Chat Completions spec
allows for tool messages, are still merged into one message (now with a
strings.Builder). Results with images keep the per-part split in order,
which preserves image positions, with the tool call id and name set on
every part.
Merging a tool result's parts collected its images separately from the
text, so renderers placed them all before the text: "Before: <A> After:
<B>" rendered as "<A><B>Before: After:". Write an [img] placeholder at
each image's position so renderContentWithImageTags and chatPrompt put
each image back where it was.
Show and list now return just "decision" for models declaring it, so clients don't offer them for general chat, tools, or thinking. The show response also omits the thinking field for these models. Scheduling and serving still check the full capability set.
A tool message with "content": [] produced no message at all, so the
tool call it answers had no result in the rendered prompt. Emit an empty
tool message that keeps its ToolCallID and ToolName, and skip it only
when it has no parts, call id, or name.
FromChatRequest splits array content into one api.Message per part. That
is intended for user content, but a tool message is a single result: the
split also dropped its tool_call_id and tool name, since those were only
copied when the message had tool_calls. Renderers that resolve tool
results by ID then see extra, unnamed results; for example, Kimi K3 fails
with "tool message 2 has no resolvable tool name" on a tool result with
two text parts, which the OpenAI spec allows.
Merge the parts of a tool message into one message with the text joined,
the images kept, and ToolCallID and ToolName set, matching how
function_call_output items are converted for /v1/responses.
Add CAPABILITY declarations to Modelfiles and an additive capabilities
field to create requests. Preserve declarations across GGUF and safetensors
creation, inheritance, and Modelfile export.
Require decision capability before scheduling System One requests instead
of matching Qwen architecture/renderer metadata. Retain main's GGUF-only
scoring restriction until the separate MLX runtime work lands.
Extracted from the capability foundation in 36d46a0c3 on system_one_mlx;
MLX scoring and manifest-list changes are intentionally separate.
* transfer: bound stall retries and let the watchdog interrupt them
A download that stalls could never recover. copy derived its own
cancellable context, but the request was built from the parent, so
cancelling on a stall never reached the in-flight body and the read
stayed parked. The stall watchdog could detect a stall it had no way
to stop.
Stall retries were also unbounded: they never incremented attempt, so
the retry loop had no limit, and because backoff was gated on
attempt > 0 they retried with no delay at all.
The request now owns the cancellable context and passes its cancel
down to copy, so a stall aborts the read. Stalls get their own retry
budget before counting against the limit, and backoff is keyed to
every retry rather than only the counted ones.
* move out of x/
* bench: use code prompts for prompt-token runs
Replace the synthetic word-list prompt with packed HumanEval Python prompts so speculative draft models see code-like continuations.
Keep the benchmark on the chat path, and calibrate -prompt-tokens with untimed chat probes that read PromptEvalCount instead of relying on a separate token-count API.
Each request still gets a nonce prefix to avoid prefix-cache hits while timed epochs measure the chat-rendered workload.
* review comments
* use different windows for each epoch
* add OpenAI API support
* address comments
Add manifest-list storage so runner-specific manifests can coexist under one tag while preserving existing v1 tags as best-effort downgrade anchors. Show/list/copy/remove/pull/push now understand runner and digest selection and transfer referenced child manifests and layers.
Add lazy local compatibility migration for legacy Ollama GGUFs into llama.cpp-compatible children, covering the patched model families and preserving parser/renderer, templates, projectors, media metadata, split GGUFs, and MTP/draft tensors where applicable.
Extend create/import/conversion paths for the same compatibility rules, harden manifest-list combine validation, and add focused manifest, migration, transfer, and integration coverage for first-load conversion plus chat/tools/vision/audio/embedding behavior.
Switch from rejecting API requests to logging a warning. Model creation still rejects the deprecated param.
Follow up from #18448 to remove the hard API failure.
Set USERPROFILE alongside HOME in both cloud-setting handler tests.
On Windows, os.UserHomeDir uses USERPROFILE, so the tests could write
to the developer's real server.json and leave cloud disabled.
Move the ollama_xgrammar target into mlxrunner/xgrammar/native so it
can be configured on its own against an installed xgrammar. cmake/mlx
now adds it as a subdirectory and still uses the pinned xgrammar.
* mlx: speed up Qwen 3.8 prompt processing
Use MLX's gated-delta kernel for long scans and fold dense MLP global scales into SwiGLU.
* address comments
Replace the fixed checkpoint image budget with per-image selection across
the supported 70, 140, 280, 560, and 1120 budgets. Choose the publisher resize
grid closest to the input resolution, accounting for aspect ratio.
This preserves more detail in high-resolution documents while allowing
smaller images to use fewer tokens, without adding an API parameter.
Cover budget boundaries, extreme dimensions, position limits, and media
expansion for both vision architectures.
A format on a thinking model ran two generations: an unconstrained one,
cancelled once the parser reported content, then a re-rendered prompt with
the parsed thinking under the grammar. The restart cost a second prefill,
dropped the chunk that crossed the boundary, needed a harmony prompt hack,
stitched metrics across the two requests, and on MLX could leak a stray
first token into the JSON. The generate endpoint never deferred at all, so
its JSON was forced inside the thinking.
Both handlers now make one completion request that names the strings
ending the response's thinking, from the builtin parser or the generic
thinking parser, and the runner constrains only the content after them in
a single generation. The prompt is evaluated once and metrics pass straight
through. A raw generate prompt names no strings, since nothing says where
its response starts, and its format applies from the first token as
before. A format now applies to whatever follows the thinking, so a tool
call can no longer take the place of formatted content, which was already
the case with thinking off; harmony is the exception, since its tool calls
precede the final message.
The per-token metrics flag both runners carried for the cancelled first
pass has no caller left and goes with the two-pass code and its tests.
Fixes#18441Fixes#17544Fixes#14196Fixes#10929
