Show and list now return just "decision" for models declaring it, so clients don't offer them for general chat, tools, or thinking. The show response also omits the thinking field for these models. Scheduling and serving still check the full capability set.
Add CAPABILITY declarations to Modelfiles and an additive capabilities
field to create requests. Preserve declarations across GGUF and safetensors
creation, inheritance, and Modelfile export.
Require decision capability before scheduling System One requests instead
of matching Qwen architecture/renderer metadata. Retain main's GGUF-only
scoring restriction until the separate MLX runtime work lands.
Extracted from the capability foundation in 36d46a0c3 on system_one_mlx;
MLX scoring and manifest-list changes are intentionally separate.
* transfer: bound stall retries and let the watchdog interrupt them
A download that stalls could never recover. copy derived its own
cancellable context, but the request was built from the parent, so
cancelling on a stall never reached the in-flight body and the read
stayed parked. The stall watchdog could detect a stall it had no way
to stop.
Stall retries were also unbounded: they never incremented attempt, so
the retry loop had no limit, and because backoff was gated on
attempt > 0 they retried with no delay at all.
The request now owns the cancellable context and passes its cancel
down to copy, so a stall aborts the read. Stalls get their own retry
budget before counting against the limit, and backoff is keyed to
every retry rather than only the counted ones.
* move out of x/
* bench: use code prompts for prompt-token runs
Replace the synthetic word-list prompt with packed HumanEval Python prompts so speculative draft models see code-like continuations.
Keep the benchmark on the chat path, and calibrate -prompt-tokens with untimed chat probes that read PromptEvalCount instead of relying on a separate token-count API.
Each request still gets a nonce prefix to avoid prefix-cache hits while timed epochs measure the chat-rendered workload.
* review comments
* use different windows for each epoch
* add OpenAI API support
* address comments
Switch from rejecting API requests to logging a warning. Model creation still rejects the deprecated param.
Follow up from #18448 to remove the hard API failure.
Set USERPROFILE alongside HOME in both cloud-setting handler tests.
On Windows, os.UserHomeDir uses USERPROFILE, so the tests could write
to the developer's real server.json and leave cloud disabled.
Move the ollama_xgrammar target into mlxrunner/xgrammar/native so it
can be configured on its own against an installed xgrammar. cmake/mlx
now adds it as a subdirectory and still uses the pinned xgrammar.
* mlx: speed up Qwen 3.8 prompt processing
Use MLX's gated-delta kernel for long scans and fold dense MLP global scales into SwiGLU.
* address comments
Replace the fixed checkpoint image budget with per-image selection across
the supported 70, 140, 280, 560, and 1120 budgets. Choose the publisher resize
grid closest to the input resolution, accounting for aspect ratio.
This preserves more detail in high-resolution documents while allowing
smaller images to use fewer tokens, without adding an API parameter.
Cover budget boundaries, extreme dimensions, position limits, and media
expansion for both vision architectures.
A format on a thinking model ran two generations: an unconstrained one,
cancelled once the parser reported content, then a re-rendered prompt with
the parsed thinking under the grammar. The restart cost a second prefill,
dropped the chunk that crossed the boundary, needed a harmony prompt hack,
stitched metrics across the two requests, and on MLX could leak a stray
first token into the JSON. The generate endpoint never deferred at all, so
its JSON was forced inside the thinking.
Both handlers now make one completion request that names the strings
ending the response's thinking, from the builtin parser or the generic
thinking parser, and the runner constrains only the content after them in
a single generation. The prompt is evaluated once and metrics pass straight
through. A raw generate prompt names no strings, since nothing says where
its response starts, and its format applies from the first token as
before. A format now applies to whatever follows the thinking, so a tool
call can no longer take the place of formatted content, which was already
the case with thinking off; harmony is the exception, since its tool calls
precede the final message.
The per-token metrics flag both runners carried for the cancelled first
pass has no caller left and goes with the two-pass code and its tests.
Fixes#18441Fixes#17544Fixes#14196Fixes#10929
llama-server applies a schema from the first sampled token, so a thinking
model asked for a format cannot think first, and the server has to run two
generations to get both the thinking and the formatted content.
The client now sends one request whose grammar leaves the text before a
closing string unconstrained and requires the format after it.
llama-server converts the schema for us: an empty completion evaluates and
generates nothing but reports the GBNF it derived, which we wrap in rules
that recognize the closing strings and cache per schema for the life of
the process. On qwen3 0.6b at temperature 0 the thinking is byte-identical
with and without a format and the JSON follows the schema.
A response that ends before a closing string is delivered unchanged. The
conversion request briefly takes a llama-server slot on a cache miss. The
server does not send the strings yet.
The MLX runner applies a format's grammar from the first sampled token,
so a thinking model asked for a format cannot think first, and the server
has to run two generations to get both the thinking and the formatted
content.
A completion request now carries the strings that end the thinking its
response begins with, and the MLX client builds from them a structural
tag: free text that cannot contain any of them, then one of them, then
the schema. The tail is optional so a response may still end inside its
thinking, as an unconstrained one can. Without a closing string the tag
is the plain schema, as before. The server does not send the strings yet.
A format on a thinking model has to leave the thinking free and constrain
only the content after it, so whatever enforces the format needs to know
where the thinking ends. Today the server guesses whether a parser's
response starts inside thinking from the think value alone, which is wrong
for parsers whose default differs, and it has no way to learn the closing
string at all.
Each parser now answers ThinkingClose after Init: the strings any of which
ends the thinking its response begins with, or none when the response
starts in content because thinking is off, an assistant prefill continues
content, or the parser suppresses thinking for tools. Parsers whose models
open a new message before content end the thinking at that message's
header. Nothing consumes the answer yet.
getExistingName canonicalizes the case of each model name part (host,
namespace, model, tag) by searching all manifests for a case-insensitive
match. The original implementation matched each part independently —
the tag from any manifest whose tag case-insensitively matched the
requested tag would overwrite the tag, regardless of whether the host,
namespace, or model matched.
A 'set' variable was intended to track which parts had already been
canonicalized and prevent overwrites, but it was never written to, so
it was always zero-valued and every match overwrote the corresponding
part unconditionally.
With 3000+ manifests, if another model had a tag that case-insensitively
matched (e.g. 'Q4_K_M' for a different model), the requested model's tag
could be canonicalized to that other model's tag casing. Go's map
iteration order is randomized, so the last match wins — producing
intermittent 'model not found' errors that succeed on retry.
Fix: when all four parts of an entry case-insensitively match the input,
return that entry's canonical name directly. Otherwise canonicalize each
part independently, with the 'set' variable now properly updated after
each part is set so it is only written once. This handles both exact
matches and new tags on existing models.
* nemotron_h: add MLX vision support
Implement the RADIO vision encoder and projector on the shared MLX media pipeline, including dynamic-resolution preprocessing, deterministic placeholder expansion, chunked feature scattering, and MTP offsets.
Expose source-advertised Nemotron vision while continuing to suppress unsupported audio, and preserve both modality towers at source precision during create.
Harden Nemotron streaming parser termination and add focused coverage for vision configuration, media placement, capability reporting, and tool-call parsing.
* review comments
* address comments
Serialize final release jobs and delete old assets before retagging to
avoid overlapping updates and mixed-version payloads.
Keep uploads parallel and retry failures twice with 15s/30s delays.
Retry cleanup using a fresh asset list and upload the checksum manifest
only after all payloads succeed.
Redirects for registry and blob transfers now validate the target scheme
and resolved addresses before following, re-check DNS on each redirect,
and do not follow redirects that switch an https session to plain http.
The --insecure option continues to relax address checks for private
registries but not scheme checks.
The decode loop releases MLX's pool of freed buffers every 256 generated
tokens, which is also how often the KV cache grows and drops its previous,
smaller buffers. The check fires only when the token count lands exactly on
a multiple of 256. Speculative decoding emits several tokens per round, so
most rounds step over the boundary and the pool is never released. Each
growth at a long context leaves several GB of buffers that no later
allocation can reuse, so the runner's footprint keeps climbing over a long
generation until the system runs out of memory.
We now release the pool whenever a round crosses a multiple of 256 tokens,
which is what a single-token round already did. With qwen3.8:27b-mlx at a
98k-token context on a 128 GB machine, a long speculative generation
previously grew the runner past 90 GB and panicked the kernel; it now stays
flat at 30 GB.
model is one package with three jobs: the contract between the runner
and the architectures, the opened checkpoint, and building nn layers
from checkpoint tensors. Its files did not say which was which. base.go
carried the folded package's name over the interfaces and the registry,
root.go held the safetensors header scan next to Root, and quant.go
mixed the nvfp4 global-scale helpers with quant parameter resolution.
base.go becomes model.go, named for what it holds. root.go keeps Root
and Open; TensorQuantInfo and the header scan join quant.go, so
everything the checkpoint says about quantization is read and resolved
in one file. The global-scale helpers move to globalscale.go with their
tests. Root.Close, a no-op with one caller, goes. No code changes
otherwise.
nn.go held every layer type in the package apart from attention,
recurrence and rope: the Linear and Embedding interfaces with their dense
and quantized types, Conv1d, RMSNorm, LayerNorm and MultiLinear, with one
test file to match. Finding a layer meant scanning the file named after
the package.
Each layer kind gets its own file: linear.go and embedding.go hold the
interface and the dense and quantized types, conv.go, norm.go and
multilinear.go take the rest, and nn_test.go splits the same way. The
Layer and MultiLinearLayer interfaces go; nothing implemented or accepted
them. No code changes otherwise.
The runner and its weight loader read a model's manifest through
x/imagegen/manifest, the last piece of the removed image generation
engine. It was a hand-rolled copy of the manifest package: its own model
name parser with the default registry and namespace spelled out, its own
blob path builder, and re-spelled media types, plus a model_index.json
reader and other helpers that nothing has called since the engine went.
The manifest package gains the three lookups the runner needs, a config
layer by path, its contents, and the tensor layers, and ReadConfigJSON is
built on the second of them. The weight loader resolves the model name
with the shared parser, which fills in the same defaults the copy did, and
locates blobs with BlobsPath. The architectures' calls to read their
config.json compile unchanged. x/imagegen is gone.
model/base held the Model interface and the architecture registry while
model held weight loading and quant parameters, and every architecture
imported both. create/client was the CLI side of safetensors imports,
with the create command as its only caller, a duplicate of the command's
adapter error, and a name that read like a second API client. The
safetensors show helpers had their own package under x/ although it
already declared package server.
base merges into model, so base.Model and base.Register become
model.Model and model.Register; nothing in the two overlapped. The create
client's two files and their tests join package cmd, and the five names
the command called are no longer exported. The show helpers join the
server package, and the three entry points routes.go calls become
unexported like the helpers around them.
The MLX runner is the only Go inference runner left and is no longer
experimental, so its packages leave x/. The bindings become a top-level
mlx package beside the carried patches in mlx/compat, mirroring how
llama/ holds the llama.cpp integration, and the runner becomes mlxrunner
with the architectures nested under the package they implement.
Subpackages move with their parent unless listed.
x/mlxrunner/mlx mlx
x/internal/mlxthread mlx/mlxthread
x/internal/mlxthreadtest mlx/mlxthread/mlxthreadtest
x/internal/mlxtest mlx/mlxtest
x/quant mlx/quant
mlx/compat/*.patch mlx/compat/mlx-c (MLX patches go in mlx/compat/mlx)
x/mlxrunner mlxrunner
x/models/nn mlxrunner/nn
x/models/<arch> mlxrunner/model/<arch>
x/mlxrunner/imports.go mlxrunner/model/architectures (new package)
x/create create
x/safetensors fs/safetensors
x/tokenizer mlxrunner/tokenizer
Every package keeps its name, so the Go changes are the import path
rewrites the moves force, and the CMake, Dockerfile, CI cache keys, drift
check and Darwin payload script follow the new paths. Four edits are not
paths: the runner's blank architecture imports become the package
mlxrunner/model/architectures, so the list to extend for a new model sits
beside the architecture directories; a depguard rule keeps the two test
harnesses out of non-test code, as the x/internal placement used to; the
CI change filter's two entries for the long-deleted x/imagegen/mlx now
name the bindings' CMake project and the carried patches, so a change to
either builds the payload; and the tokenizer parity test reads its
fixtures from its own testdata instead of walking out of x/.
x/server and x/imagegen/manifest stay for the next two commits.
Several pieces outlived the code that used them. The root tokenizer
package implemented the GGUF-side vocabularies for the Go engine and the
safetensors-to-GGUF converter; nothing has imported it since the
converter went. ml/backend.go held the Go engine's Backend, Context and
Tensor interfaces, with fs.Config existing only to be returned from them,
and a single CUDA template instance under ml/backend/ggml survived the
engine removal along with the gitattributes entries for that tree and the
CI change-filter globs for it and for the long-gone llama/llama.cpp. From
the image generation engine, an integration test group that no test
registers, its build tag, and the StepBar progress widget remained.
DeviceInfo.IsBetter has no caller at all.
All of it goes. Tidying the module file drops the regexp2 dependency and
leaves protobuf as an indirect requirement. The llama3.2 tokenizer
fixtures stay: the MLX runner's tokenizer uses them for its GGML parity
test.
The runner package used to pick an engine from the first argument of the
runner subcommand. Only the MLX engine is left, so the dispatcher has a
single arm, its README still describes the removed Go runner's flags and
endpoints, and the standalone cmd/runner binary exists only to invoke it.
We call mlxrunner.Execute directly from the hidden runner subcommand and
drop the --mlx-engine argument from the command line the MLX client
spawns. Both sides ship in the same binary, so nothing has to accept both
forms. The runner package and cmd/runner are removed.
The subcommand's help hook hands the runner a bare --help. cobra calls the
hook with no arguments for `ollama help runner`, which used to index past
the end of the slice.
