CERBERUS/ollama
1
0
Fork 0
mirror of https://github.com/ollama/ollama.git synced 2026-10-20 18:41:17 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions

server: allow parallel requests for qwen35 / qwen35moe

Browse source 
The llama.cpp crash under parallel load on this hybrid architecture
(ggml-org/llama.cpp#20222, "Chunk not found") that motivated blocking
qwen35/qwen35moe here was fixed upstream in llama.cpp PR #20232
(merged 2026-03-08). Ollama vendored llama.cpp b9888, which already
includes the fix, so this restriction is now stale.

Validated empirically on real hardware (RTX 3060, CUDA v13 backend)
across three context sizes (98K/64K/72K) with repeated concurrent
requests and slot reuse: correct, distinct outputs per slot, no
crashes. Also soak-tested in a real production deployment at
OLLAMA_NUM_PARALLEL=3.
This commit is contained in:
robin-selder 2026-07-11 23:22:12 -06:00
parent 82f905cd9c
commit 36e51e15a2
1 changed files with 1 additions and 1 deletions
Download patch file Download diff file Expand all files Collapse all files

2
server/sched.go
View file

@ -510,7 +510,7 @@ func (s *Scheduler) load(req *LlmRequest, systemInfo ml.SystemInfo, gpus []ml.De
// Some architectures are not safe with num_parallel > 1.
// ref: https://github.com/ollama/ollama/issues/4165
if slices.Contains([]string{"mllama", "qwen3vl", "qwen3vlmoe", "qwen35", "qwen35moe", "qwen3next", "lfm2", "lfm2moe", "nemotron_h", "nemotron_h_moe", "nemotron_h_omni"}, req.model.Config.ModelFamily) && numParallel != 1 {
if slices.Contains([]string{"mllama", "qwen3vl", "qwen3vlmoe", "qwen3next", "lfm2", "lfm2moe", "nemotron_h", "nemotron_h_moe", "nemotron_h_omni"}, req.model.Config.ModelFamily) && numParallel != 1 {
numParallel = 1
slog.Warn("model architecture does not currently support parallel requests", "architecture", req.model.Config.ModelFamily)
}