Local inference on a Mac is straightforward until several requests overlap. Then time to first token, memory growth, and admission control become serving problems rather than model-execution problems. vllm-metal brings vLLM’s scheduler, paged KV cache, and OpenAI-compatible server to Apple Silicon, with MLX and Metal handling execution.

v0.4.0 adds batched MTP, broader model and workload support, and automatic prefill acceleration on M5. On SiliconBench’s agent split, vllm-metal keeps TTFT flatter as concurrency rises while serving from a fixed memory budget.

How vllm-metal fits into vLLM

vllm-metal is a plugin, not a fork. Upstream vLLM provides the V1 scheduler, paged KV block management, chunked prefill, sampling, and the OpenAI-compatible frontend with streaming and tool-call parsing. mlx_lm provides the model implementations and MLX executes them. The plugin connects the two, with most of its model-specific code concentrated in one layer.

Architecture overview: clients speak the OpenAI API to upstream vLLM's frontend and V1 scheduler, which hand the vllm-metal model runner a packed step plus block tables; the runner reuses mlx_lm's token-wise layers and adds custom Metal paths for paged varlen attention, MTP, and M5 NAX prefill, all executing through MLX and Metal on Apple Silicon unified memory

Start an OpenAI-compatible server

Install vllm-metal into its own virtual environment and activate it:

curl -fsSL https://raw.githubusercontent.com/vllm-project/vllm-metal/main/install.sh | bash
source ~/.venv-vllm-metal/bin/activate

The installer adds the plugin, vLLM core, and their dependencies to ~/.venv-vllm-metal.

Then launch a model:

# --gpu-memory-utilization caps vLLM's share of unified memory; see below.
vllm serve Qwen/Qwen3.5-0.8B --gpu-memory-utilization 0.5

# 64 GB Macs: the 27B hybrid
# vllm serve mlx-community/Qwen3.8-27B-8bit --gpu-memory-utilization 0.7

# Speculative decoding: Gemma 4 with its MTP assistant
# vllm serve google/gemma-4-E4B-it --gpu-memory-utilization 0.5 \
#   --max-model-len 16384 --no-async-scheduling \
#   --speculative-config '{"method":"mtp","model":"mlx-community/gemma-4-E4B-it-assistant-bf16","num_speculative_tokens":1}'

More models: model matrix. Speculative options: speculative decoding guide.

The server speaks the OpenAI API:

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model": "Qwen/Qwen3.5-0.8B",
       "messages": [{"role": "user", "content": "Say hi"}]}'

Anything that takes an OpenAI-compatible base URL can point at http://localhost:8000/v1, coding agents included; the vLLM docs cover Claude Code and Codex setup.

Set a predictable memory budget

vllm-metal reserves its KV cache at startup and serves every request from that fixed pool. If you are used to runtimes whose memory footprint moves with load, this is the main mental-model change. The key setting is --gpu-memory-utilization: it sets the share of the Mac’s GPU memory budget used for serving.

On Apple Silicon there is no separate VRAM: GPU allocations come from the same unified memory used by macOS, your browser, and your editor. Treat the setting as a serving budget, not a hard cap on the process. As a starting point, 0.5 keeps a laptop usable while it serves; a dedicated machine can go higher.

The reservation buys admission control. The scheduler knows exactly how many KV pages exist, packs requests against that budget, and queues what does not fit, so a burst of requests changes queue depth rather than memory footprint.

At startup, vllm-metal accounts for model weights and temporary buffers before sizing the KV pool, keeping memory predictable as batch shapes change (PR #268).

Concurrent serving under agent load

Agent workloads are the concurrency case. A coding agent fans out tool calls, each one a request with a few thousand tokens of context and a short reply, with several in flight at once. Each round trip pays TTFT before work can continue; end-to-end request latency sets the duration of the turn.

Qwen3.8-27B

We measured this shape with the agent split of SiliconBench, our benchmark harness for LLM inference engines on Apple Silicon: 100 requests averaging 4.6K input and 70 output tokens, run closed-loop at concurrency 1, 2, and 4 against Qwen3.8-27B in 8-bit on an M5 Pro with 64 GB. oMLX appears twice because its default mode spills KV cache to SSD without eviction, which no other engine here does; we report it both with that offload and in its bounded in-memory mode.

SiliconBench agent split on Qwen3.8-27B: TTFT, output token throughput, and end-to-end request latency versus concurrency for llama.cpp, vllm-metal, and oMLX with and without SSD offload

  • At concurrency 1, the three bounded engines sit within 0.7 s on roughly 10-second TTFTs; oMLX’s SSD-offload mode is fastest at 7.8 s.
  • As requests overlap, vllm-metal’s TTFT rises from 10.1 s to 14.7 s while completing all 100 requests at every level. llama.cpp also completes every request, about one second behind at concurrency 4. oMLX’s SSD mode reaches roughly 27 s, while bounded oMLX rejects 37 of 100 requests.
  • At concurrency 4, mean request latency is 52.7 s for vllm-metal, 47.9 s for oMLX’s SSD-offload arm, and 57.6 s for llama.cpp. Chunked prefill shares each engine step between new prefills and active decodes, prioritizing the repeated TTFT costs of short agent replies.

Gemma 4 E4B

Gemma 4 E4B is small enough that the same machine sweeps to concurrency 16, on the same agent split.

SiliconBench agent split on Gemma 4 E4B: TTFT, output token throughput, and end-to-end request latency versus concurrency for llama.cpp, vllm-metal with and without the MTP drafter, and oMLX in its default SSD-offload mode

At concurrency 16, vllm-metal averages 1.9 s TTFT, against 14.7 s for oMLX and 28.3 s for llama.cpp, and generates 63.6 output tokens per second, against 50.4 and 40.2. The dashed MTP arm raises vllm-metal to 71.6 output tokens per second and is analyzed below.

llama.cpp uses its default --parallel 4 configuration; --parallel 16 is not uniformly better (sensitivity results). At concurrency 16 it decodes four streams and queues the other twelve, producing 36.1 s mean request latency. oMLX ran only its default configuration, with SSD KV offload on; there is no bounded-memory arm for this model.

Replacing attention, reusing the rest

At the model level, vllm-metal reuses mlx_lm’s weight loading, RMSNorm, linear, MoE, and MLP layers unchanged. Those layers are token-wise: they process each token independently and do not care how tokens are grouped into sequences, so they run as happily on a packed token axis as on a padded one. Attention is the only layer that needs sequence boundaries, and it is the one we replaced with a custom paged varlen flash-attention Metal kernel.

mlx_lm’s stock attention runs on a contiguous cache of shape [B, H, T, D], with every sequence padded to the length of the longest and prefill and decode handled as separate phases. MLX’s scaled_dot_product_attention accepts no varlen argument.

vllm-metal instead flattens each step onto a single token axis with cu_seqlens marking request boundaries, the packed-query layout vLLM’s unified Triton kernel consumes on NVIDIA. KV remains in fixed-size pages located by per-request block tables, so admitted requests can grow without reshaping a padded cache. One kernel launch covers whatever the V1 scheduler packed into the step: prefill chunks, decode tokens, and speculative-decoding verify windows.

The swap flattens the whole forward pass, not only attention, so the scheduler’s packing reaches every layer. MLP and MoE blocks compute only real tokens, while a padded engine pushes padding rows through the network.

A padded rectangle versus vllm-metal's packed varlen step for a 30,000 + 5,000 + 10 token batch, with KV read from paged storage

Among the serving stacks we audited on Apple Silicon, this pairing of a packed query axis with paged KV storage is what distinguishes vllm-metal:

Engine Encoding Query axes KV
mlx_lm padding [B,T_max] contiguous
oMLX padding [B,T_max] contiguous
llama.cpp mask [total_q] fixed cells
vllm-metal cu_seqlens [total_q] paged

A padded rectangle spends memory and attention time on the longest sequence in the batch whether or not the others need it. A concurrent mix that fits comfortably on a large machine can push a smaller one into macOS memory compression and slow down without an explicit error. Sizing one paged pool up front removes that failure mode.

The Metal kernel ports vLLM’s unified Triton kernel to Apple GPUs.

What v0.4.0 adds

Batched MTP under concurrent load

vllm-metal batches MTP drafting and verification across active requests without leaving the continuous-batching path. The dashed blue line in the Gemma 4 figure is this arm. Change against the vllm-metal baseline:

Concurrency Wall Output tok/s TTFT avg Acceptance
1 +1% −3% +5% 73.4%
8 −21% +23% +15% 73.5%
16 −12% +13% +16% 73.3%

Single-stream, MTP is a wash on this model: the drafter’s cost roughly cancels the accepted tokens. At concurrency 8, it raises output throughput by 23% and cuts mean request latency from 13.7 s to 10.9 s; acceptance stays near 73% across the sweep. Today the Metal MTP path is limited to Gemma 4 and requires --no-async-scheduling; the quickstart above includes both. In this comparison, TTFT rises by about 15% under load. MTP is opt-in through --speculative-config, so prefill-dominated deployments leave it off.

Faster prefill on M5

On M5 Macs, vllm-metal automatically uses the NAX kernel for compatible prefill batches; no extra configuration is required. Pre-M5 Macs keep using the existing path.

NAX prefill kernel A/B benchmark: TTFT, throughput, and TPOT with the tensor units on and off

NAX cuts mean TTFT by 41% on the prefill-heavy split and 26% on the standard split, while total throughput rises 33% and 8%. It also lowers TPOT by 25% and 7% because faster chunked prefill returns time to active decode streams.

Prefix caching on hybrid models

vLLM 0.28 enables prefix caching by default for hybrid models such as Qwen3.5, which must checkpoint recurrent state alongside KV at block boundaries.

v0.4.0 updates that state in place on Metal instead of copying an entire state pool for each row (PR #634). On Qwen3.5-0.8B, shared-prefix traffic improves from 1.03x to 1.26x relative to prefix caching off, while traffic with no shared prefix moves from 2.24x slower back to neutral.

The feature remains experimental on Metal. It requires paged attention and does not yet coexist with speculative decoding; for either unsupported combination, the plugin disables prefix caching and logs the reason instead of refusing to start.

Models, formats, and deployment modes

v0.4.0 also adds:

  • GGUF checkpoints, including Hugging Face config sources for local GGUF weights.
  • Hybrid-attention models: the Qwen3.5, 3.6, and 3.8 family and Qwen3-Next alternate standard attention with gated-delta-net linear attention; v0.4.0 serves mlx-community/Qwen3.8-27B-8bit on a single Mac.
  • Pipeline parallelism across multiple Macs over the MLX ring backend.
  • Experimental vision-language models, text embeddings and reranking, and speech-to-text.

The supported-model matrix and feature guides are in the vllm-metal documentation.

The same stack on DGX Spark

Apple Silicon is one unified-memory target, not the only one vLLM serves. NVIDIA’s DGX Spark also exposes a shared CPU/GPU memory pool and runs the same V1 scheduler, chunked prefill, and paged KV management. The execution layer changes from vllm-metal, MLX, and Metal to upstream vLLM and CUDA.

The two machines balance memory and compute differently:

  Apple M5 Pro DGX Spark (GB10)
Unified memory 64 GB LPDDR5X-9600 128 GB LPDDR5X-8533
Memory bandwidth 307 GB/s 273 GB/s
Nominal scalar shader width (rough) ≈2,560 lanes (20 cores × ≈128) 6,144 CUDA cores (48 SMs × 128)
Serving stack vllm-metal (MLX + Metal) upstream vLLM (CUDA)

Nominal scalar width is a rough architectural comparison, not equivalent FLOPS. Apple publishes the 20-core GPU but not the M5 Pro’s lane count or absolute GPU throughput; the Apple value estimates 128 lanes per core from prior Apple GPU designs. NVIDIA publishes 6,144 CUDA cores. Clock rates, instruction issue, matrix accelerators, and kernel efficiency differ.

The lane counts provide a ballpark parallel-width comparison, not a performance ratio. The Spark exposes roughly 2.4 times as many nominal scalar lanes, while the M5 Pro has slightly higher memory bandwidth. Neural Accelerators and Tensor Cores are omitted because Apple does not publish a comparable throughput figure. The benchmark below is the end-to-end compute comparison.

SiliconBench agent split across three models, vllm-metal on an M5 Pro against upstream vLLM on a DGX Spark: TTFT, output token throughput, and end-to-end request latency versus concurrency

The Spark pulls ahead as concurrency grows. Qwen3.5-0.8B starts within 8% on one stream, 76 against 82 output tokens per second, then reaches 685 against 297 at concurrency 16. Gemma 4 E4B reaches 225 against 64 at the same point. On Qwen3.8-27B, where the Mac uses an 8-bit MLX conversion and the Spark uses Qwen’s FP8 checkpoint, throughput reaches 24.0 against 5.3 at concurrency 4, with TTFT at 0.6 s against 14.7 s.

End-to-end latency shows how that throughput changes request completion. At concurrency 16, Gemma 4 E4B averages 5.5 s per request on the Spark against 21.7 s on the Mac; on the 27B at concurrency 4, the gap is 11.2 s against 52.7 s.

The point is portability, not parity. Both expose the same OpenAI-compatible interface and V1 scheduling model, so the SiliconBench workload runs unchanged while the execution backend and hardware set the performance ceiling.

Reproducing the benchmarks

The cross-engine serving benchmarks use the SiliconBench agent split: 100 prompts averaging 4.6K input and 70 output tokens, run closed loop at fixed concurrency on an Apple M5 Pro with 64 GB running macOS 26.6. The NAX A/B instead uses the two Sonnet configurations described above: 100 prompts at request rate 10 and concurrency 32.

The DGX Spark comparison reuses that split unchanged, on a GB10 box with 121 GB of usable unified memory running the same harness and the same client.

Stats cover completed requests; an empty response counts as failed. The harness and per-engine configurations live in the SiliconBench repo. The MTP arms ran vllm-metal 0.3.0.dev20260821152549 with the serve command from the quickstart.

Serving benchmark reproduction settings
  • llama.cpp: -ngl 99 --parallel 4 -c 49152. The context is divided across slots, giving 12,288 tokens per slot; the agent split’s longest prompt is 8.7K tokens.
  • vllm-metal (27B): VLLM_METAL_MEMORY_FRACTION=0.7 with --max-model-len 16384. Prefix caching was enabled explicitly with --enable-prefix-caching, which these runs predate needing; vLLM 0.28 turns it on for hybrid models by default. At the default fraction of 0.5, the available KV cache held 36,408 tokens and preemption began at concurrency 4.
  • oMLX: --paged-ssd-cache-dir <fresh-empty-dir> and --hot-cache-max-size 0, with a restart and a new directory before each concurrency level. Its default 100 GB prefix cache persists in ~/.omlx/cache, while CLI values persist in ~/.omlx/settings.json. The model directory contained only the target checkpoint because oMLX auto-discovers every entry and returns them in ASCII order. At concurrency 4, its bounded-memory admission guard rejected 37 of 100 requests, so the main chart reports a failure count rather than survivor-only latency.
  • vllm-metal MTP: --no-async-scheduling with "num_speculative_tokens":1. Without the scheduling flag, server health and /v1/models succeed, but inference returns HTTP 500. Values above 1 are ignored rather than rejected, so they do not test a wider speculation window.
  • DGX Spark: upstream vLLM 0.27.2rc1.dev568+gf25c580af built from source against torch 2.13.0+cu130 and flashinfer 0.6.17, on GB10 with driver 580.159.03, CUDA 13.0, Ubuntu 24.04 and kernel 6.17. Same harness, same agent split, same 100 prompts, --max-model-len 16384 with prefix caching on by default. vLLM selects its FLASH_ATTN backend on this GPU, not FlashInfer. The 27B arm serves Qwen/Qwen3.8-27B-FP8, which is not the checkpoint the Mac serves.
  • --gpu-memory-utilization on the Spark: the 27B needs an explicit 0.7, matching the fraction the Mac uses for the same model. The knob behaves exactly as the memory section above describes, which is the problem: GB10 shares one 121 GB pool between the GPU and the operating system, so vLLM’s default 0.9 reserves around 109 GB of it. Small models absorb that (Qwen3.5-0.8B took a 102 GiB KV cache and served fine), but 28.5 GiB of 27B weights on top of it exhausted the machine, the driver returned NV_ERR_NO_MEMORY, and the desktop session died. At 0.7 the KV pool holds 619,557 tokens, 37x the 16K budget of a single request, so nothing is preempted at concurrency 4.
llama.cpp server-slot sensitivity

llama.cpp defaults to four server slots, which is the configuration in the main figures. A 16-slot sensitivity run improves output throughput at concurrency 16 but regresses at concurrency 8:

Split Concurrency --parallel 4 --parallel 16 Change
Chat 1 22.4 tok/s 24.2 tok/s +8%
Chat 8 77.0 tok/s 42.7 tok/s −45%
Chat 16 81.5 tok/s 104.7 tok/s +28%
Agent 1 18.3 tok/s 19.0 tok/s +4%
Agent 8 44.2 tok/s 26.7 tok/s −40%
Agent 16 40.2 tok/s 49.7 tok/s +24%

Acknowledgments

vllm-metal builds on MLX and mlx_lm from Apple’s MLX team, on mlx-vlm for the vision-language paths, and on the vLLM engine and its hardware-plugin interface. Thanks to the upstream vLLM maintainers for review and support along the way, and to everyone who filed issues and shared benchmarks against the v0.2 and v0.3 releases.