Comparing Inference Engines on DGX Spark - Ollama, vLLM, DS4

Most will tell you it is impossible to run a 284-billion-parameter model on a single NVIDIA DGX™ Spark. But we got DeepSeek V4 (284B) running on a single DGX Spark, a model many opt for by connecting two DGX Sparks.

But should you? A more honest question to ask is less glamorous: Which inference engine should you use for your LLM?

We spent a couple of weeks answering that, benchmarking Ollama, vLLM, and DwarfStar across quality, speed, concurrency, and memory, on real workloads until we had clear numbers.

Ollama vs vLLM vs DS4

We ran these tests to evaluate inference engines for running agents (using Hermes) on the Spark. Our recommendation targets single user with a general-purpose setup use case. We tested three engines, and they sort into three jobs:

• Ollama — For most people. Fits bigger models, single sequential request, and it runs almost anything you throw at it. Its weakness is parallel load — but if you're one person talking to one model, you never feel that.
• vLLM — For scale out multi-tenants. Its superpower is concurrency: it batches many requests and scales hard under load. Not noticeable for a single user and it can't fit the very biggest models on this box, but when deploying a model to multiple users, this is the way to go.
• DwarfStar — A single-model specialist that runs that 284B model the other two can't touch. Incredible for raw capability. However, it can only do so because it runs exactly one model: DeepSeek V4 Flash.

Ollama on the DGX Spark: The Engine Most Use for Single Users

For a single user, Ollama is the default choice. It installs in minutes, runs virtually any model, and fits the largest ones — including 122B-class flagships that vLLM can't load on this box.

Tuning note: The NP=4+q8 boost only applies to transformer MoE models. Mamba-based models (e.g., Nemotron) see no concurrency lift on Ollama, and the largest models are pinned to NP=1 by memory.

vLLM on the DGX Spark: When you Scale to Multiple Requests

vLLM is built for concurrency. Single-request speed is modest with 30 tok/s for Gemma-4-26B, but its superpower is batching many requests simultaneously.

DS4: For Running a 284B Model on NVIDIA DGX Spark

DS4 (DwarfStar, by antirez) is a specialist engine built to run one model: DeepSeek V4 Flash at 284B parameters. It builds from source in under a minute and loads an ~87 GB quantized DeepSeek V4 Flash that fits almost exactly in VRAM (~117 of 119 GB).

It's not a daily driver, but it does prove the ceiling on this machine is far higher than we thought. We can run a huge model parameter-wise, when accounting for MoE and lower active parameters per query.

How to Keep Your DGX Spark from Going Unresponsive

A lot of DGX Spark owners suffer from this problem: If you push a model past what fits, it crashes and goes unresponsive (we learned this the hard way). You’d need to hard reset the system by pulling the plug.

The Spark’s unified memory needs headroom for the OS and background services. We used a memory watchdog that intervenes before hitting the memory limit, plus a hard per-container memory cap so the kernel kills the runaway process instead of the machine.

Which Model Should You Use Under Your Agentic AI Harness?

If you're picking a model for agentic AI tool-calling, real context, and decent speed, here's the shortlist from our testing. We feel that 30 tok/s is a good speed for LLM throughput.

First, the size lens on the Spark, model size trades brain for speed and headroom. An MoE model best fits the NVIDIA DGX Spark's hardware architecture.

• Large (70B–284B) = biggest brain, slowest (~6–25 tok/s), eats memory.
• Medium MoE (~26–34B) = the sweet spot: fast (~30–75 tok/s), smart enough, room to spare.
• Small (≤13B) = fastest (80+ tok/s), but noticeably less capable.

Good picks:

• Qwen3.5-122B (MoE) — Top quality that fits (on Ollama), 64K context, tools work. The flagship default.
• Gemma-4-26B (MoE) — Fast, capable, tools work and the efficient pick.
• Qwen3-Coder-80B (MoE) — strong for coding/heavy tool use.
• Nemotron-120B (Mamba MoE) — 15/15 quality, tools work (with the qwen3_coder parser), batches well on vLLM, and ~26 tok/s single-stream with MTP speculative decoding. Very strong option once you get it running.

Skip, or handle with care:

• DeepSeek V4 Flash (DS4) — the smartest, but on the Spark you can't give it the 64K context an agent framework needs (the model eats the RAM), and it runs only itself.
• DiffusionGemma — fixed 256-token output and genuinely weaker at math/logic/knowledge (a real diffusion trade-off). A fascinating demo, not an agent backend.
• Llama-3.3-70B (great quality, slow + hot) and Llama-3.1-8B (fast but has weak tool call).

The pattern: an MoE with working tools and real context — which points at Qwen3.5-122B or Gemma-4-26B on Ollama for most people. Note that speed is active-parameter dependent: a 70B dense drops to ~6 tok/s, a 120B-class MoE runs ~20–26.

FAQ

What’s the largest model can you run on an NVIDIA DGX Spark?

Comfortably up to 120–122B parameters on Ollama, and even a 284B model (DeepSeek V4 Flash, via the DS4 engine). The real limits are unified memory and how the model's weights are sharded, not raw capability.

vLLM or Ollama on the DGX Spark — which should I use?

• For a single user, Ollama: it fits bigger models, is faster per request, and runs almost any model.
• For many concurrent requests or parallel agents, vLLM — its batching scales aggregate throughput past 300 tok/s.

How many tokens per second does a DGX Spark get?

It depends on the model's active parameters, because the box is memory-bandwidth-bound: a 26B MoE runs ~30–64 tok/s, a 120B-class MoE ~15–26, a 70B dense model ~6. Fewer active params per token = faster.

How to Keep my DGX Spark from Crashing?

To keep our DGX Spark from crashing, we leave headroom for OS memory and run a watchdog to kill applications if they hit a memory threshold. If not, a memory overload crashed the DGX Spark requiring us to pull the plug.

Why does a smaller model sometimes beat a bigger one here?

The DGX Spark is bandwidth-bound. A Mixture-of-Experts model that activates only a few billion parameters per token (e.g., Gemma-4-26B-A4B) outruns a dense 70B that activates all of them. Total size matters less than active parameter size.

Conclusion: When to Use Which Engine & Model

After all of it, the decision is refreshingly simple:

• For most single users, general purpose → Ollama, with a MoE model (Qwen3.5-122B for quality, Gemma-4-26B for speed). It fits the big models, it's faster for single requests, and its only weakness doesn't apply to you.
• Running parallel agents or serving multiple users → vLLM. Its batching turns a single Spark into a real little inference server for a small team.
• You want the absolute biggest brain and can live with one model → DS4. A 284B on your desk for a single user.

And the lesson under all three, the one thing to take away about this machine: on the DGX Spark, memory decides everything; how an engine manages it, and how few parameters a model activates per token. If you are interested in purchasing an NVIDIA DGX Spark (or Sparks), talk to an Exxact sales engineer today! Exxact is an NVIDIA Solution Partner and offer multi-Spark configurations with switches and cables to best fit your deployment style. Contact us today!