How To Read This Benchmark

The result tables in this post are not meant to be read like a normal model leaderboard. A single score does not tell the whole story.

T1–T17 is the structured agent test suite. It checks whether the model can call the right tools, pass the right arguments, handle malformed outputs, follow formatting rules, resist prompt injection, and track simple state changes. A clean 17/17 is the best score.
- T1–T4 Basic tool calling: Correct tool choice; correct arguments; doesn’t hallucinate tools
- T5–T7 Parallel tool calls: Multiple calls in one turn; result attribution; conflict handling
- T8–T11 Stress inputs: 404s; malformed/partial tool outputs; timeouts; very large payloads
- T12–T15 Instruction adherence: One-call constraints; JSON-only discipline; prompt injection resistance; instruction conflict handling
- T16–T17 Edge cases: Knowing when not to call a tool; state mutation across operations
Hop Depth is the escalating chain test. Each tool result becomes input to the next step, so the model has to preserve context and keep the task moving. This is one of the most important numbers for agent work because real agents rarely stop after one clean tool call. A hop-depth of 4 chain might look like:
- search_places → get_place_details → get_directions → search_parking
Avg Tok/s is speed. We would ideally like to see faster than 20 Tok/s with 40+ Tok/s being the most responsive. It matters, especially for an always-on local agent, but it is not the same thing as reliability. A fast model that breaks the tool contract can still be the wrong choice for an agent.

The best result is the model that balances all three: an Agentic AI that has clean structured test behavior, deep multi-hop stability, and highly usable speed.

The strongest Spark-side model in this run was nemotron-3-super:120b-a12b with a clean 17/17, best hop depth, and strongest agentic profile across the suite.
The Qwen family also held up very well, especially after the benchmark prompt/scoring track stabilized and the T11 timeout path was handled fairly.
All three Gemma models reached clean 17/17 t-score and handled T-11 (timeouts) well.
gemma4:26b was the strongest Gemma result, with hop-depth of 4 at 52.7 tok/s
gemma4:e4b was almost as fast, but stopped after 2 hops
gemma4:31b was clean but much too slow, and also stopped after 2 hops

That makes gemma4:26b the surprise practical result. It did not beat nemotron-3-super:120b-a12b on agent depth, but it matched qwen3.5:35b-a3b on hop-depth while having higher tok/s speed.

Speed vs Reliability

One of the most important conclusions is that the fastest models were not automatically the best agent models. For example, nemotron-3-nano:4b and nemotron-3-nano:30b were very fast, but the remaining benchmark issues clustered in the smaller Nemotron models. gemma4:e4b and gemma4:26b were also fast, but only gemma4:26b combined that speed with a stronger multi-hop result.

That creates a real tradeoff:

smaller models can be operationally attractive for speed
larger models can be much more trustworthy when the task becomes agentic
mid-size models can be the sweet spot when they keep the tool contract clean without giving up too much latency

For an always-on agent, reliability is often more valuable than raw tokens per second.

That is one of the core points of the benchmark.

The Gemma results sharpen that point. gemma4:31b was larger than gemma4:26b, but it was, by far, not better in this benchmark: it was slower and stopped earlier in hop depth. The best Gemma result came from the model with the best balance, not the largest parameter count within the DGX Spark memory limit.

Blackmagic Design OGX-FR-CN-P openGear Frame with Cooling and Advanced Networking