Model Benchmark Report

Comprehensive benchmark results across all tested models, hosts, and configurations. Generated: 2026-08-25 22:44 (Granite 4.1 results) Updated: 2026-09-01 18:45 (Granite 4.2 upgrade notes added)

1. Hardware Inventory

Host GPUs VRAM/GPU Memory Bandwidth/GPU Total Bandwidth Memory Type
rhel-ai 4× NVIDIA L4 21.95 GB 300 GB/s 1,200 GB/s GDDR6
rhtevan-work 1× NVIDIA RTX A500 4 GB 128 GB/s 128 GB/s GDDR6

Key constraint: Both hosts use GDDR6, not HBM. Memory bandwidth — not compute — is the bottleneck for autoregressive LLM inference. Every output token requires reading the entire model weights from GPU memory once:

Theoretical max tok/s ≈ Total Bandwidth (GB/s) ÷ Model Size in Memory (GB)

2. Benchmark Suite (7 Tests)

# Test What It Measures Max Tokens
1 Instruction Following Strict format compliance (exactly 5 items, numbered, no intro/outro) 500
2 Reasoning Step-by-step arithmetic (64 GB RAM allocation) 1,000
3 Tool Calling Proper function call parsing via OpenAI tool_calls API 500
4 Code Generation Python function with docstring, edge case handling 1,000
5 Multi-step Reasoning K8s scheduling decision with capacity + balance analysis 1,000
6 Conciseness One-sentence answer (measures verbosity control) 200
7 Speed / Long Generation Extended structured output (Prometheus monitoring guide) 2,000

All tests run with thinking/reasoning disabled for consistent comparison. Tool calling tested with OpenAI-compatible function calling schema.

3. Results — rhtevan-work (1× RTX A500, 4 GB)

Runtime: llama.cpp (GGUF quantized models)

Metric Granite 350M (BF16) Granite 3B (Q4_K_M) Granite 8B (Q4_K_M)
Parameters 350M 3B 8B
Model size in VRAM ~0.7 GB ~1.7 GB ~4.5 GB (partial offload)
Context window 2K 16K 4K
Slots (concurrent) 4 1 1
Token speed 44 tok/s 37 tok/s 6.5 tok/s

Quality Results

Test Granite 350M Granite 3B Granite 8B
1. Instruction Follow ❌ Too small ✅ Pass ✅ Pass
2. Reasoning ❌ Too small ✅ 8 GB correct ✅ 8 GB correct
3. Tool Calling ❌ Too small ✅ Parsed ✅ Parsed
4. Code Gen ❌ Too small ✅ Clean, with docstring ✅ Clean, with docstring
5. Multi-step ❌ Too small ✅ Node B correct ✅ Node B correct
6. Conciseness ❌ Too small ✅ Concise (25 tok) ✅ Concise
7. Speed (long gen) 44 tok/s (no quality) 37 tok/s 6.5 tok/s
Verdict ❌ Unusable ✅ Production-grade ✅ Production-grade

Analysis

4. Results — rhel-ai Solo Deployments (4× L4, TP=4)

Runtime: vLLM nightly (–enforce-eager, 128K context)

Metric Qwen3.8-27B FP8 Gemma 4 31B FP8-block
Parameters 27B 31B
Quantization FP8 FP8-block
Model size in VRAM ~14 GB (3.5 GB/GPU) ~31 GB (7.75 GB/GPU)
Context window 128K 128K
Token speed 11.8 tok/s 13.4 tok/s

Quality Results (Thinking Disabled)

Test Qwen3.8-27B FP8 Gemma 4 31B FP8-block
1. Instruction Follow ✅ 5 items correct ✅ 5 items correct
2. Reasoning ✅ 8 GB, step-by-step (331 tok) ✅ 8 GB, step-by-step (243 tok)
3. Tool Calling ✅ Parsed, 3.4s latency ✅ Parsed, 1.2s latency
4. Code Gen ✅ Clean (168 tok) ✅ Thorough (720 tok)
5. Multi-step ✅ Node B, table (832 tok) ✅ Node B, table (569 tok)
6. Conciseness ⚠️ 36 tok (wordy) ✅ 28 tok (tight)
7. Speed (long gen) 11.8 tok/s (2000 tok) 13.4 tok/s (1358 tok)
Verdict ✅ Excellent ✅ Excellent

Gemma 4 Advantages Over Qwen3.8

5. Qwen3.8 Thinking Mode Issue

When Qwen3.8 runs with thinking enabled (default behavior), internal reasoning tokens consume the max_tokens budget before producing visible content:

Test Reasoning Tokens Content Tokens Visible Output
4. Code Gen (max=2000) 2000 0 ❌ NULL
7. Speed (max=3000) 3000 0 ❌ NULL
2. Reasoning (max=2000) 73 135 ✅ Works
5. Multi-step (max=2000) 355 543 ✅ Works

Root cause: Complex tasks trigger extended thinking chains that exhaust the token budget. Workaround: set chat_template_kwargs.enable_thinking: false or use very high max_tokens (16K+).

Gemma 4 does not have this problem — it does not have built-in thinking mode.

6. Speculative Decoding Results — rhel-ai

6.1 N-gram Speculation (Gemma 4 31B)

Config: --speculative-config '{"method": "ngram", "num_speculative_tokens": 5, "prompt_lookup_max": 4}'

Test Baseline (tok/s) N-gram (tok/s) Delta
1. Instruction Follow 8.9 4.1 ❌ 2× slower
2. Reasoning 13.4 13.8 ~Same
3. Tool Calling 12.7 12.9 ~Same
4. Code Gen 13.4 14.0 +4%
5. Multi-step 13.4 13.6 ~Same
6. Conciseness 13.0 12.9 ~Same
7. Speed (long gen) 13.4 13.7 +2%

Verdict: ❌ No benefit. N-gram speculation pattern-matches against prior text for repetitive output. Agentic workloads produce novel content — low acceptance rate makes speculation overhead a net negative.

6.2 Native MTP (Gemma 4 31B)

Config: --speculative-config '{"method": "gemma4_mtp", "num_speculative_tokens": 2}'

Verdict: ❌ Not applicable. The RedHatAI/gemma-4-31B-it-FP8-block checkpoint does not contain MTP heads (num_mtp_modules not present in config). The gemma4_mtp method exists in vLLM but requires a model trained with MTP modules.

6.3 Draft Model Speculation (Granite 8B + 3B Draft) ✅

Config: --speculative-config '{"method": "draft_model", "model": "ibm-granite/granite-4.1-3b", "num_speculative_tokens": 5, "draft_tensor_parallel_size": 4}'

Test Granite 8B + 3B Draft (tok/s) Gemma 4 Baseline (tok/s) Qwen3.8 Baseline (tok/s)
1. Instruction Follow 10.3 8.9 11.7
2. Reasoning 25.0 13.4 11.8
3. Tool Calling 20.3 12.7 11.3
4. Code Gen 25.5 13.4 11.8
5. Multi-step 22.8 13.4 11.8
6. Conciseness 17.8 13.0 11.5
7. Speed (long gen) 18.7 13.4 11.8

Verdict: ✅ Significant speedup. Draft model speculation delivers ~1.5-2× speedup on structured/reasoning tasks (25 tok/s vs ~13 tok/s baseline). Acceptance rate is high because Granite 3B and 8B share the same architecture and tokenizer.

Constraint discovered: vLLM nightly requires draft_tensor_parallel_size to match target tensor_parallel_size. The draft model cannot run on fewer GPUs — it runs TP=4 alongside the target.

6.4 FP8 + CUDA Graphs + Draft Speculation (Granite 8B FP8 + 3B FP8 Draft) ✅✅

Config: FP8 quantized target + FP8 draft, CUDA graphs enabled (no --enforce-eager), num_speculative_tokens: 5, draft_tensor_parallel_size: 4

Test BF16 + enforce-eager (tok/s) FP8 + CUDA graphs (tok/s) Speedup
1. Instruction Follow 10.3 22.8 2.2×
2. Reasoning 25.0 76.0 3.0×
3. Tool Calling 20.3 47.9 2.4×
4. Code Gen 25.5 79.2 3.1×
5. Multi-step 22.8 72.3 3.2×
6. Conciseness 17.8 32.1 1.8×
7. Speed (long gen) 18.7 58.2 3.1×

Verdict: ✅✅ Breakthrough result. Three optimizations stacked multiplicatively:

  1. FP8 quantization — halved weight reads (16 GB → 8 GB target, 6 GB → 3 GB draft)
  2. CUDA graphs — eliminated Python/framework overhead per step
  3. Speculative decoding — multiple tokens verified per weight read

GPU utilization during inference: 97-100% compute, 93.8% VRAM (21.6/23 GB per GPU). Tool call latency: 0.56 seconds. 128K context preserved.

This config achieves comparable or faster speed than Opus 4.6 cloud for agentic workloads (tool calls: 62 tok/s vs 26 tok/s). Cloud Opus is faster only on sustained long generation (67 vs 60 tok/s). See Section 12.

7. Failed Experiments

Experiment Why It Failed
Gemma 4 31B BF16 @128K OOM — 78 GB model + 16 GB KV > 89 GB (4× L4)
Gemma 4 31B FP8 without –enforce-eager OOM — CUDA graph overhead ~10 GB/GPU
Gemma 4 31B gemma4_mtp speculation Checkpoint lacks MTP heads
Granite 8B + 3B draft with draft_tp=1 vLLM requires draft_tp == target_tp
Granite 2B as draft (ibm-granite/granite-4.1-2b) Model repo does not exist on HuggingFace
Granite 3B Q4_K_M @32K on RTX A500 OOM — KV cache needs 2.5 GB, only 0.3 GB free
N-gram speculation on Gemma 4 No speedup — novel content has low n-gram match rate
Qwen3.8 with thinking enabled (max_tokens ≤ 3000) Reasoning tokens exhaust budget, NULL content output

8. Complete Speed Comparison

Model Host GPUs Quant TP Context Spec Decode tok/s Quality
Granite 350M rhtevan-work 1× A500 BF16 1 2K — 44 ❌ Unusable
Granite 3B rhtevan-work 1× A500 Q4_K_M 1 16K — 37 ✅ Pass all
Granite 8B rhtevan-work 1× A500 Q4_K_M 1 4K — 6.5 ✅ Pass all
Granite 8B + 3B draft rhel-ai 4× L4 BF16 4 128K draft_model 19-25 ✅ Pass all
Granite 4.1 8B FP8 + 3B FP8 draft rhel-ai 4× L4 FP8 4 128K draft_model + CUDA graphs 58-79 ✅ Pass all
Granite 4.2 8B FP8 + 3B FP8 draft rhel-ai 4× L4 FP8 4 128K draft_model + CUDA graphs ~50 (47 content) ✅ Pass all
Gemma 4 31B rhel-ai 4× L4 FP8-block 4 128K — 13.4 ✅ Pass all
Gemma 4 31B rhel-ai 4× L4 FP8-block 4 128K ngram 13.5 ✅ Pass all
Qwen3.8-27B rhel-ai 4× L4 FP8 4 128K — 11.8 ✅ Pass all
Qwen3 14B FP8 + 4B FP8 draft rhel-ai 2× L4 FP8 2 32K draft_model + CUDA graphs 36.4 total (23.5 content) ✅ Pass all
Qwen3 14B FP8 + 4B FP8 draft rhel-ai 4× L4 FP8 4 40K draft_model + CUDA graphs 57.8 total (41.2 content) ✅ Pass all
Opus 4.6 Cloud (GCP Vertex) — — — — — 17-67 ✅ Reference

9. Key Findings

9.1 Memory Bandwidth Is the Bottleneck

L4 GPUs at 300 GB/s/GPU cap 27-31B FP8 models at ~13 tok/s (TP=4). This is a hardware wall — no software optimization can exceed:

1,200 GB/s ÷ 31 GB weights = ~38 tok/s theoretical
1,200 GB/s ÷ 16 GB weights = ~75 tok/s theoretical

Actual efficiency is 35-50% of theoretical due to TP communication overhead, attention computation, and Python/framework overhead.

9.2 Smaller Models Win on Speed

Granite 3B on a $200 laptop GPU (37 tok/s) is nearly 3× faster than Gemma 4 31B on $10,000 worth of L4 GPUs (13.4 tok/s). Both pass all 7 benchmark tests.

The 31B model produces higher quality on complex tasks, but for routine agentic work (tool calling, simple reasoning, code gen), the 3B is indistinguishable.

9.3 Speculative Decoding: Hardware-Dependent (Updated 2026-10-15)

Updated finding: Spec decode benefit depends critically on GPU architecture and vLLM engine version. On L4 GPUs (bandwidth-constrained, no native FP8, PCIe interconnect), vLLM 0.32 V1 engine makes spec decode net-negative. Removing it from Qwen3 14B profiles yielded +24% throughput (51.3 vs 41.2). See Section 16 for full analysis.

9.4 TP Does Not Increase Token Speed

TP spreads model weights so they fit across multiple GPUs, but each GPU still reads its shard through the same bandwidth pipe. More GPUs = model fits, not model runs faster. TP adds all-reduce communication overhead.

9.5 Co-hosting Was Explored But Not Practical

Co-hosting two TP=2 models was evaluated but not adopted. It provides model diversity but drops large model speed to ~8 tok/s on GDDR6 — below usability threshold. Speculative decoding on a single 8B model (58-79 tok/s) proved far more effective than co-hosting a slow 27-31B model at 8 tok/s.

10. Recommendations

Default Deployment: rhtevan-work — Profile g3b-16k

Setting Value
Model Granite 4.1 3B (Q4_K_M GGUF)
Runtime llama.cpp
Context 16K
Slots 1 (single-user agentic workload)
Speed ~37 tok/s
Why Best speed-to-quality ratio. Passes all 7 tests. 5.7× faster than 8B on same hardware. 16K context is hardware ceiling.

Default Deployment: rhel-ai — Profile g8b-fp8-spec-128k

Setting Value
Model Granite 4.1 8B FP8 + Granite 4.1 3B FP8 draft
Runtime vLLM nightly
TP 4 (both target and draft)
Context 128K
Speculative Decoding draft_model, 5 speculative tokens, CUDA graphs enabled
Speed 58-79 tok/s (task dependent)
GPU Utilization 94% VRAM, 97-100% compute during inference
Why FP8 + CUDA graphs + speculative decoding achieves 58-79% of Opus cloud speed. 128K context. Same Granite family as rhtevan-work for consistent behavior.

Alternate Profile: rhel-ai — g8b-spec-128k

Setting Value
Model Granite 4.1 8B BF16 + Granite 4.1 3B BF16 draft
Speed 19-25 tok/s
When If FP8 quality is insufficient for a specific task, or CUDA graph compilation overhead is problematic

On-Demand Models (Not Default Profiles)

Alias Model Speed When to Use
gemma4-31b Gemma 4 31B FP8-block 13.4 tok/s Complex multi-hop reasoning requiring 31B quality
qwen38-27b Qwen3.8-27B FP8 11.8 tok/s Qwen-ecosystem compatibility, thinking mode

Deploy via setup.sh <alias> — stops current model on port 9000 first.

When to Use Cloud (Opus 4.6)

Updated Default: rhel-ai — Profile qw14b-40k ✅ (2026-10-15)

Setting Value
Model Qwen3-14B-FP8
Runtime vLLM 0.32 nightly
TP 4 (all GPUs)
Context 40K
Speculative Decoding ❌ Disabled (net-negative on L4)
Speed 51.3 tok/s
Why Higher quality than Granite 8B, faster than Granite 8B with spec decode (51 vs 47 content), decision endpoints via NextTokenStrategy

Alternate: rhel-ai — Profile qw14b-32k

Setting Value
Model Qwen3-14B-FP8
TP 2 (GPU 0-1 only)
Speed 30.2 tok/s
When GPU 2-3 needed for co-hosted decision model or other workload

11. Deployment Commands

rhtevan-work (default)

bash ~/.agents/skills/hosted-model-ctl/scripts/setup.sh g3b-16k
# or directly:
bash ~/.agents/skills/hosted-model-ctl/scripts/setup.sh g3b

rhel-ai (default — FP8 speculative decoding)

bash ~/.agents/skills/hosted-model-ctl/scripts/setup.sh g8b-fp8-spec-128k
# or directly:
bash ~/.agents/skills/hosted-model-ctl/scripts/setup.sh g8b-fp8-spec

rhel-ai (BF16 speculative decoding — safer, slower)

bash ~/.agents/skills/hosted-model-ctl/scripts/setup.sh g8b-spec-128k

rhel-ai (alternate models — on demand)

bash ~/.agents/skills/hosted-model-ctl/scripts/setup.sh gemma4-31b
bash ~/.agents/skills/hosted-model-ctl/scripts/setup.sh qwen38-27b

12. Self-Hosted vs Cloud Provider Comparison

Measured: 2026-08-26. Non-streaming requests via OpenAI-compatible API. Self-hosted: Granite 8B FP8 + 3B FP8 draft (spec decode, CUDA graphs) via Skupper VAN on localhost:9000. Cloud: Claude Opus 4.6 via LiteLLM proxy → GCP Vertex AI on localhost:4000.

12.1 Raw Results

Test Max Tokens Granite 8B FP8   Opus 4.6 (Cloud)  
    tok/s time tok/s time
Short text 50 58.2 0.39s 17.5 1.89s
Tool call 200 62.1 0.34s 26.4 2.54s
Medium text 200 54.9 3.65s 46.8 4.28s
Long generation 1,000 60.4 16.56s 66.6 15.02s

12.2 Analysis

Latency profile is inverted. Self-hosted wins on short/interactive workloads (3.3× faster on short text, 2.4× on tool calls). Cloud wins on sustained long generation (1.1× faster at 1,000 tokens).

Metric Granite 8B FP8 Opus 4.6 Winner
Tool call latency 0.34s 2.54s Granite (7.5×)
Short response latency 0.39s 1.89s Granite (4.8×)
Sustained throughput (1K tok) 60.4 tok/s 66.6 tok/s Opus (1.1×)
10-step agentic chain (est.) ~3.4s ~25s Granite (7.4×)

Why the crossover happens:

12.3 Agentic Workload Implications

For typical agentic patterns (many short tool calls, occasional long text generation), self-hosted Granite 8B FP8 with speculative decoding is significantly faster wall-clock than cloud Opus:

12.4 When to Use Which

Use Case Recommended Why
Routine agentic work (file ops, shell, tool calling) Granite 8B 7× lower tool-call latency, sufficient quality
Complex reasoning / architecture decisions Opus 4.6 Frontier-class reasoning required
Long document generation Opus 4.6 Slightly faster sustained throughput
Cost-sensitive batch operations Granite 8B ~75× cheaper per session
Benchmarking / quality baseline Opus 4.6 Reference-grade output quality

13. Granite 4.2 Upgrade Results (2026-09-01)

Granite 4.1 8B FP8 upgraded to Granite 4.2 8B FP8 on g8b-fp8-spec-128k profile. Parser changes: --tool-call-parser hermes to qwen3_coder, added --reasoning-parser nemotron_v3. Draft model: granite-4.2-3b-fp8 (also upgraded from 4.1).

13.1 Quality Results (7 Tests, max_tokens=8192)

Test 4.1 Result 4.2 Result Winner
1. Instruction Follow Pass Pass (5 items, concise) Tie
2. Reasoning Pass (8 GB) Pass (8 GB, richer analysis) 4.2
3. Tool Calling Pass (hermes) Pass (qwen3_coder, reasoning separated) Tie
4. Code Gen Pass (1000 tok) Pass at 8192 only (budget exhaustion at 1000/4096) 4.1 (no budget issue)
5. Multi-step Pass (Node B) Pass (Node B, superior analysis with capacity + load detail) 4.2
6. Conciseness Pass (25 tok) Pass (28 tok) Tie
7. Speed (long gen) 58-79 tok/s 49.6 tok/s total, 47.2 content 4.1 (faster)

13.2 Speed Comparison

Workload 4.1 tok/s 4.2 total tok/s 4.2 content tok/s Reasoning %
Short text (say hello) ~58 ~64 ~7 89%
Tool call ~62 ~77 ~22 71%
Medium (1000 tok) ~60 ~44 ~21 52%
Long generation (8192 tok) ~58 ~50 ~47 5%

Key finding: reasoning overhead is inversely proportional to output length. Short responses spend 70-90% on invisible thinking. Long generation only 5%.

13.3 Thinking Budget Exhaustion

Granite 4.2 reproduces the same problem observed with Qwen3.8 (Section 5): complex tasks exhaust the max_tokens budget on reasoning before producing content.

Test max_tokens=1000 max_tokens=4096 max_tokens=8192
Code Gen NULL (1000 reasoning) NULL (4096 reasoning) Pass (3227 reasoning + 806 content)

Workarounds:

13.4 rhtevan-work Decision: Stay on 4.1

Granite 4.2 3B Q4_K_M was evaluated for rhtevan-work (1× RTX A500, 4 GB) but rejected:

Factor 4.1 4.2 Verdict
GGUF size 2.10 GB 2.24 GB +145 MB
VRAM headroom 0.26 GiB 0.14 GiB OOM risk
Reasoning parser N/A (no reasoning) N/A (llama.cpp has none) Tags leak into content
3B reasoning quality N/A Marginal Not worth latency cost

Decision: rhtevan-work g3b-16k stays on Granite 4.1 3B Q4_K_M. The 4.2 upgrade is rhel-ai only (vLLM with proper parser support and sufficient VRAM).

13.5 Compatibility

Client Status Notes
vLLM API (curl) Pass qwen3_coder tool parser + nemotron_v3 reasoning parser
Goose 1.48 (Skupper provider) Pass Thinking blocks displayed, tool calls work
OpenShift LightSpeed (CRC) Pass OLSConfig model name updated
Skupper VAN Pass Model-agnostic, no changes needed

14. Qwen3 14B FP8 + 4B FP8 Draft — Profile qw14b-32k (2026-10-15)

First non-Granite deployment profile. Tests 14B model with speculative decoding on a GPU subset (2× L4), leaving GPU 2+3 free for future Decision model co-hosting.

14.1 Configuration

Setting Value
Target model Qwen/Qwen3-14B-FP8 (FP8, ~14 GB)
Draft model Qwen/Qwen3-4B-FP8 (FP8, ~4 GB)
Runtime vLLM nightly (v0.26.1rc1)
TP 2 (GPU 0 + GPU 1 only, via CUDA_VISIBLE_DEVICES=0,1)
Context 32,768 (32K)
Speculative decoding draft_model, 5 speculative tokens
CUDA graphs Enabled
Tool call parser hermes
Reasoning parser qwen3
Port 9000

14.2 GPU Layout

GPU 0: 21,147 / 23,034 MiB (91.8%) — Qwen3 14B FP8 shard + draft shard
GPU 1: 21,147 / 23,034 MiB (91.8%) — Qwen3 14B FP8 shard + draft shard
GPU 2:      1 / 23,034 MiB (idle)  — FREE for Decision model
GPU 3:      1 / 23,034 MiB (idle)  — FREE for Decision model

14.3 Parser Discovery

Tool call parser required trial and error:

Reasoning parser: qwen3 separates <think>...</think> blocks into reasoning_content field. Without it, thinking tokens leak into content (same issue as Granite 4.2 without nemotron_v3).

14.4 Quality Results (7 Tests, max_tokens=8192)

Test Result Content Tok Reasoning Tok Total Tok
1. Instruction Follow ✅ 5 items, numbered, no intro 68 147 215
2. Reasoning ✅ 8 GB correct, LaTeX steps 165 434 599
3. Tool Calling ✅ Parsed (hermes), 2.4s — 69 —
4. Code Gen ✅ Docstring, ValueError, iterative 466 1,469 1,935
5. Multi-step ✅ Node B, structured capacity analysis 196 266 462
6. Conciseness ✅ 34 tok, one sentence 34 227 261
7. Speed (long gen) 36.4 total / 23.5 content tok/s 2,038 1,116 3,154

Verdict: ✅ Excellent quality. All 7 tests pass. Output is richer and more structured than Granite 4.2 8B on tests 2, 4, and 5. Reasoning overhead is proportionally higher (~35% of total tokens on long generation vs Granite’s ~5%).

14.5 Speed Comparison

Metric g8b-fp8-spec (Granite 4.2, 4×L4) qw14b-32k (Qwen3 14B, 2×L4)
Total tok/s (long gen) ~50 36.4
Content tok/s (long gen) ~47 23.5
Tool call latency 0.56s 2.4s
GPUs consumed 4 2
VRAM per GPU 21.6 / 21.95 GB 21.1 / 23.0 GB

Speed is lower on both total and content tok/s. The primary cause is half the memory bandwidth (600 GB/s vs 1,200 GB/s) rather than model size. Tool call latency (2.4s) is 4.3× slower than Granite’s 0.56s — the reasoning overhead on short outputs is significant.

14.6 Reasoning Token Analysis

Test Reasoning % of Total Pattern
Short (conciseness) 87% High overhead — model thinks extensively before one sentence
Tool call ~80% Reasoning before generating tool call XML
Multi-step 58% Balanced — significant reasoning on complex task
Long generation 35% Low overhead — content dominates on extended output

Same pattern as Granite 4.2: reasoning overhead is inversely proportional to output length. Short responses are disproportionately penalized.

14.7 Fitting Score (Actual Measured)

Trait g8b-fp8-spec (current) qw14b-32k (measured)
Quality 60 70
Speed 50 24
Context 12 16
Size 27 55
Score 39.3 38.0
Band Gap analysis Gap analysis

Nearly identical scores. Quality and Size gains are offset by Speed loss. The framework correctly identifies these as equivalent trade-offs.

14.8 Strategic Value

The score doesn’t capture the architectural option qw14b-32k enables: GPU 2+3 are free for a co-hosted Decision model. The aggregate system value (LLM on GPU 0-1 + SDM on GPU 2-3) may exceed a single LLM on all 4.

14.9 Failed Parser Attempts

Parser Result
--tool-call-parser qwen3 ❌ Not a valid parser name
--tool-call-parser qwen3_coder ❌ Tool calls generated but not extracted
--tool-call-parser hermes ✅ Works — Qwen3 uses Hermes tool call format
No --reasoning-parser ❌ <think> tags leak into content
--reasoning-parser qwen3 ✅ Separates reasoning into reasoning_content

15. Qwen3 14B FP8 + 4B FP8 Draft — Profile qw14b-40k (2026-10-15)

Same model as qw14b-32k but on all 4× L4 GPUs with TP=4 and 40K context (the model’s max_position_embeddings hard limit).

15.1 Configuration

Setting Value
Target model Qwen/Qwen3-14B-FP8 (FP8, ~14 GB)
Draft model Qwen/Qwen3-4B-FP8 (FP8, ~4 GB)
Runtime vLLM nightly (v0.26.1rc1)
TP 4 (all GPUs)
Context 40,960 (40K — model hard limit)
Speculative decoding draft_model, 5 speculative tokens
CUDA graphs Enabled
Tool call parser hermes
Reasoning parser qwen3
Port 9000

15.2 Context Window Discovery

Original profile was qw14b-64k with 65,536 context. Deployment failed:

ValueError: User-specified max_model_len (65536) is greater than the
derived max_model_len (max_position_embeddings=40960.0)

Qwen3-14B config.json: max_position_embeddings: 40960, no RoPE scaling. The model is a 40K model, not 128K. Profile renamed to qw14b-40k.

This is consistent with the Qwen3 technical report — the 14B dense model uses standard attention without YaRN or other context extension techniques. Only Qwen3.6+ and Qwen3.8 models have 262K+ native context.

15.3 GPU Layout

GPU 0: 21,523 / 23,034 MiB (93.4%) — Qwen3 14B FP8 shard + draft shard
GPU 1: 21,523 / 23,034 MiB (93.4%)
GPU 2: 21,523 / 23,034 MiB (93.4%)
GPU 3: 21,523 / 23,034 MiB (93.4%)

15.4 Quality Results (7 Tests, max_tokens=8192)

Test Result Content Tok Reasoning Tok Total Tok
1. Instruction Follow ✅ 5 items, numbered, no intro 71 146 217
2. Reasoning ✅ 8 GB correct, LaTeX steps 183 722 905
3. Tool Calling ✅ Parsed (hermes), 2.0s — — —
4. Code Gen ✅ Docstring, ValueError, iterative 456 941 1,397
5. Multi-step ✅ Node B, structured capacity analysis 333 478 811
6. Conciseness ✅ 29 tok, one sentence 29 153 182
7. Speed (long gen) 57.8 total / 41.2 content tok/s 2,076 836 2,912

15.5 Speed Comparison (qw14b-32k vs qw14b-40k)

Metric qw14b-32k (2×L4) qw14b-40k (4×L4) Improvement
Total tok/s 36.4 57.8 +59%
Content tok/s 23.5 41.2 +75%
Tool call latency 2,424ms 2,010ms -17%
Context 32K 40K +25%
VRAM per GPU 21.1 / 23.0 GB 21.5 / 23.0 GB Similar

Speed improvement is primarily from doubled memory bandwidth (1,200 GB/s vs 600 GB/s). TP=4 has more all-reduce communication overhead than TP=2, but the bandwidth gain more than compensates.

15.6 Agentic Session Observations

Deployed as default profile and tested in real agentic sessions:

Strengths:

Weaknesses (14B model limitation):

Assessment: Strong on structured tasks within skill framework. Weak on unstructured novel tasks requiring improvised tool chains. Characteristic 14B limitation vs frontier models.

15.7 Fitting Score (Actual Measured + Agentic Session Data)

Trait Score Rationale
Quality 60 Structured: 75 (skill execution). Unstructured: 50 (hallucination). Blended: 60
Speed 41 41.2 content tok/s vs 100 baseline
Context 20 40K / 200K baseline
Size 27 88 GB (4× L4) / 24 GB baseline
Total 39.0 Gap analysis

16. vLLM 0.32 Upgrade + Speculative Decoding Removal (2026-10-15)

Upgraded vLLM from nightly v0.26.1 to v0.32.0rc1. During benchmarking, discovered that speculative decoding reduces throughput on L4 GPUs under the V1 engine. Both qw14b profiles switched from vllm-spec to vllm engine (spec decode disabled).

16.1 Root Cause Analysis

vLLM 0.32 introduces the V1 engine by default. Speculative decoding on bandwidth-constrained GPUs (L4, Ada sm_89) is net-negative due to:

  1. Scheduler auto-throttle: V1 caps max_num_scheduled_tokens=2048 when spec decode is active. --max-num-batched-tokens 8192 changes compile range but does not recover throughput.
  2. Draft TP overhead: 4B draft model at TP=4 → 1B/GPU, dominated by NCCL all-reduce latency. At TP=2 → 2B/GPU, still excessive.
  3. Acceptance rate: 56.7% overall (pos 0: 79%, pos 4: 39%). Average 2.83 tokens accepted per draft cycle — overhead exceeds gain.
  4. No native FP8: L4 uses Marlin weight-only FP8 (dequant to FP16). Draft model pays same penalty.

16.2 Benchmark Results — qw14b-40k (4×L4)

Configuration tok/s vs old baseline
vLLM 0.27 + spec decode (old) 41.2 —
vLLM 0.32 + spec decode 32.5 -21%
vLLM 0.32 + spec decode + --max-num-batched-tokens 8192 32.5 -21%
vLLM 0.32, no spec decode 51.3 +24%

Six-run benchmark (3× no-thinking, 3× thinking):

16.3 Benchmark Results — qw14b-32k (2×L4)

Configuration tok/s
vLLM 0.32, no spec decode 30.2

Six-run benchmark (3× no-thinking, 3× thinking):

GPU layout:

GPU 0: 21,893 / 23,034 MiB (95.1%) — Qwen3 14B FP8 shard
GPU 1: 21,893 / 23,034 MiB (95.1%) — Qwen3 14B FP8 shard
GPU 2:      1 / 23,034 MiB (idle)  — FREE
GPU 3:      1 / 23,034 MiB (idle)  — FREE

16.4 Agentic Session Performance (qw14b-40k, no spec decode)

Measured from Goose Test qw14b-40k session via Skupper VAN:

Interaction Time (no spec) Time (with spec, prev) Improvement
hello → canary response 5s 30s 83% faster
Identity question 14s 11s Similar
Violation check (e2e, 4 LLM calls) 32s 57s 44% faster

16.5 Decision Endpoint Verification

NextTokenStrategy decision endpoints work on both profiles without spec decode. No extra flags needed — vLLM 0.32 enables them by default.

Test Result
/v1/decisions predicate ✅ probability=1.0 (sky is blue)
/v1/decisions choice ✅ “blue” with confidence=1.0
3-question batched ✅ All correct, 1079ms

16.6 Updated Fitting Scores

Profile Q S C Z Score Change
qw14b-32k 70 30 16 55 40.3 was 38.0 (S: 24→30)
qw14b-40k ✅ 60 51 20 27 42.0 was 39.0 (S: 41→51)

Both profiles improved their fitting scores. qw14b-40k is now the highest-scoring profile at 42.0.

16.7 Updated Speed Comparison (All Profiles)

Profile Host GPUs tok/s Cloud %
g350m-2k rhtevan-work 1× A500 44 44%
g3b-16k rhtevan-work 1× A500 37 37%
g8b-spec-128k rhel-ai 4× L4 19-25 20-25%
g8b-fp8-spec-128k rhel-ai 4× L4 ~50 ~50%
qw14b-32k rhel-ai 2× L4 30.2 30%
qw14b-40k ✅ rhel-ai 4× L4 51.3 51%

16.8 Key Lesson

Speculative decoding is a compute-for-bandwidth trade. It only helps when:

  1. Excess compute headroom exists (GPU utilization < 50% during decode)
  2. TP communication overhead is low (NVLink, or TP=1)
  3. Native FP8 compute is available (Hopper+)
  4. Draft model doesn’t compete for memory bandwidth

L4 GPUs fail all four conditions. Removing spec decode yielded the largest single performance improvement in the project’s history. Full analysis captured in knowledge bundle: speculative-decoding-tradeoffs.md

17. D3 Decision Model Deployment (2026-10-15)

Deployed purpose-built Decision 3.0 models alongside the Qwen3-14B gen-ai model, creating the composite profile qw14b+d3.

17.1 D3 Family and L4 Fit Analysis

The D3 family (vllm-sr) uses the Qwen3.5 backbone with GDN hybrid attention (1 in 4 layers full attention, rest linear attention). All models have 262K max_position_embeddings. The GDN architecture makes KV cache ~4× cheaper than standard transformers.

Model Params FP8 Weights Decision Index Fits 1×L4? Fits 2×L4?
d3-edge 0.6B ~0.6 GB ~35 ✅ —
d3-lite 1.2B ~1.2 GB ~38 ✅ —
d3-nano 2.5B ~2.5 GB ~42 ✅ —
d3-mini 4B ~4 GB ~48 ✅ —
d3-flash 9B ~9 GB 55.8 ✅ —
d3 (27B) 27B ~13 GB/GPU 66.26 (SOTA) ❌ ✅ (TP=2)

17.2 Side-by-Side Benchmark: NextToken vs D3-Flash vs D3-27B

All tests via Skupper VAN from localhost. Gold answers human-judged.

Accuracy (19 questions across 4 test categories):

Test qw14b NextToken D3-Flash 9B D3-27B
T1 Trivial (4q) 4/4 ✅ 4/4 ✅ 4/4 ✅
T2 Nuanced (4q) 4/4 ✅ 3/4 ⚠️ 4/4 ✅
T3 Trick (3q) 3/3 ✅ 2/3 ❌ 3/3 ✅
T4 Batch (8q) 8/8 ✅ 8/8 ✅ 8/8 ✅
Total 19/19 17/19 19/19

D3-Flash errors: T2 risk_level “low” vs “moderate” (0.538 confidence), T3 has_eight “yes” (0.995 confidence — strongly wrong on trick question).

Latency (avg of 3 runs, via Skupper VAN):

Test qw14b NextToken D3-Flash 9B D3-27B
T1 (4q) 157ms 215ms 484ms
T2 (4q) 186ms 234ms 533ms
T3 (3q) 160ms 189ms 381ms
T4 (8q) 199ms 315ms 792ms

17.3 CUDA Graph OOM Analysis

Neither D3-27B nor D3-Flash can run with CUDA graphs on L4 GPUs.

Root cause: The Qwen3.5 GDN architecture includes a vision encoder that requires an 18.4 GB fixed allocation during CUDA graph capture. This is the vision encoder profiling workspace — not KV cache, not compile workspace, and not tunable via any vLLM parameter.

Attempted mitigations (all failed to enable CUDA graphs):

--kv-cache-dtype fp8 results:

17.4 Optimization Results

Config 2q Latency 4q Latency Notes
Baseline (enforce-eager, 32K) ~300ms ~490ms First deployment
+ context 8K ~300ms ~490ms Marginal
+ expandable_segments ~240ms ~470ms ~20% on short
+ gpu-mem 0.80 ~270ms ~470ms No improvement
+ FP8 KV cache ~38ms ~36ms 💥 Crashed (encoder incompatible)

Final config: --enforce-eager --quantization fp8 --max-model-len 8192

17.5 Composite Profile: qw14b+d3

GPU 0: qw14b-32k  (22.1 GB) — gen-ai + NextToken decisions, port 9000
GPU 1: qw14b-32k  (22.1 GB) — gen-ai + NextToken decisions, port 9000
GPU 2: d3-27b-8k  (20.8 GB) — D3 SOTA decisions, port 9001
GPU 3: d3-27b-8k  (20.8 GB) — D3 SOTA decisions, port 9001

Both models coexist and serve simultaneously. Skupper VAN multiplexes both ports over the single AMQPS link between localhost and rhel-ai.

17.6 Decision API Format Reference

Field NextToken (qw14b, :9000) D3 (:9001)
Endpoint /v1/decisions /v1/systemone (or /v1/decisions)
Input "input": "..." "state": "..."
Questions Array with name field Dict keyed by name
Yes/No "type": "predicate" "type": "noul"
Choice "choices": [{"value": "..."}] "criteria": {"key": "desc"}
Scale Not supported "type": "score", "criteria": ["low", "med", "high"]

17.7 Key Findings

  1. NextToken on 14B ≥ D3-Flash 9B: The logit-reading approach on a larger generalist model (19/19) outperformed the purpose-built 9B SDM (17/19) on both accuracy and speed.

  2. D3-27B justifies the GPUs: Perfect accuracy (19/19), better calibration than NextToken (hedges appropriately on uncertain questions), and supports score-type questions that NextToken cannot.

  3. CUDA graphs are blocked on L4 for all D3 models: Vision encoder profiling requires 18.4 GB — an upstream Qwen3.5 architecture constraint. FP8 KV cache was the only path to sub-100ms but is incompatible with encoder attention.

  4. expandable_segments helps: ~20% improvement on short requests by reducing PyTorch memory fragmentation.