---

base_model: InternScience/Agents-A1

base_model_relation: quantized

library_name: llama.cpp

license: mit

language:

- en

pipeline_tag: image-text-to-text

tags:

- llama.cpp

- NVFP4

- MXFP4

- Blackwell

- MoE

- MTP

- speculative-decoding

- multimodal

quantized_by: s-batman

---

Agents-A1 NVFP4/MXFP4 GGUF with MTP

Quantized GGUF files for InternScience/Agents-A1, a Qwen3.5-35B-A3B MoE multimodal model.

Attribution

• Original model: InternScience/Agents-A1 — a Qwen3.5-35B-A3B MoE multimodal derivative
• MTP heads: Grafted from unsloth/Qwen3.6-35B-A3B-MTP-GGUF
• Quantization framework: llama.cpp with NVFP4/MXFP4 MoE routing patches (commits 1f8583c6b, 6e761dbc9)
• Tensor mapping: Follows Unsloth's gold standard MoE quantization strategy
• Quantized by: s-batman

Files

| File | Format | Size | BPW | Tensor Type | Notes |

|------|--------|------|-----|-------------|-------|

| agents-a1-NVFP4-MTP.gguf | NVFP4 (type 40) | 19.8 GB | 4.80 | 80 NVFP4 + 320 Q8_0 + 308 F32 + 5 MTP donor | Blackwell-native FP4, MTP-grafted |

| agents-a1-MXFP4-MTP.gguf | MXFP4 (type 39) | 18.9 GB | 4.57 | 80 MXFP4 + 320 Q8_0 + 308 F32 + 5 MTP donor | MXFP4, MTP-grafted |

| mmproj-agents-a1-f16.gguf | F16 | 858 MB | 16 | 334 F16 tensors | Vision tower (mmproj), never quantized |

Tensor Type Mapping

Follows Unsloth's gold standard MoE quantization mapping:

| Tensor Category | Count | Quant Format |

|----------------|-------|--------------|

| 3D expert tensors (ffn_*_exps) | 80 | NVFP4 / MXFP4 |

| Sensitive 2D (attn, embeddings, output, shared experts) | 312 | Q8_0 |

| Routers (ffn_gate_inp, ffn_gate_inp_shexp) | 2 | F32 |

| Norms, biases | 301 | F32 |

| MTP block (blk.40.*) | 20 | Mixed (from MTP donor) |

MTP (Multi-Token Prediction)

MTP heads grafted from unsloth/Qwen3.6-35B-A3B-MTP-GGUF for speculative decoding support.

• block_count: 40 → 41
• nextn_predict_layers: 1
• 20 MTP tensors at blk.40.* (separate gate/up experts, not fused)

Usage with MTP speculative decoding

llama-server \
  -m agents-a1-NVFP4-MTP.gguf \
  --mmproj mmproj-agents-a1-f16.gguf \
  -c 4096 -ngl 99 -fa 1 \
  -b 1024 -ub 4096 \
  --spec-type draft-mtp --spec-draft-n-max 8

Architecture

• Architecture: Qwen3.5 MoE (qwen35moe)
• Layers: 40 (30 linear attention + 10 full attention, every 4th)
• Experts: 256 total, 8 per token, 1 shared expert
• Hidden size: 2048
• Head dim: 256
• Vocab: 248,320
• Context: 262,144
• Vision: ViT depth 27, hidden 1152 (multimodal)

Quantization Details

Converted from BF16 GGUF using llama-quantize with:

• --output-tensor-type q8_0
• --token-embedding-type q8_0
• --no-mtp --fuse-gate-up-exps (during HF→GGUF conversion)
• MoE routing: 3D expert tensors (ne[2] > 1) → FP4, 2D linear tensors → Q8_0, norms → F32

Benchmarks

All benchmarks run on NVIDIA GB10 (DGX Spark) — 124 GB unified memory, compute capability 12.1, BLACKWELL_NATIVE_FP4=1.

Benchmark tool: llama-bench (build 9277) and llama-server with chat completions.

Prompt Processing (prefill) — varying ubatch size

Fixed: b=4096, pp2048, ngl=99, fa=1, 3 repetitions.

| Model | ub=512 | ub=1024 | ub=2048 | ub=4096 |

|-------|-------:|--------:|--------:|--------:|

| NVFP4 | 1,142 t/s | 1,254 t/s | 1,470 t/s | 1,642 t/s |

| MXFP4 | 1,320 t/s | 1,562 t/s | 1,619 t/s | 1,618 t/s |

Best prefill: NVFP4 at ub=4096 (1,642 t/s), MXFP4 at ub=2048 (1,619 t/s). Larger ubatch consistently improves prefill throughput.

Token Generation (decode) — varying batch size

Fixed: ub=512, tg256, ngl=99, fa=1, 3 repetitions.

| Model | b=512 | b=1024 | b=2048 | b=4096 |

|-------|------:|-------:|-------:|-------:|

| NVFP4 | 23.1 t/s | 25.4 t/s | 25.4 t/s | 23.6 t/s |

| MXFP4 | 24.8 t/s | 25.4 t/s | 25.3 t/s | 22.2 t/s |

Best decode: Both formats at b=1024 (25.4 t/s). Decode performance is relatively flat across batch sizes, with b=4096 showing slight degradation.

Chat completions — MTP speculative decoding sweep

Fixed: b=1024, ub=4096, c=4096, ngl=99, fa=1, 256 max tokens, temperature 0.7. Values are averages of 3 runs. MTP heads grafted from unsloth/Qwen3.6-35B-A3B-MTP-GGUF.

NVFP4 — varying --spec-draft-n-max

| n-max | Decode (t/s) | Accept Rate | Accepted/Generated | vs Baseline |

|------:|-------------:|------------:|-------------------:|------------:|

| — (baseline) | 69.3 | — | — | — |

| 2 | 84.1 | 62.2% | 423/680 | +21.3% |

| 3 | 77.9 | 50.5% | 460/911 | +12.4% |

| 4 | 71.6 | ~43.7% | 483/1113 | +3.3% |

| 8 | 58.0 | 36.0% | ~190/502 | −16.3% |

Key finding: --spec-draft-n-max 2 is the optimal setting on GB10, delivering a 21% speedup over baseline (84.1 t/s vs 69.3 t/s). The accept rate at n-max=2 is 62% — substantially higher than at n-max=8 (36%) because shorter draft sequences have higher per-token match probability and lower overhead.

At n-max=8, the overhead of generating 8 draft tokens per step with only 36% acceptance makes MTP net-negative. The break-even point on GB10 is around n-max=4.

MXFP4 — n-max=8 (prior runs, with mmproj)

| n-max | Decode (t/s) | Accept Rate | vs Baseline |

|------:|-------------:|------------:|------------:|

| — (baseline) | 69.0 | — | — |

| 8 | 58.2 | 35% | −15.7% |

MXFP4 shows the same pattern as NVFP4 at n-max=8. Based on the NVFP4 sweep, n-max=2 is expected to provide a similar ~20% speedup for MXFP4 as well.

> Note: The tg256 numbers from llama-bench (25 t/s) differ from the chat completion numbers (69 t/s) because llama-bench measures raw decode throughput with no chat template overhead, while the chat tests include Jinja template processing, thinking mode, and warmup effects. The relative comparison between MTP and no-MTP is valid within the same test type.

Recommended configuration

# Best decode with MTP speculative decoding (recommended)
llama-server -m agents-a1-NVFP4-MTP.gguf --mmproj mmproj-agents-a1-f16.gguf \
  -c 4096 -ngl 99 -fa 1 -b 1024 -ub 4096 -np 1 \
  --spec-type draft-mtp --spec-draft-n-max 2

# Baseline without MTP
llama-server -m agents-a1-NVFP4-MTP.gguf --mmproj mmproj-agents-a1-f16.gguf \
  -c 4096 -ngl 99 -fa 1 -b 1024 -ub 4096 -np 1

License

MIT

---

Quantized by s-batman. For additional GGUF quants, benchmarks, and articles on LLM memory and context engineering, visit sbatman.com.