cstr/voxtral-mini-3b-2507-gguf overview

{
  "metadata": {},
  "card_data": {
    "license": "apache-2.0",
    "language": [
      "en",
      "fr",
      "de",
      "es",
      "it",
      "pt",
      "nl",
      "hi"
    ],
    "pipeline_tag": "automatic-speech-recognition",
    "tags": [
      "audio",
      "speech-recognition",
      "transcription",
      "ggml",
      "gguf",
      "voxtral",
      "mistral",
      "whisper",
      "speech-llm",
      "multilingual"
    ],
    "library_name": "ggml",
    "base_model": "mistralai/Voxtral-Mini-3B-2507",
    "frontmatter": {
      "license": "apache-2.0",
      "language": [
        "en",
        "fr",
        "de",
        "es",
        "it",
        "pt",
        "nl",
        "hi"
      ],
      "pipeline_tag": "automatic-speech-recognition",
      "tags": [
        "audio",
        "speech-recognition",
        "transcription",
        "ggml",
        "gguf",
        "voxtral",
        "mistral",
        "whisper",
        "speech-llm",
        "multilingual"
      ],
      "library_name": "ggml",
      "base_model": "mistralai/Voxtral-Mini-3B-2507"
    },
    "hero_image_url": "",
    "summary": "GGUF / ggml conversions of mistralai/Voxtral-Mini-3B-2507 for use with the voxtral-main CLI from **CrispStrobe/CrispASR**. Voxtral Mini is Mistral's **3B-parameter speech-LLM** — an enhancement of Ministral 3B with state-of-the-art audio input capabilities while retaining best-in-class text performance. It excels at speech transcription, translation, and audio understanding.",
    "quick_links": [],
    "benchmark_table_html": "",
    "readme_markdown": "---\nlicense: apache-2.0\nlanguage:\n- en\n- fr\n- de\n- es\n- it\n- pt\n- nl\n- hi\npipeline_tag: automatic-speech-recognition\ntags:\n- audio\n- speech-recognition\n- transcription\n- ggml\n- gguf\n- voxtral\n- mistral\n- whisper\n- speech-llm\n- multilingual\nlibrary_name: ggml\nbase_model: mistralai/Voxtral-Mini-3B-2507\n---\n\n# Voxtral-Mini-3B-2507 — GGUF\n\nGGUF / ggml conversions of [`mistralai/Voxtral-Mini-3B-2507`](https://huggingface.co/mistralai/Voxtral-Mini-3B-2507) for use with the `voxtral-main` CLI from **[CrispStrobe/CrispASR](https://github.com/CrispStrobe/CrispASR)**.\n\nVoxtral Mini is Mistral's **3B-parameter speech-LLM** — an enhancement of [Ministral 3B](https://mistral.ai/news/ministraux) with state-of-the-art audio input capabilities while retaining best-in-class text performance. It excels at speech transcription, translation, and audio understanding.\n\n- **8 languages** (English, French, German, Spanish, Italian, Portuguese, Dutch, Hindi) with automatic language detection\n- **Built-in audio Q&A and summarization** — ask questions about audio content directly\n- **Function calling from voice** — trigger backend functions based on spoken intents\n- **Long-form context** — up to 30 minutes of audio for transcription, 40 minutes for understanding\n- **Natively multilingual** with state-of-the-art WER across the world's most widely used languages\n- **Highly capable at text** — retains the text understanding capabilities of its Ministral 3B backbone\n- **Apache-2.0** licence\n\n## Files\n\n| File | Size | Notes |\n| --- | ---: | --- |\n| `voxtral-mini-3b-2507-q4_k.gguf` | 2.5 GB | **Q4_K — recommended default** |\n| `voxtral-mini-3b-2507-q8_0.gguf` | 5.0 GB | Q8_0, near-lossless |\n\nBoth quantisations produce the correct transcript on `samples/jfk.wav`:\n> And so, my fellow Americans, ask not what your country can do for you, ask what you can do for your country.\n\nThe mel filterbank from `WhisperFeatureExtractor` and the Tekken tokenizer vocab are **baked into the GGUF**, so the C++ runtime computes everything natively — no Python/torch/librosa at inference time.\n\n## Quick Start\n\n```bash\n# 1. Build the runtime\ngit clone https://github.com/CrispStrobe/CrispASR\ncd CrispASR\ncmake -B build -DCMAKE_BUILD_TYPE=Release\ncmake --build build -j$(nproc) --target voxtral-main\n\n# 2. Download a quantisation\nhuggingface-cli download cstr/voxtral-mini-3b-2507-GGUF \\\n    voxtral-mini-3b-2507-q4_k.gguf --local-dir .\n\n# 3. Transcribe\n./build/bin/voxtral-main \\\n    -m voxtral-mini-3b-2507-q4_k.gguf \\\n    -f audio.wav -t 8 -l en\n```\n\nAudio must be 16 kHz mono 16-bit PCM WAV. Pre-convert with:\n```bash\nffmpeg -i input.mp3 -ar 16000 -ac 1 -c:a pcm_s16le output.wav\n```\n\n### Language selection\n\nUse `-l LANG` with a two-letter code:\n\n| Flag | Language |\n| --- | --- |\n| `-l en` | English (default) |\n| `-l de` | German |\n| `-l fr` | French |\n| `-l es` | Spanish |\n| `-l it` | Italian |\n| `-l pt` | Portuguese |\n| `-l nl` | Dutch |\n| `-l hi` | Hindi |\n\n## Performance\n\nMeasured on `samples/jfk.wav` (11 seconds), 4-core CPU:\n\n| Variant | Mel | Encoder | Prefill | Decode/tok | **Total** |\n| --- | ---: | ---: | ---: | ---: | ---: |\n| F16 (8.8 GB) | 264 ms | 48.7 s | 78.4 s | 1134 ms | 157 s |\n| **Q4_K (2.5 GB)** | 246 ms | 32.7 s | **30.8 s** | **242 ms** | **70 s** |\n\nQ4_K gives a **2.2× speedup** over F16 while producing identical transcripts. The 3B model is larger than the Qwen3-ASR 0.6B — for fastest CPU inference on short clips, Qwen3-ASR Q4_K (6.6s for 11s audio) is faster; Voxtral's advantage is the richer capabilities (audio understanding, function calling, text Q&A) and superior multilingual WER.\n\n## Architecture\n\nVoxtral-Mini-3B is a three-module speech-LLM:\n\n| Component | Details |\n| --- | --- |\n| **Audio encoder** | 32-layer Whisper-large-v3 encoder: d=1280, 20 heads, head_dim=64, FFN=5120, 128 mels, **learned absolute positional embedding** (1500, 1280). Conv1d front-end: conv1(128→1280, k=3, **stride=1**, pad=1) + GELU → conv2(1280→1280, k=3, stride=2, pad=1) + GELU. Note: conv1 stride is 1 (not 2 like standard Whisper), so only conv2 does temporal downsampling (2×). 3000 mel frames → 3000 → 1500 encoder frames. |\n| **Projector** | Stack-4-frames + 2× Linear: reshape (1500, 1280) → (375, 5120), then Linear(5120→3072) → GELU → Linear(3072→3072). 4× temporal downsampling: 50 fps Whisper output → **12.5 fps audio embeddings** matching the documented frame rate. |\n| **LLM** | 30-layer Llama 3 / Ministral 3B: d=3072, **32 Q heads / 8 KV heads (GQA, ratio 4)**, head_dim=128, FFN=8192, SwiGLU, RMSNorm, NEOX-style RoPE θ=1e8, vocab=131072, max_pos=131072. **No biases** anywhere. **No Q-norm/K-norm** (unlike Qwen3-ASR's Qwen3 backbone). |\n| **Tokenizer** | Mistral Tekken (tiktoken-style rank BPE, 150k vocab entries + 1000 special tokens). Stored in the GGUF as a binary blob. |\n| **Audio injection** | `audio_token_id=24` placeholder in the `[INST]` prompt; the LLM input embeddings at those positions get replaced with the projector output frames. |\n| **Parameters** | ~3B total |\n\n### Transcription prompt format\n\n```\n<s> [INST] [BEGIN_AUDIO] <audio_pad>×375 [/INST] lang:en [TRANSCRIBE]\n```\n\nToken IDs: `[1, 3, 25, 24×375, 4, 9909, 1058, <lang_id>, 34]`\n\n### Key differences from standard Whisper\n\n1. **Conv1 stride is 1** (Whisper uses stride 2). This means the conv front-end only does 2× temporal reduction (just conv2), not 4×. 3000 mel frames → 1500 encoder frames (vs Whisper's 750).\n2. **K-proj has no bias** in the encoder's self-attention (Whisper quirk preserved from the Whisper-large-v3 weights).\n3. The encoder output is **not** consumed by a Whisper decoder — it's fed through a 4-frame-stack projector into a general-purpose Llama 3 LLM that generates the transcript (or any other text response) autoregressively.\n\n## Implementation notes\n\nThe C++ runtime is verified against the PyTorch reference (bf16) at every architectural boundary:\n\n| Stage | Diff metric | Result |\n| --- | --- | --- |\n| LLM forward (30 layers, text-only) | cosine sim at last position | 0.999973, **top-5 5/5 match** |\n| Audio encoder + projector (32 layers + stack-4) | per-row cosine sim vs `proj2_out.npy` | mean 0.998, min 0.870 (bf16 ref precision) |\n| End-to-end transcription on jfk.wav | generated token sequence | **Correct transcript** |\n\nThe 0.87 min cosine sim on the encoder is from the bf16 reference precision (7-bit mantissa) vs F16 GGUF weights (10-bit) with F32 compute in C++. An F32 reference would give tighter numbers — the end-to-end transcript is the real correctness test and it passes.\n\n### Bugs found during the port\n\n1. **`ggml_conv_1d` output layout**: returns `(OL, OC, N)` not `(OC, OL)`. Bias needs `(1, OC, 1)` reshape to broadcast over time+batch.\n2. **Post-conv transpose**: `ggml_conv_1d` puts time on `ne[0]`, but LayerNorm needs feature dim on `ne[0]`. Fixed by reshape+transpose to `(d, T_enc)`.\n3. **GELU approximation**: `ggml_gelu` (tanh approx) → `ggml_gelu_erf` (exact) for correctness matching.\n4. **Tekken vocab blob storage**: gguf-py's `add_array` with Python int lists stores as INT32, corrupting the uint8 byte stream. Fixed by storing as a 1D F32 tensor.\n\n## How this was made\n\n1. HF safetensors converted to GGUF F16 by [`models/convert-voxtral-to-gguf.py`](https://github.com/CrispStrobe/CrispASR/blob/main/models/convert-voxtral-to-gguf.py). All 765 tensors (762 model + mel_filters + mel_window + Tekken vocab blob) map cleanly.\n2. Quantised variants produced by [`cohere-quantize`](https://github.com/CrispStrobe/CrispASR/blob/main/examples/cohere-main/cohere-quantize.cpp) with the Q4_0 fallback for 1280-wide audio encoder tensors (1280 % 256 ≠ 0 for Q4_K, same situation as Qwen3-ASR).\n3. Inference implemented in [`src/voxtral.{h,cpp}`](https://github.com/CrispStrobe/CrispASR/blob/main/src/voxtral.cpp) (~1300 LOC): encoder and LLM each run as one ggml graph, with a persistent F16 KV cache `(head_dim, max_ctx, n_kv_heads, n_layers)` shared between prefill and per-token decode steps. Flash attention (`ggml_flash_attn_ext`) used on both prefill (F16 causal mask) and decode (no mask) paths.\n\n## Related\n\n- **Original model**: [`mistralai/Voxtral-Mini-3B-2507`](https://huggingface.co/mistralai/Voxtral-Mini-3B-2507) (Apache-2.0)\n- **C++ runtime**: [CrispStrobe/CrispASR](https://github.com/CrispStrobe/CrispASR)\n- **Research paper**: [arxiv.org/abs/2507.13264](https://arxiv.org/abs/2507.13264)\n- Sister releases in the same family:\n  - [`cstr/qwen3-asr-0.6b-GGUF`](https://huggingface.co/cstr/qwen3-asr-0.6b-GGUF) — Qwen3-ASR 0.6B (faster, 30 languages + Chinese dialects)\n  - [`cstr/parakeet-tdt-0.6b-v3-GGUF`](https://huggingface.co/cstr/parakeet-tdt-0.6b-v3-GGUF) — Parakeet TDT 600M (free word timestamps)\n  - [`cstr/canary-1b-v2-GGUF`](https://huggingface.co/cstr/canary-1b-v2-GGUF) — Canary 978M (speech translation)\n  - [`cstr/cohere-transcribe-03-2026-GGUF`](https://huggingface.co/cstr/cohere-transcribe-03-2026-GGUF) — Cohere Transcribe 2B (lowest English WER)\n\n## License\n\nApache-2.0, inherited from the base model.\n",
    "related_quantizations": []
  },
  "tags": [
    "ggml",
    "gguf",
    "audio",
    "speech-recognition",
    "transcription",
    "voxtral",
    "mistral",
    "whisper",
    "speech-llm",
    "multilingual",
    "automatic-speech-recognition",
    "en",
    "fr",
    "de",
    "es",
    "it",
    "pt",
    "nl",
    "hi",
    "arxiv:2507.13264",
    "base_model:mistralai/Voxtral-Mini-3B-2507",
    "base_model:quantized:mistralai/Voxtral-Mini-3B-2507",
    "license:apache-2.0",
    "region:us"
  ],
  "likes": 0,
  "downloads": 109,
  "gated": false,
  "private": false,
  "last_modified": "2026-04-09T14:56:33.000Z",
  "created_at": "2026-04-09T14:47:01.000Z",
  "pipeline_tag": "automatic-speech-recognition",
  "library_name": "ggml"
}

{
  "_id": "69d7bbe54606fd9f1b439462",
  "id": "cstr/voxtral-mini-3b-2507-GGUF",
  "modelId": "cstr/voxtral-mini-3b-2507-GGUF",
  "sha": "7a6ffdc7ff9ef90d291554546f3a54562ff5beda",
  "createdAt": "2026-04-09T14:47:01.000Z",
  "lastModified": "2026-04-09T14:56:33.000Z",
  "author": "cstr",
  "downloads": 109,
  "likes": 0,
  "gated": false,
  "private": false,
  "pipeline_tag": "automatic-speech-recognition",
  "library_name": "ggml",
  "siblings_count": 4
}