cstr/voxtral-mini-4b-realtime-gguf overview

{
  "metadata": {},
  "card_data": {
    "language": [
      "en",
      "fr",
      "es",
      "de",
      "ru",
      "zh",
      "ja",
      "it",
      "pt",
      "nl",
      "ar",
      "hi",
      "ko"
    ],
    "license": "apache-2.0",
    "base_model": "mistralai/Voxtral-Mini-4B-Realtime-2602",
    "pipeline_tag": "automatic-speech-recognition",
    "tags": [
      "gguf",
      "speech-to-text",
      "realtime",
      "streaming",
      "voxtral"
    ],
    "frontmatter": {
      "language": [
        "en",
        "fr",
        "es",
        "de",
        "ru",
        "zh",
        "ja",
        "it",
        "pt",
        "nl",
        "ar",
        "hi",
        "ko"
      ],
      "license": "apache-2.0",
      "base_model": "mistralai/Voxtral-Mini-4B-Realtime-2602",
      "pipeline_tag": "automatic-speech-recognition",
      "tags": [
        "gguf",
        "speech-to-text",
        "realtime",
        "streaming",
        "voxtral"
      ]
    },
    "hero_image_url": "",
    "summary": "GGUF quantizations of mistralai/Voxtral-Mini-4B-Realtime-2602, a **4.4B-parameter realtime streaming speech-to-text model** with a causal audio encoder and configurable transcription delay. Converted and tested with CrispASR, a multi-model ASR framework built on ggml.",
    "quick_links": [],
    "benchmark_table_html": "",
    "readme_markdown": "---\nlanguage:\n- en\n- fr\n- es\n- de\n- ru\n- zh\n- ja\n- it\n- pt\n- nl\n- ar\n- hi\n- ko\nlicense: apache-2.0\nbase_model: mistralai/Voxtral-Mini-4B-Realtime-2602\npipeline_tag: automatic-speech-recognition\ntags:\n- gguf\n- speech-to-text\n- realtime\n- streaming\n- voxtral\n---\n\n# Voxtral-Mini-4B-Realtime — GGUF\n\nGGUF quantizations of [mistralai/Voxtral-Mini-4B-Realtime-2602](https://huggingface.co/mistralai/Voxtral-Mini-4B-Realtime-2602), a **4.4B-parameter realtime streaming speech-to-text model** with a causal audio encoder and configurable transcription delay.\n\nConverted and tested with [CrispASR](https://github.com/CrispStrobe/CrispASR), a multi-model ASR framework built on ggml.\n\n## Files\n\n| File | Quant | Size | Description |\n|------|-------|------|-------------|\n| `voxtral-mini-4b-realtime.gguf` | F16 | 8.3 GB | Full precision (reference) |\n| `voxtral-mini-4b-realtime-q8_0.gguf` | Q8_0 | 4.5 GB | 8-bit quantized |\n| `voxtral-mini-4b-realtime-q4_k.gguf` | Q4_K | 2.4 GB | 4-bit K-quant (recommended) |\n\n## Performance (CPU, 4 threads, AVX2, jfk.wav 11s)\n\n| Quant | Encoder | Prefill | Decode (ms/tok) | Total | RTFx |\n|-------|---------|---------|-----------------|-------|------|\n| F16 | 39s | 30s | 430 | 133s | 0.08× |\n| Q8_0 | 30s | 9s | 257 | 79s | 0.14× |\n| **Q4_K** | **19s** | **3s** | **177** | **49s** | **0.22×** |\n\nQ4_K recommended — 3.5× smaller than F16, 2.7× faster, identical transcription quality.\n\n## Usage\n\n```bash\n# Build CrispASR\ngit clone https://github.com/CrispStrobe/CrispASR\ncd CrispASR\ncmake -B build -DCMAKE_BUILD_TYPE=Release\ncmake --build build -j$(nproc) --target voxtral4b-main\n\n# Download Q4_K (recommended)\nhuggingface-cli download cstr/voxtral-mini-4b-realtime-GGUF \\\n    voxtral-mini-4b-realtime-q4_k.gguf --local-dir .\n\n# Transcribe\n./build/bin/voxtral4b-main -m voxtral-mini-4b-realtime-q4_k.gguf -f audio.wav\n```\n\n### With word-level timestamps\n\n```bash\n# Also download the CTC aligner\nhuggingface-cli download cstr/canary-ctc-aligner-GGUF \\\n    canary-ctc-aligner-q4_k.gguf --local-dir .\n\n./build/bin/voxtral4b-main -m voxtral-mini-4b-realtime-q4_k.gguf \\\n    -f audio.wav -am canary-ctc-aligner-q4_k.gguf -timestamps\n```\n\n### CLI options\n\n```\n-m  FNAME   GGUF model file (required)\n-f  FNAME   Input audio, 16 kHz mono WAV (required)\n-t  N       Threads (default: 4)\n-l  LANG    Language hint (default: en)\n-n  N       Max new tokens (default: 512)\n-am FNAME   CTC aligner GGUF for word timestamps\n-timestamps Enable word-level timestamps (requires -am)\n-np         Suppress stderr info\n```\n\n## Architecture\n\n- **Audio encoder**: 32-layer causal transformer (RoPE, SwiGLU, RMSNorm, sliding window 750)\n- **LLM decoder**: 26-layer Mistral (GQA 32/8, SwiGLU, adaptive RMSNorm, sliding window 8192)\n- **Projector**: 4-frame stack → Linear(5120→3072) → GELU → Linear(3072→3072)\n- **Tokenizer**: Mistral Tekken (150K vocab, 1000 special tokens)\n- **Audio injection**: adapter output ADDED to token embeddings (streaming format)\n\n### Key features\n\n- **Natively streaming** architecture with causal encoder\n- **13 languages**: en, fr, es, de, ru, zh, ja, it, pt, nl, ar, hi, ko\n- **Configurable delay**: 480ms default (6 tokens × 80ms)\n- **Apache 2.0** license\n\n## Conversion\n\n```bash\npython models/convert-voxtral4b-to-gguf.py \\\n    --input /path/to/Voxtral-Mini-4B-Realtime-2602 \\\n    --output voxtral-mini-4b-realtime.gguf\n\n# Then quantize\n./build/bin/cohere-quantize voxtral-mini-4b-realtime.gguf \\\n    voxtral-mini-4b-realtime-q4_k.gguf q4_k\n```\n\n## Credits\n\n- Model: [Mistral AI](https://mistral.ai/) — Apache 2.0\n- GGUF conversion: [CrispASR](https://github.com/CrispStrobe/CrispASR)\n- Port cross-referenced against [voxtral.c](https://github.com/antirez/voxtral.c), [voxmlx](https://github.com/awni/voxmlx), [voxtral-mini-realtime-rs](https://github.com/TrevorS/voxtral-mini-realtime-rs)\n",
    "related_quantizations": []
  },
  "tags": [
    "gguf",
    "speech-to-text",
    "realtime",
    "streaming",
    "voxtral",
    "automatic-speech-recognition",
    "en",
    "fr",
    "es",
    "de",
    "ru",
    "zh",
    "ja",
    "it",
    "pt",
    "nl",
    "ar",
    "hi",
    "ko",
    "base_model:mistralai/Voxtral-Mini-4B-Realtime-2602",
    "base_model:quantized:mistralai/Voxtral-Mini-4B-Realtime-2602",
    "license:apache-2.0",
    "region:us"
  ],
  "likes": 0,
  "downloads": 110,
  "gated": false,
  "private": false,
  "last_modified": "2026-04-10T04:57:40.000Z",
  "created_at": "2026-04-10T04:54:49.000Z",
  "pipeline_tag": "automatic-speech-recognition",
  "library_name": ""
}

{
  "_id": "69d8829945b020acfd198764",
  "id": "cstr/voxtral-mini-4b-realtime-GGUF",
  "modelId": "cstr/voxtral-mini-4b-realtime-GGUF",
  "sha": "38f0e025ff6091079acad38502f376dc03b6818e",
  "createdAt": "2026-04-10T04:54:49.000Z",
  "lastModified": "2026-04-10T04:57:40.000Z",
  "author": "cstr",
  "downloads": 110,
  "likes": 0,
  "gated": false,
  "private": false,
  "pipeline_tag": "automatic-speech-recognition",
  "library_name": "",
  "siblings_count": 4
}