cstr/qwen3-asr-0.6b-gguf overview

{
  "metadata": {},
  "card_data": {
    "license": "apache-2.0",
    "language": [
      "ar",
      "cs",
      "da",
      "de",
      "el",
      "en",
      "es",
      "fa",
      "fi",
      "fil",
      "fr",
      "hi",
      "hu",
      "id",
      "it",
      "ja",
      "ko",
      "mk",
      "ms",
      "nl",
      "pl",
      "pt",
      "ro",
      "ru",
      "sv",
      "th",
      "tr",
      "vi",
      "yue",
      "zh"
    ],
    "pipeline_tag": "automatic-speech-recognition",
    "tags": [
      "audio",
      "speech-recognition",
      "transcription",
      "ggml",
      "gguf",
      "qwen3",
      "speech-llm",
      "multilingual"
    ],
    "library_name": "ggml",
    "base_model": "Qwen/Qwen3-ASR-0.6B",
    "frontmatter": {
      "license": "apache-2.0",
      "language": [
        "ar",
        "cs",
        "da",
        "de",
        "el",
        "en",
        "es",
        "fa",
        "fi",
        "fil",
        "fr",
        "hi",
        "hu",
        "id",
        "it",
        "ja",
        "ko",
        "mk",
        "ms",
        "nl",
        "pl",
        "pt",
        "ro",
        "ru",
        "sv",
        "th",
        "tr",
        "vi",
        "yue",
        "zh"
      ],
      "pipeline_tag": "automatic-speech-recognition",
      "tags": [
        "audio",
        "speech-recognition",
        "transcription",
        "ggml",
        "gguf",
        "qwen3",
        "speech-llm",
        "multilingual"
      ],
      "library_name": "ggml",
      "base_model": "Qwen/Qwen3-ASR-0.6B"
    },
    "hero_image_url": "",
    "summary": "GGUF / ggml conversions of Qwen/Qwen3-ASR-0.6B for use with the qwen3-asr-main CLI from **CrispStrobe/CrispASR**. Qwen3-ASR 0.6B is Alibaba's **speech-LLM** ASR model: This is the **first speech-LLM** in the CrispASR family — every other model in the set uses a dedicated CTC / transducer / encoder-decoder. The Qwen3-ASR runtime ships with a persistent KV cache so per-token decode is O(1) in cache size, not O(N) full re-forwards.",
    "quick_links": [],
    "benchmark_table_html": "",
    "readme_markdown": "---\nlicense: apache-2.0\nlanguage:\n- ar\n- cs\n- da\n- de\n- el\n- en\n- es\n- fa\n- fi\n- fil\n- fr\n- hi\n- hu\n- id\n- it\n- ja\n- ko\n- mk\n- ms\n- nl\n- pl\n- pt\n- ro\n- ru\n- sv\n- th\n- tr\n- vi\n- yue\n- zh\npipeline_tag: automatic-speech-recognition\ntags:\n- audio\n- speech-recognition\n- transcription\n- ggml\n- gguf\n- qwen3\n- speech-llm\n- multilingual\nlibrary_name: ggml\nbase_model: Qwen/Qwen3-ASR-0.6B\n---\n\n# Qwen3-ASR 0.6B — GGUF (ggml-quantised)\n\nGGUF / ggml conversions of [`Qwen/Qwen3-ASR-0.6B`](https://huggingface.co/Qwen/Qwen3-ASR-0.6B) for use with the `qwen3-asr-main` CLI from **[CrispStrobe/CrispASR](https://github.com/CrispStrobe/CrispASR)**.\n\nQwen3-ASR 0.6B is Alibaba's **speech-LLM** ASR model:\n\n- **30 languages + 22 Chinese dialects** with automatic language detection\n- **6.42 % avg WER** on the HuggingFace [Open ASR Leaderboard](https://huggingface.co/spaces/hf-audio/open_asr_leaderboard)\n- **Apache-2.0** licence\n- **Speech-LLM architecture**: Whisper-style audio encoder (2D-conv subsampler + 18-layer Transformer + projector head, 896 → 1024) feeds frames into a stock **Qwen3 0.6B LLM** (28 layers, GQA 16/8, head_dim=128, Q-norm/K-norm, SwiGLU, RoPE θ=1e6) via embedding splice at `<|audio_pad|>` placeholder positions in a ChatML prompt. The LLM autoregressively generates the transcript.\n\nThis is the **first speech-LLM** in the CrispASR family — every other model in the set uses a dedicated CTC / transducer / encoder-decoder. The Qwen3-ASR runtime ships with a persistent KV cache so per-token decode is O(1) in cache size, not O(N) full re-forwards.\n\n## Files\n\n| File | Size | Notes |\n| --- | ---: | --- |\n| `qwen3-asr-0.6b.gguf`        | 1.88 GB | F16 |\n| `qwen3-asr-0.6b-q8_0.gguf`   | 961 MB  | Q8_0, near-lossless |\n| `qwen3-asr-0.6b-q4_k.gguf`   | 676 MB  | **Q4_K — recommended default**, faster than realtime on a 4-core CPU |\n\nAll quantisations produce the correct transcript on `samples/jfk.wav`:\n> And so, my fellow Americans, ask not what your country can do for you; ask what you can do for your country.\n\nThe mel filterbank from `WhisperFeatureExtractor` is **baked into the GGUF** as `audio.mel_filters` (along with `audio.mel_window`), so the C++ runtime computes the log-mel spectrogram natively without needing torch / librosa / scipy at inference time.\n\n## Quick Start\n\n```bash\n# 1. Build the runtime\ngit clone https://github.com/CrispStrobe/CrispASR\ncd CrispASR\ncmake -B build -DCMAKE_BUILD_TYPE=Release\ncmake --build build -j$(nproc) --target qwen3-asr-main\n\n# 2. Download a quantisation\nhuggingface-cli download cstr/qwen3-asr-0.6b-GGUF \\\n    qwen3-asr-0.6b-q4_k.gguf --local-dir .\n\n# 3. Transcribe\n./build/bin/qwen3-asr-main \\\n    -m qwen3-asr-0.6b-q4_k.gguf \\\n    -f your-audio.wav -t 8\n```\n\nAudio must be 16 kHz mono 16-bit PCM WAV. Pre-convert with:\n```bash\nffmpeg -i input.mp3 -ar 16000 -ac 1 -c:a pcm_s16le output.wav\n```\n\n## Performance\n\nMeasured on `samples/jfk.wav` (11 seconds), Apple-class 4-core CPU:\n\n| Variant | Mel | Encoder | Prefill | Decode/tok | **Total** |\n| --- | ---: | ---: | ---: | ---: | ---: |\n| F16  | 250 ms | 2660 ms | 3032 ms | 151 ms | 10.3 s |\n| Q8_0 | 236 ms | 2459 ms | 2840 ms | 137 ms |  9.5 s |\n| **Q4_K** | 246 ms | 2851 ms | 2721 ms | **118 ms** | **9.3 s** |\n\nQ4_K runs **slightly faster than realtime** with no quality loss on this clip.\n\n## Architecture\n\n| Component | Details |\n| --- | --- |\n| Audio encoder | 18-layer Whisper-style pre-LN Transformer, d=896, heads=14, head_dim=64, FFN=3584 |\n| Conv subsampler | 3 × Conv2D stride-2 (1→480→480→480), then linear (480·16=7680 → 896). Output frame rate ~13 frames / second of audio (~77 ms / frame) |\n| Projector | ln_post → proj1 (896→896) → GELU → proj2 (896→1024) |\n| LLM | Qwen3 0.6B: 28 layers, hidden=1024, **16 Q heads / 8 KV heads (GQA)**, head_dim=128, FFN=3072, SwiGLU, RMSNorm, **per-head Q-norm / K-norm**, NEOX-style RoPE θ=1e6 |\n| Vocab | 151 936 tokens (Qwen2 BPE, GPT-2 byte encoding) |\n| Audio | 16 kHz mono, 128 mel bins, n_fft=400, hop=160, win=400 (matches `WhisperFeatureExtractor`) |\n| Audio injection | `<|audio_pad|>` placeholder positions in ChatML prompt get their token embedding replaced with the encoder output frames |\n| Parameters | ~900 M |\n\n## Implementation notes (correctness)\n\nThe C++ runtime is verified to F16 numerical precision against the PyTorch reference at every architectural boundary on `samples/jfk.wav`:\n\n| Stage | Diff metric | Result |\n| --- | --- | --- |\n| Conv front-end (per-chunk Conv2D + flatten + linear) | max abs vs `conv_out.npy` | 1.43e-4 |\n| Full audio encoder (18 layers + projector) | per-row cosine sim vs `proj2_out.npy` | mean 1.000000, min 0.999999 |\n| Qwen3 LLM forward (28 layers, no audio) | per-position cosine sim vs `llm_logits.npy` | mean 0.999999, top-1 9/9 |\n| End-to-end (audio → spliced embeds → LLM → greedy decode) | reproduced reference token sequence | 26 / 26 |\n| Mel filterbank (C++ STFT vs `WhisperFeatureExtractor`) | max abs vs `mel_input.npy` | 2.2e-2 |\n\n### Bugs that would have been hours of debugging\n\nA few non-obvious gotchas the port had to handle:\n\n1. **`ggml_permute` semantics** are inverted from the obvious reading: `permute(t, p0, p1, p2, p3)` means \"source axis i goes to NEW position `p_i`\", not \"new axis i comes from source axis `p_i`\".\n2. **PyTorch hooks fire pre-GELU** when registered on an `nn.Conv2d` module — the `F.gelu` is applied externally in the forward function.\n3. **`cu_seqlens` is GPU-only**: `eager_attention_forward` (used on CPU) **ignores** `cu_seqlens` and does standard full self-attention. The \"windowed attention\" path only kicks in for FlashAttention2 on GPU. **Don't apply the windowed mask on CPU** — the reference produces full-attention output.\n4. **`WhisperFeatureExtractor.mel_filters` shape is `(n_freqs=201, n_mels=128)`**, not `(n_mels, n_freqs)` as the parameter ordering might suggest.\n5. **Qwen3 attention output width** is `hd × n_q_heads = 2048`, not `d_model = 1024`. The o_proj is `(2048 → 1024)`, so the attention output is reshaped to `(2048, T)` before o_proj.\n6. **mrope sidestep**: Qwen3-ASR uses interleaved multi-modal RoPE with `mrope_section=[24,20,20]`. For text-only or 1D-position input (which includes our spliced audio frames), the three mrope sections all receive identical position_ids and **collapse to standard 1D RoPE**. The simpler RoPE matches the reference perfectly for our use case.\n\nSee [`qwen3-asr-todo.md`](https://github.com/CrispStrobe/CrispASR/blob/main/qwen3-asr-todo.md) in the runtime repo for the complete work log.\n\n## How this was made\n\n1. The HF safetensors model was converted to GGUF F16 by [`models/convert-qwen3-asr-to-gguf.py`](https://github.com/CrispStrobe/CrispASR/blob/main/models/convert-qwen3-asr-to-gguf.py). All 612 tensors map cleanly. The mel filterbank (from `WhisperFeatureExtractor.mel_filters`) and Hann window are baked into the GGUF as `audio.mel_filters` / `audio.mel_window`.\n2. Quantised variants are produced by `cohere-quantize` (the same llama.cpp-style quantiser used for the other GGUF releases in this family).\n3. Inference is implemented in [`src/qwen3_asr.{h,cpp}`](https://github.com/CrispStrobe/CrispASR/blob/main/src/qwen3_asr.cpp): the encoder and the LLM each run as one ggml graph, with a persistent F32 KV cache `(head_dim, max_ctx, n_kv_heads, n_layers)` shared between prefill and per-token decode steps.\n\n## Reference implementation\n\n[`predict-woo/qwen3-asr.cpp`](https://github.com/predict-woo/qwen3-asr.cpp) (MIT) was read for architecture discovery and tensor name mapping. **No source code was vendored** — the CrispASR runtime is a re-implementation in this repo's existing FastConformer / cohere-style ggml infrastructure, sharing structures with the four other ASR runtimes in the family.\n\n## Supported languages\n\n`ar cs da de el en es fa fi fil fr hi hu id it ja ko mk ms nl pl pt ro ru sv th tr vi yue zh` plus 22 Chinese dialects (auto-detected at inference time).\n\n## Attribution\n\n- **Original model**: [`Qwen/Qwen3-ASR-0.6B`](https://huggingface.co/Qwen/Qwen3-ASR-0.6B) (Apache-2.0). Alibaba Cloud Qwen team.\n- **GGUF conversion + ggml runtime**: [CrispStrobe/CrispASR](https://github.com/CrispStrobe/CrispASR) — community contribution.\n- **Reference implementation**: [predict-woo/qwen3-asr.cpp](https://github.com/predict-woo/qwen3-asr.cpp) (MIT) — used for architecture discovery only, no code vendored.\n\n## Related\n\n- C++ runtime: **[CrispStrobe/CrispASR](https://github.com/CrispStrobe/CrispASR)**\n- Sister releases in the same family:\n  - [`cstr/cohere-transcribe-03-2026-GGUF`](https://huggingface.co/cstr/cohere-transcribe-03-2026-GGUF) — Cohere Transcribe 2B (Open ASR Leaderboard #1)\n  - [`cstr/parakeet-tdt-0.6b-v3-GGUF`](https://huggingface.co/cstr/parakeet-tdt-0.6b-v3-GGUF) — Parakeet TDT 600M (free word timestamps)\n  - [`cstr/canary-1b-v2-GGUF`](https://huggingface.co/cstr/canary-1b-v2-GGUF) — Canary 978M (speech translation)\n  - [`cstr/canary-ctc-aligner-GGUF`](https://huggingface.co/cstr/canary-ctc-aligner-GGUF) — universal multilingual forced aligner\n\n## License\n\nApache-2.0, inherited from the base model.\n",
    "related_quantizations": []
  },
  "tags": [
    "ggml",
    "gguf",
    "audio",
    "speech-recognition",
    "transcription",
    "qwen3",
    "speech-llm",
    "multilingual",
    "automatic-speech-recognition",
    "ar",
    "cs",
    "da",
    "de",
    "el",
    "en",
    "es",
    "fa",
    "fi",
    "fil",
    "fr",
    "hi",
    "hu",
    "id",
    "it",
    "ja",
    "ko",
    "mk",
    "ms",
    "nl",
    "pl",
    "pt",
    "ro",
    "ru",
    "sv",
    "th",
    "tr",
    "vi",
    "yue",
    "zh",
    "base_model:Qwen/Qwen3-ASR-0.6B",
    "base_model:quantized:Qwen/Qwen3-ASR-0.6B",
    "license:apache-2.0",
    "region:us"
  ],
  "likes": 0,
  "downloads": 903,
  "gated": false,
  "private": false,
  "last_modified": "2026-04-08T22:23:35.000Z",
  "created_at": "2026-04-08T11:25:32.000Z",
  "pipeline_tag": "automatic-speech-recognition",
  "library_name": "ggml"
}

{
  "_id": "69d63b2c6f125a33ef7eea2d",
  "id": "cstr/qwen3-asr-0.6b-GGUF",
  "modelId": "cstr/qwen3-asr-0.6b-GGUF",
  "sha": "ad086c22597ed47af05cc159dd61c98bd6e945f9",
  "createdAt": "2026-04-08T11:25:32.000Z",
  "lastModified": "2026-04-08T22:23:35.000Z",
  "author": "cstr",
  "downloads": 903,
  "likes": 0,
  "gated": false,
  "private": false,
  "pipeline_tag": "automatic-speech-recognition",
  "library_name": "ggml",
  "siblings_count": 5
}