---

license: cc-by-4.0

library_name: gguf

pipeline_tag: automatic-speech-recognition

tags:

- asr

- speech-recognition

- parakeet

- tdt

- fastconformer

- transducer

- ggml

- gguf

- cpp

language:

- en

- es

- fr

- de

- bg

- hr

- cs

- da

- nl

- et

- fi

- el

- hu

- it

- lv

- lt

- mt

- pl

- pt

- ro

- sk

- sl

- sv

- ru

- uk

base_model:

- nvidia/parakeet-tdt-0.6b-v3

base_model_relation: quantized

---

Parakeet TDT 0.6B v3 GGUF

GGUF weights of NVIDIA Parakeet TDT 0.6B v3 for s2s.cpp, a C++17/GGML voice assistant. FastConformer encoder and TDT transducer decoder, 25 European languages detected on their own, punctuation and capitalization included. Runs on CPU, CUDA, Metal, Vulkan, SYCL.

s2s.cpp loads Parakeet Ultra when it is present and this model otherwise.

Files

| variant | size | use case |

|---|---|---|

| F32 | 2.5 GB | reference, source of the quants |

| Q8_0 | 668 MB | recommended default |

| Q6_K | 537 MB | |

| Q5_K_M | 470 MB | |

| Q4_K_M | 407 MB | lowest VRAM |

The server loads the best quant it finds, up to Q8_0.

Quick start

git clone https://github.com/ServeurpersoCom/s2s.cpp.git
cd s2s.cpp
git submodule update --init
./buildcuda.sh
hf download Serveurperso/Parakeet-TDT-0.6b-v3-GGUF parakeet-tdt-0.6b-v3-Q8_0.gguf --local-dir models
./build/parakeet-transcribe --model models/parakeet-tdt-0.6b-v3-Q8_0.gguf --file qwentts.cpp/examples/freeman.wav

./models.sh --asr v3 then fetches the rest of the pipeline with this model and ./server.sh starts the voice assistant on http://localhost:8088.

Quantization policy

The quantizer of s2s.cpp mirrors llama-quantize: every 2D weight takes the K-quant of the variant, and in the M variants the attention values and the second feed forward linear are bumped to Q6_K on the first few, the last few and every third layer, the two places a low bit weight hurts the transcript first. Three groups stay F32 in every variant:

| tensor | why |

|---|---|

| subsampling kernels, depthwise conv kernels, sinusoid table | read by the f32 convolution path, precision worth more than the bytes |

| relative attention biases | added to the queries, no quantized add |

| 1D tensors (norms, biases) | as in llama.cpp |

The conversion folds each conformer BatchNorm into its depthwise kernel, so the graph has none.

Parity

test-parakeet holds each stage against transformers ParakeetForTDT on the example recording: mel, subsampling, first block, encoder states, projection, prediction network, joint, and the transcript.

| variant | encoder states cossim | transcript |

|---|---|---|

| F32 | 0.999999 | identical |

| Q8_0 | 0.999768 | identical |

| Q4_K_M | 0.984579 | 0.98 sequence match |

CUDA figures, the transcripts are the same on Vulkan and CPU.

License

Upstream model: Parakeet TDT 0.6B v3 by NVIDIA, CC BY 4.0

GGUF tooling: s2s.cpp, MIT