---

license: apache-2.0

base_model: yandex/AliceAI-T5-35B-A0.6B

base_model_relation: quantized

library_name: llama.cpp

pipeline_tag: text-generation

tags:

• gguf
• llama.cpp
• encoder-decoder
• mixture-of-experts
• quantized
• text-generation

---

AliceAI-T5-35B-A0.6B GGUF

Community GGUF quantizations of yandex/AliceAI-T5-35B-A0.6B.

<div align="center" style="background-color:#f59e0b;color:#ffffff;padding:16px 20px;border-radius:10px;line-height:1.7;">

☕ If this GGUF made your day easier, a coffee would make mine.<br>

<a href="https://ko-fi.com/ngquocvinh" style="color:#ffffff;"><strong style="color:#ffffff;">Send a coffee ☕</strong></a><br>

I build and test these releases myself. Your coffee helps keep me going.<br>

Thank you for supporting this work.

</div>

> Compatibility warning: AliceAI-T5 is a base encoder-decoder model, not

> an instruction/chat model. It requires an AliceAI-aware llama.cpp runtime

> with encoder, cross-attention, and decoder support. Stock llama-cli,

> Ollama, vLLM, and LM Studio runtimes have not been validated for this

> architecture. The runtime used for this release is derived from the

> XHToken/llama.cpp fork.

About AliceAI-T5-35B-A0.6B

AliceAI-T5-35B-A0.6B is a Yandex encoder-decoder language model with sparse

mixture-of-experts layers. The upstream model card describes approximately

34.35B unique parameters, 512 experts per MoE layer with top-8 routing, a

1,536-wide hidden state, 16 encoder layers, 12 decoder layers, shared

encoder/decoder embeddings, and RoPE with YaRN. The upstream card advertises a

128K-token context window; this GGUF release has not been validated at that

maximum context length.

See the official upstream model card

for the original model description, benchmark claims, dependencies, and

Transformers usage. This is a custom encoder-decoder GGUF architecture, not a

decoder-only model. The runtime must execute the encoder, cross-attention, and

decoder generation path.

No training or fine-tuning was performed. Every published artifact was

converted directly from the locked upstream BF16 snapshot; no quantized file

was used as the source of another quantized file.

Download

Install the Hugging Face CLI and download a compact validated candidate:

python -m pip install -U huggingface_hub

mkdir -p aliceai-t5
cd aliceai-t5

hf download ngquocvinh/AliceAI-T5-35B-A0.6B-GGUF \
  AliceAI-T5-35B-A0.6B-IQ4_XS.gguf \
  aliceai-chat.jinja \
  SHA256SUMS.txt \
  --local-dir .

sha256sum -c SHA256SUMS.txt --ignore-missing

IQ4_XS is the compact release candidate used in the GPU profile for this

project. Its file size is 18.506586 GB; runtime memory also depends on context,

batch size, backend, and offload settings.

Build the AliceAI-aware runtime

Stock llama.cpp binaries are not sufficient for this custom encoder-decoder

architecture. The runtime patch used for this release can be applied to the

matching XHToken/llama.cpp source revision:

git clone https://github.com/XHToken/llama.cpp.git
cd llama.cpp
git checkout 4a3635c32fc9f044c2bde9ebeabf50c7e1ec5991

mkdir -p /tmp/aliceai-runtime
hf download ngquocvinh/AliceAI-T5-35B-A0.6B-GGUF \
  reproducibility/aliceai-runtime.patch \
  --local-dir /tmp/aliceai-runtime
git apply /tmp/aliceai-runtime/reproducibility/aliceai-runtime.patch

cmake -B build -DGGML_CUDA=ON
cmake --build build --config Release -j 8 \
  --target llama-server llama-simple

For a CPU-only build, use -DGGML_CUDA=OFF. The patch is also available at

reproducibility/aliceai-runtime.patch.

Published files

| File | Size (GB) | Size (bytes) | Intended use |

|---|---:|---:|---|

| AliceAI-T5-35B-A0.6B-BF16.gguf | 69.177 | 69,176,812,608 | BF16 reference |

| AliceAI-T5-35B-A0.6B-Q8_0.gguf | 36.796 | 36,795,843,648 | Highest-fidelity quantization tier |

| AliceAI-T5-35B-A0.6B-Q6_K.gguf | 28.431 | 28,430,760,320 | High-quality lower-memory tier |

| AliceAI-T5-35B-A0.6B-Q5_K_M.gguf | 24.624 | 24,623,656,832 | Balanced quality/memory tier |

| AliceAI-T5-35B-A0.6B-Q4_K_M.gguf | 21.041 | 21,040,500,608 | General balanced tier |

| AliceAI-T5-35B-A0.6B-Q3_K_M.gguf | 16.575 | 16,575,128,960 | Smaller-memory tier |

| AliceAI-T5-35B-A0.6B-Q2_K.gguf | 12.684 | 12,683,637,632 | Experimental low-memory tier |

| AliceAI-T5-35B-A0.6B-IQ4_XS.gguf | 18.507 | 18,506,585,984 | Compact experimental tier |

Q8_0 through Q3_K_M and IQ4_XS passed the encoder-decoder load and

deterministic generation smoke test used for this release. Q2_K also loads and

generates, but its factual smoke result was not semantically correct; treat it

as a runtime/low-memory experiment, not as a quality recommendation. These

smoke checks are not task benchmarks.

Three additional local candidates were not published: IQ2_XS produced no

decoded token, IQ3_M produced only blank or control-token output, and IQ1_M

entered a repeated <SPAN#511>Post loop. All three were excluded by the

functional smoke gate.

Quick start

Use an AliceAI-aware llama.cpp build containing the llama-simple encoder-

decoder test harness from the runtime used for this release. The ordinary

upstream llama-cli path is not sufficient for this architecture:

./build/bin/llama-simple \
  -m ./AliceAI-T5-35B-A0.6B-Q4_K_M.gguf \
  -ngl 0 -n 1 \
  '[_S_]Question: What is the capital of France? Answer:<SPAN#0>'

The tested protocol prepends the upstream [_S_] mode token, appends

<SPAN#0> to the encoder input, and seeds the decoder with

[decoder_bos, <SPAN#0>]; the harness handles these encoder-decoder details.

The Q4_K_M smoke returns Paris in the tested CPU configuration. A generic

decoder-only llama-cli invocation is not sufficient for this architecture.

To expose an OpenAI-compatible endpoint with the same runtime:

CUDA_VISIBLE_DEVICES=0 ../llama.cpp/build/bin/llama-server \
  -m ./AliceAI-T5-35B-A0.6B-IQ4_XS.gguf \
  --alias aliceai-t5 \
  --host 127.0.0.1 --port 8001 \
  --n-gpu-layers 99 \
  --ctx-size 8192 --parallel 1 \
  --batch-size 8192 --ubatch-size 2048 \
  --flash-attn on --jinja \
  --chat-template-file ./aliceai-chat.jinja \
  --temp 0

Test it with:

curl http://127.0.0.1:8001/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"aliceai-t5","messages":[{"role":"user","content":"What is the capital of France?"}],"temperature":0,"max_tokens":64}'

Android / CPU notes

For Termux or another CPU-only device, use the AliceAI-aware binary and the

downloaded template explicitly:

./build/bin/llama-server \
  -m ./AliceAI-T5-35B-A0.6B-IQ4_XS.gguf \
  --host 0.0.0.0 --port 5001 \
  --jinja \
  --chat-template-file ./aliceai-chat.jinja \
  --n-gpu-layers 0 \
  --ctx-size 2048 --parallel 1 \
  --batch-size 2048 --ubatch-size 2048 \
  --threads 6 --threads-batch 8 \
  --flash-attn off \
  --temp 0

Do not omit --chat-template-file: without it, llama.cpp may select a

decoder-only ChatML template and expose tokens such as <|im_start|> or

<|im_end|>. An encoder input of 1,073 tokens requires an encoder

--ubatch-size greater than that input; if the log still reports an encoder

batch size of 512 after setting --ubatch-size 2048, the binary is not the

patched runtime described above and must be rebuilt from the pinned revision

and reproducibility/aliceai-runtime.patch.

On a memory-constrained phone, remove --no-mmap and start a new, shorter

conversation before lowering the batch sizes. Reducing --ubatch-size below

the input length cannot make the encoder request succeed. Longer contexts can

also reduce CPU speed substantially because this model runs an encoder plus a

decoder and needs significant memory bandwidth.

For the standalone CPU smoke test, pass -ngl 0; the default GPU-layer

setting is not appropriate for a CPU-only Termux build:

./build/bin/llama-simple \
  -m ./AliceAI-T5-35B-A0.6B-IQ4_XS.gguf \
  -ngl 0 -n 16 \
  '[_S_]Question: What is the capital of France? Answer:<SPAN#0>'

The expected text-to-text prompt format is:

Question: What is the capital of France?
Answer:

Do not add a ChatML system prompt or expect normal multi-turn assistant

behavior from this base model. The bundled Jinja file is only a single-turn

QA adapter: it intentionally uses the latest user message and does not

preserve earlier chat history. For factual QA, use temperature=0; the

default sampling temperature can produce unstable or empty answers from this

base model.

Known limitations

• This release is not instruction-tuned or chat-tuned; task-specific behavior

can differ substantially from an assistant model.

• The upstream 128K context claim has not been validated for these GGUF files.
• GGUF file size is not a minimum VRAM requirement; leave headroom for runtime

buffers, context, and the selected backend.

• Some runtimes may display UTF-8 byte-fallback pieces as <0xHH> tokens until

their output detokenizer reconstructs the original bytes.

• The table above describes the validated release subset. Other quantization

files in the repository should be treated as experimental unless they are

covered by the checksum and validation records.

Reproducibility and validation

The input is pinned to upstream revision

a0d71f58c40d6affe461797b30b35ff47f52a5f2 and the GGUF files were converted

from the generated BF16 reference. The release used a model-specific

AliceAI calibration set plus Wikitext training text to build a combined

imatrix. The exact source hashes, imatrix hashes, converter/runtime state,

quantization commands, file sizes, and smoke-test results are recorded in

reproducibility/manifest.md.

Verify downloads with SHA256SUMS.txt. Raw converter,

imatrix, quantization, and smoke-test logs remain local under reports/ and

are not part of the public package.

The GGUF files require the AliceAI-aware runtime described above; the runtime

revision and model conversion details are recorded in

reproducibility/manifest.md.

No hold-out BF16 fidelity table is included yet. Mean KLD, Top-1 agreement,

delta PPL, and RMS delta-p will be added only after a fixed encoder-decoder

hold-out evaluator measures every published file against the same BF16

reference.

License and attribution

The upstream model is released under the Apache License 2.0; see the included

LICENSE. These are community GGUF quantizations, not an official

Yandex release or endorsement.