---

license: mit

base_model: baidu/Unlimited-OCR

tags:

• gguf
• llama.cpp
• ocr
• vision-language
• quantized

pipeline_tag: image-text-to-text

---

Unlimited-OCR — GGUF (llama.cpp)

GGUF conversions of baidu/Unlimited-OCR

(MIT) with a measured per-quant quality ladder: every number below traces to

a reproducible evaluation run against exact rendered ground truth, so the cost of

each quantization is interpretable rather than assumed.

Part of a measured-ladder series for this model; MLX conversions evaluated with

the same harness are in the collection.

⚠️ Runtime requirement — read first

These files load on **stock llama.cpp built at commit 4fc4ec5 (build 168,

2026-07-01) or newer** — the first mainline llama.cpp with Unlimited-OCR

(deepseek2-ocr) support (merged via PR #24969). You need two files: a

language GGUF (e.g. unlimited-ocr-Q5_K_M.gguf) and the multimodal

projector mmproj-unlimited-ocr-F16.gguf.

One-click loaders that bundle an older llama.cpp (e.g. Ollama, LM Studio) will

not load these until they update to a build that includes this architecture.

If your tool reports an unknown architecture deepseek2-ocr, its llama.cpp is

too old. Image inference is verified with llama-mtmd-cli (see Usage);

llama-server's multimodal endpoint does not yet accept this model's image

prompt in this build (details in Usage).

*(This runtime line is a property of the files. Note it is separate from how they

were produced — see "How this was made" for the one converter-side change, which

does not affect loading.)*

Which file should I use?

Recommended default: Q5_K_M (2.07 GiB) — the smallest variant with **no

measured degradation** on this corpus (CER within noise of the BF16 baseline).

Q8_0 and Q6_K reproduce the BF16 baseline's recognized text exactly here

(identical CER on every page) but are larger for no measured quality gain —

though both decode ~25% faster than Q5_K_M on this hardware (see tok/s); prefer

Q8_0 if RAM allows and throughput matters. Below Q5_K_M, quality falls off a

cliff.

| File | Size (GiB) | Overall CER | Δ vs BF16 | CER excl. loop pages | Loop rate | Notes |

|---|---|---|---|---|---|---|

| unlimited-ocr-BF16.gguf | 5.473 | 0.78% | — (baseline) | 0.78% | 0/24 | Public anchor / baseline |

| unlimited-ocr-Q8_0.gguf | 2.911 | 0.78% | +0.00 pp | 0.78% | 0/24 | Recognized text identical to BF16 here |

| unlimited-ocr-Q6_K.gguf | 2.434 | 0.78% | +0.00 pp | 0.78% | 0/24 | Recognized text identical to BF16 here |

| unlimited-ocr-Q5_K_M.gguf | 2.067 | 0.74% | −0.04 pp (within noise) | 0.74% | 0/24 | Recommended default |

| unlimited-ocr-Q4_K_M.gguf | 1.816 | 15.64% | +14.86 pp | 3.88% | 1/24 | Usable for clean/numeric pages; dense-tier CER 45%, one loop page — not the default here |

| unlimited-ocr-Q4_0.gguf | 1.585 | 44.02% | +43.24 pp | 9.12% | 2/24 | Published as measured 4-bit cliff evidence |

mmproj-unlimited-ocr-F16.gguf (0.769 GiB) is required by every variant.

Measured quality (vs GGUF-BF16 baseline)

Baseline is the BF16 GGUF running under llama.cpp — never a different runtime.

Deltas are computed on unrounded values.

| Variant | Overall CER | Overall WER | CER clean | CER dense | CER numeric | CER excl. loop | Loop rate | max len-ratio | gen tok/s |

|---|---|---|---|---|---|---|---|---|---|

| BF16 | 0.7792% | 0.7206% | 0.00% | 1.60% | 0.74% | 0.7792% | 0/24 | 1.13 | 172.5 |

| Q8_0 | 0.7792% | 0.7206% | 0.00% | 1.60% | 0.74% | 0.7792% | 0/24 | 1.13 | 243.7 |

| Q6_K | 0.7792% | 0.7206% | 0.00% | 1.60% | 0.74% | 0.7792% | 0/24 | 1.13 | 237.3 |

| Q5_K_M | 0.7399% | 0.6881% | 0.00% | 1.60% | 0.62% | 0.7399% | 0/24 | 1.13 | 194.3 |

| Q4_K_M | 15.6406% | 15.6419% | 0.95% | 45.23% | 0.74% | 3.8810% | 1/24 | 3.85 | 256.9 |

| Q4_0 | 44.0241% | 50.1559% | 1.85% | 121.44% | 8.79% | 9.1207% | 2/24 | 5.53 | 299.2 |

• CER = character error rate (Levenshtein / reference length); WER

analogous on whitespace tokens. Lower is better.

• CER excl. loop pages is the mean over pages that did not run away, so a

single collapsed page shows up as a loop rather than silently inflating the

headline. Q8_0 and Q6_K match the BF16 baseline's per-page CER exactly on

all 24 pages; Q5_K_M differs on one page (net −0.04 pp, within noise).

• Loop rate counts pages whose output exceeded 1.5× the reference length

(the runaway-repetition failure mode).

• gen tok/s is isolated generation throughput (generated tokens ÷ time

after model-load and vision-encode), single-image, M3 Max 36 GB, Metal.

• Peak memory is not reported — the CLI does not expose it and it is not

estimated.

Corpus: 24 synthetic pages (3 difficulty tiers × 8 — clean prose, dense

small-font, digit-heavy invoice), deterministic seed, exact rendered ground

truth. Prompt document parsing., temperature 0. **Decoding runs with

repetition suppression OFF** (--repeat-penalty 1.0, no DRY), deliberately, so

quantization instability shows up as honest high CER and visible loop pages

rather than being masked. Normalization (grounding tokens <|det|>…<|/det|>,

bbox coordinates, markdown, whitespace) is applied identically to reference and

hypothesis, so CER reflects recognition, not layout markup. Detokenization is

handled natively by llama.cpp (no byte-marker post-processing needed). Eval

harness (corpus generator + CER/WER scorer) is open-source:

https://github.com/vimalnakrani08/unlimited-ocr-eval-harness (the scoring core

is backend-agnostic; the runner is MLX-specific).

The headline: the 4-bit cliff, and mixed vs single-recipe 4-bit

8-, 6-, and 5-bit carry no measured recognition cost on this corpus; then 4-bit

drops sharply, and the damage concentrates in the dense small-font tier

(Q4_K_M 45% / Q4_0 121% dense CER, while clean pages stay ≤1.85%).

The two 4-bit files use different quantization recipes (verified from the GGUF

tensor types — described exactly, not assumed):

• Q4_0 — every transformer weight matrix at Q4_0 (118 tensors), the

output projection kept at Q6_K (1), and all normalization tensors in F32

(36). A single low-bit recipe across the body.

• Q4_K_M — a mixed-precision K-quant: Q4_K (95 tensors) with

selected tensors promoted — Q6_K (12, incl. output and some attention-value

projections), Q8_0 (6, some feed-forward down projections), Q5_0 (6, some

expert down projections), and F32 norms (36).

Mixed precision helps a lot at 4-bit — Q4_K_M (15.64% CER, 1 loop) is

2.8× lower CER than Q4_0 (44.02% CER, 2 loops) — but **neither reaches

BF16-level recognition**. The interpretable takeaway: for this architecture,

the usable floor on this corpus is Q5_K_M; 4-bit is a genuine quality cliff that

mixed precision softens without removing.

Loop taxonomy (the failure modes behind the loop rate)

The high-CER 4-bit pages fail in two distinct ways; len-ratio catches both:

• Verbatim cycle — the model repeats a fixed phrase indefinitely.

Example (Q4_0, one dense page): "The results of the data management system

were recorded by the last year of the last year." repeated to the token cap.

• Non-terminating drift — the model stays on the document's topic but

confabulates fresh, wrong sentences and never emits end-of-sequence.

Example (Q4_0, one dense page): the template "The quarterly report …"

continued with varying invented predicates.

The BF16/8/6/5-bit variants terminate cleanly on every page.

Secondary: DRY sampler ("deployed behavior") — a measured trade, not a free win

The primary numbers above use no repetition suppression. The base model's

official pipeline applies no_repeat_ngram_size; llama.cpp's analogue is the DRY

sampler. As a clearly-labeled secondary measurement, the two 4-bit files were

re-run over all 24 pages with DRY

(`--dry-multiplier 0.8 --dry-base 1.75 --dry-allowed-length 2

--dry-penalty-last-n -1 --dry-sequence-breaker none`, else identical):

| Variant | Overall CER OFF → DRY | Previously-looping pages | Previously-clean pages | New loop induced by DRY |

|---|---|---|---|---|

| Q4_K_M | 15.64% → 10.47% | 286.1% → 1.3% | 3.88% → 10.86% | one dense page (14.5% → 136.7%) |

| Q4_0 | 44.02% → 28.46% | 428.0% → 146.9% | 9.12% → 17.69% | one dense page (0% → 218%) |

DRY lowers the aggregate by rescuing the worst pre-existing loops, **but it

damages previously-clean pages** — most on the numeric/dense tiers whose correct

output is legitimately repetitive (template invoices; Q4_K_M numeric rises

0.74% → 7.27%) — and can induce new catastrophic loops on pages that were

fine without it. So repetition suppression is a trade specific to your document

mix, not a universal fix, which is why the honest primary ladder is scored with

it off. (The legitimately-repetitive-content explanation was hypothesized before

the measurement rather than fitted to it, is consistent with the observations,

and is not a proven mechanism.)

Secondary: R-SWA (Reference Sliding Window Attention)

Unlike current MLX ports, llama.cpp does implement Unlimited-OCR's R-SWA

(constant-KV sliding window with an always-visible reference/visual prefix) — in

open PR #24975

(LLAMA_SWA_TYPE_REFERENCE), implemented but not yet merged at publish time.

• **The published GGUFs run under stock mainline (build 168 / 4fc4ec5), which

uses full multi-head attention (MHA)** for this architecture — the ladder

above is all full-MHA.

• To measure R-SWA itself, the same BF16 GGUF was run on a local build of

master + #24975 (build 173 / 4d242d6, isolated worktree). Activation is

proven, not assumed: the identical file loads as n_swa = 128 on that build

vs n_swa = 0 on stock mainline. That build also promotes the V-cache to F32

(a requirement noted in the PR).

BF16, single-page fidelity, same 24 pages, same flags:

| | BF16 · MHA (mainline) | BF16 · R-SWA (#24975) |

|---|---|---|

| Overall CER | 0.7792% | 0.3344% |

| CER dense | 1.60% | 0.00% |

| CER numeric | 0.74% | 1.00% |

| gen tok/s | 172.5 | 168.2 |

R-SWA and MHA produce identical output on 21 of 24 pages; of the three that

differ, one dense page drives the overall improvement (MHA 12.8% → R-SWA 0%),

numeric moves the other way (+0.27 pp), and no loops are introduced. So at BF16

on single pages, R-SWA ≈ MHA fidelity — which also validates that the

full-MHA ladder above is a fair proxy for this model. Important scope note:

this corpus is single-page documents, so it measures fidelity parity only, not

R-SWA's headline multi-page constant-memory regime — that regime is not

exercised here and is a logged follow-up.

Cross-runtime note (secondary — never part of any degradation column)

On the identical frozen corpus, this **GGUF BF16 baseline scores 0.78% overall

CER vs 1.62% for the MLX BF16** conversion of the same weights. The per-tier

picture is mixed and does not point one way: GGUF is much better on the dense

tier (1.60% vs 4.86%) but the numeric tier goes the other way (GGUF 0.74%

vs MLX 0.00%). This reflects unspecified differences between the two runtimes

(detokenization, image preprocessing, kernels); isolating the cause is a logged

follow-up. It is reported here only as an observation and never enters a

quantization-degradation figure.

Usage

llama-mtmd-cli (verified — this is what produced every number above)

llama-mtmd-cli \
  -m unlimited-ocr-Q5_K_M.gguf \
  --mmproj mmproj-unlimited-ocr-F16.gguf \
  --image page.png \
  -p "document parsing." \
  --chat-template deepseek-ocr \
  --temp 0 \
  --repeat-penalty 1.0 \
  --flash-attn off \
  -n 2600 -c 16384 --no-warmup

Every flag matters: --chat-template deepseek-ocr (required), --temp 0

(greedy), --repeat-penalty 1.0 (suppression off — the measured setting;

--flash-attn off matches the reference), -c 16384 (large enough that the

2600-token cap, not the context, bounds any loop). Prompts follow DeepSeek-OCR

vocabulary: document parsing. (layout + grounding markup, used for all

measurements here), Free OCR. (plain text). Prompt viability is

runtime-specific: on this build Free OCR. emits end-of-sequence immediately,

while on mlx-vlm 0.6.3 the same prompt runs away into a repetition loop. Prefer

document parsing. and strip the <|det|>…<|/det|> grounding markup downstream

if you only want text.

llama-server

The server hosts the model (`llama-server -m unlimited-ocr-Q5_K_M.gguf --mmproj

mmproj-unlimited-ocr-F16.gguf --chat-template deepseek-ocr -c 16384

--flash-attn off`), but in this build its multimodal request path returns

Failed to tokenize prompt for this model (the chat template's image placeholder

is not injected on the server endpoint). Use llama-mtmd-cli for image

inference until upstream server support lands.

How this was made

1. Converted from baidu/Unlimited-OCR (original config, unmodified

hyperparameters) with llama.cpp's convert_hf_to_gguf.py → BF16 language

GGUF + F16 mmproj.

1. One converter-side change, disclosed for reproducibility: mainline's

converter did not yet have this checkpoint's tokenizer pre-tokenizer hash

registered, so conversion aborted. The tokenizer is DeepSeek-V3 family — its

pre-tokenizer regexes match llama.cpp's deepseek-v3 type byte-for-byte — so

the hash was registered to deepseek-v3 (a one-line addition to

get_vocab_base_pre). This is a converter change only; it does not

affect the runtime or how these files load (they load on stock mainline —

see the runtime line at the top). It is a candidate for upstreaming via

convert_hf_to_gguf_update.py.

1. Quantized with llama-quantize (Q8_0, Q6_K, Q5_K_M, Q4_K_M, Q4_0). Q4_K_M

measured at 1.816 GiB.

1. Smoke-tested (BF16, CER 0.0000% on a held-out page) before quantizing, then

the full ladder was evaluated (tables above).

Limitations — read this

• These are single-page OCR measurements. R-SWA's long-horizon,

constant-memory multi-page regime is not exercised by this corpus (see the

R-SWA section). Very-long single-pass multi-page behavior is unmeasured here.

• 4-bit is a measured quality cliff on this architecture — do not deploy

Q4_0/Q4_K_M for dense documents without running the harness on your own

distribution first.

• Synthetic corpus: clean renders, three fonts, English. Real-world scans (skew,

noise, handwriting, other scripts) are not covered by these numbers.

• Figures are specific to this corpus and to llama.cpp build 168 (4fc4ec5).

Attribution

Base model © Baidu, released under the MIT License — please cite their

technical report. This repository contains

converted and quantized weights and adds no training. GGUF conversion and

quantization by the uploader; R-SWA is the work of llama.cpp PR #24975.