---

license: apache-2.0

base_model:

• Qwen/Qwen3.5-4B
• Ttimms/Bible-Assistant-Qwen3.5-4B-v3.2

base_model_relation: quantized

tags:

- bible

- rag

- gguf

- llama.cpp

- qwen3.5

- nvfp4

- text-generation

language:

- en

pipeline_tag: text-generation

quantized_by: Ttimms

license_link: https://huggingface.co/Qwen/Qwen3.5-4B/blob/main/LICENSE

---

Bible AI Assistant v3.2 — GGUF

GGUF quants of Ttimms/Bible-Assistant-Qwen3.5-4B-v3.2

— a Qwen3.5-4B continued-fine-tune for retrieval-grounded Bible Q&A, and the

first checkpoint in the project's line to clear every acceptance gate: 80.3%

verbatim verse recall, 98.9% citation, 1.9% hallucination, and the strongest

result in its size class on a cross-encoder semantic metric against every

other ~4B bible-tuned model tested. See the base repo for the full model

card, version history, and the honest evaluation against larger models too.

Architecture

graph TD
    Base["Qwen/Qwen3.5-4B"]
    V31["v3.1 adapter (thematic-synthesis SFT)"]
    Cont["DMT-style continued fine-tune - RAFT-fixed thematic_qa + rehearsal, lr 5e-5"]
    Merge["merge adapter -> bf16"]
    Conv["convert_hf_to_gguf --no-mtp + llama-quantize"]
    ST["Bible-Assistant-Qwen3.5-4B-v3.2 (safetensors)"]
    GG["...-v3.2-GGUF (Q4_K_M / Q5_K_M / Q6_K / Q8_0 / F16)"]
    NV["...-v3.2-GGUF NVFP4 (advanced-gguf-quantizer, Blackwell-native)"]
    RAG["hybrid RAG: dense (nomic) + BM25 + RRF + bge-reranker-v2-m3"]
    LLM["llama.cpp / LM Studio / Ollama"]
    Base --> V31 --> Cont --> Merge --> ST
    Merge --> Conv --> GG
    Merge --> Conv --> NV
    ST --> RAG --> LLM
    GG --> LLM
    NV --> LLM

Download

Grab one file, not the whole repo.

| File | Quant | Size | Notes |

|---|---|--:|---|

| bible-v3.2-4b-Q4_K_M.gguf | Q4_K_M | 2.7 GB | recommended for general use |

| bible-v3.2-4b-nvfp4.gguf | NVFP4 | 2.5 GB | smallest; lower mean PPL ratio, higher KLD; see NVFP4-GGUF below before choosing this over Q4_K_M |

| bible-v3.2-4b-Q5_K_M.gguf | Q5_K_M | 3.1 GB | |

| bible-v3.2-4b-Q6_K.gguf | Q6_K | 3.5 GB | |

| bible-v3.2-4b-Q8_0.gguf | Q8_0 | 4.5 GB | near-lossless |

| bible-v3.2-4b-f16.gguf | F16 | 8.4 GB | full precision |

NVFP4-GGUF (Blackwell-native)

bible-v3.2-4b-nvfp4.gguf is a mixed NVFP4 quant built with

advanced-gguf-quantizer

(deep-mode tensor search, allow_diagnostic sensitivity ranking, hard-gated

on tail KLD) targeting native Blackwell FP4 tensor cores

(BLACKWELL_NATIVE_FP4=1 confirmed on an RTX 5070 Ti / SM120 at eval time) —

a genuinely different quantization technique than the k-quant ladder above,

not just another point on the same size/quality curve.

**Real, measured result — a mixed outcome, stated plainly rather than

rounded to a single "better/worse":** identical methodology, same bf16

reference, same calibration/eval corpus, both files run through

llama-perplexity --kl-divergence back-to-back.

| | Size | bpw | Mean PPL ratio vs bf16 | Mean KL-divergence vs bf16 |

|---|--:|--:|--:|--:|

| NVFP4 (this file) | 2.45 GB | 4.66 | 1.1130 ± 0.0008 | 0.1109 ± 0.0003 |

| Q4_K_M (published) | 2.71 GB | 5.15 | 1.1228 ± 0.0008 | 0.0929 ± 0.0003 |

NVFP4 is smaller and has a lower mean-token perplexity ratio than

Q4_K_M despite spending fewer bits per weight (4.66 vs 5.15 bpw) — but it

has a higher mean KL-divergence against the bf16 reference, meaning its

per-token probability distribution drifts further from the base model even

where average next-token accuracy holds up. Passed this project's own

coherence + hard tail-KLD gate (p99/p999) either way.

Which to pick: Q4_K_M if you want the closest per-token match to the

unquantized model's behavior (the safer default for a RAG assistant that

quotes verses verbatim). NVFP4 if size and mean perplexity matter more than

tail-distribution fidelity, or you're specifically exercising Blackwell's

native FP4 path.

**Status (2026-09-17): this is the current NVFP4 candidate, not a final

answer.** A follow-up run (nvfp4_mxfp6 profile, promoting the most

KLD-sensitive tensors to MXFP6_E2M3 instead of leaving the whole model at a

uniform NVFP4 floor) is queued to test whether real additional bits closes

the KLD gap to Q4_K_M without losing NVFP4's size/PPL win. This card will be

updated with that result when it lands — check back before assuming this is

the final NVFP4 quality bar for this checkpoint.

Run it in

• llama.cpp — llama-server -m <file>.gguf -ngl 99
• LM Studio (bundles a recent llama.cpp)
• koboldcpp
• Jan
• text-generation-webui
• Ollama — once its bundled llama.cpp includes this arch (see Requirements)

Requirements

Qwen3.5 is a hybrid architecture (qwen35 / Gated-DeltaNet + attention). You

need a recent llama.cpp — a build that includes the qwen35 hybrid arch

(commit 3173a56 or newer). Verified working with llama-cli/llama-server

from a source build — coherent output, correctly quoted John 3:16, 175 tok/s

on an RTX 5070 Ti.

• ✅ llama.cpp (current): llama-server -m bible-v3.2-4b-Q4_K_M.gguf -ngl 99
• ✅ LM Studio (recent versions bundle a current llama.cpp)
• ⚠️ Ollama 0.33.x: the bundled llama.cpp is too old for the qwen35 arch

(check_tensor_dims: tensor 'blk.32.attn_norm.weight' not found). Use once

Ollama updates its runtime, or run llama.cpp directly.

bible-v3.2-4b-nvfp4.gguf was built and evaluated end-to-end on an RTX 5070 Ti

(Blackwell, SM120) using llama.cpp's native NVFP4 GEMM path. It has not been

verified on non-Blackwell hardware in this session — if your llama.cpp build

or GPU lacks native FP4 kernel support, prefer one of the k-quant files above.

Thinking mode

The Qwen3.5 chat template defaults to thinking on. This model was fine-tuned

without <think> traces, so for a grounded RAG assistant you want it off:

• llama.cpp /v1/chat/completions: pass

"chat_template_kwargs": {"enable_thinking": false}, or

• use a chat template that emits a closed empty <think>\n\n</think>\n\n after

<|im_start|>assistant\n (the project's deployment/pc/Modelfile does this).

Intended use

Retrieval-augmented Bible Q&A — the model expects retrieved verses in a Context:

block, then the question. It is not designed for context-free use, medical /

legal / financial advice, counselling (it redirects those to a pastor / crisis

line), or authoritative theological rulings.

License

Weights: Apache-2.0 (inherits from Qwen3.5-4B). Project code: MIT. Bible

translations: public domain. See the base repo for full attribution and the

version history that led to this checkpoint.