---

license: apache-2.0

library_name: llama.cpp

base_model: Qwen/Qwen3.5-0.8B

tags:

- gguf

- multimodal

- vision-language

- retrieval

- late-interaction

- on-device

---

HydraQwen3.5-0.8B GGUF

This bundle runs retrieval and image-conditioned generation from one

Qwen3.5-0.8B backbone:

• Retrieval: Q6_K base + F16 LoRA + the external 320-dimensional

projection head.

• Generation: Q6_K base + F16 vision projector, with the LoRA detached.

Keep the LoRA separate. Merging it into the base removes the clean generation

mode.

Files

| File | Purpose |

|---|---|

| hydra-base-q6_k.gguf | Q6_K Qwen3.5-0.8B backbone, 630 MB |

| hydra-lora-f16.gguf | F16 retrieval LoRA, 87 MB |

| hydra-mmproj-f16.gguf | F16 vision projector for image-conditioned generation, 205 MB |

| retrieval_head/head_w.bin | Retrieval projection weight, little-endian Float32, row-major (320, 1024) |

| retrieval_head/head_b.bin | Retrieval projection bias, little-endian Float32, shape (320,) |

| ARTIFACTS.sha256 | SHA-256 checksums for the five runtime files |

No document corpus or application is included.

Download

pip install -U huggingface_hub
hf download athrael-soju/HydraQwen3.5-0.8B-GGUF \
  --local-dir HydraQwen3.5-0.8B-GGUF
cd HydraQwen3.5-0.8B-GGUF
sha256sum -c ARTIFACTS.sha256

The bundle was tested with

llama.cpp commit 2d973636.

Generation

Load the base without the retrieval LoRA:

llama-mtmd-cli \
  -m hydra-base-q6_k.gguf \
  --mmproj hydra-mmproj-f16.gguf \
  --image page.png \
  -p "What does this page contain?" \
  -n 128

Retrieval

Start a per-token embedding server with the LoRA attached:

llama-server \
  -m hydra-base-q6_k.gguf \
  --lora hydra-lora-f16.gguf \
  --embeddings --pooling none -c 1024

For a text query:

1. Tokenize the raw query without BOS or chat-template tokens.
2. Append token ID 248044 (<|endoftext|>) ten times.
3. Request all per-token hidden states from the embeddings endpoint. The result

is shaped (tokens, 1024).

1. Apply the external projection head and L2-normalize every token row.

import numpy as np

hidden = ...  # (tokens, 1024), per-token llama.cpp embeddings
weight = np.fromfile("retrieval_head/head_w.bin", dtype="<f4").reshape(320, 1024)
bias = np.fromfile("retrieval_head/head_b.bin", dtype="<f4")

embedding = hidden @ weight.T + bias
embedding /= np.linalg.norm(embedding, axis=-1, keepdims=True)

Encode document pages through the vision projector with the retrieval LoRA

attached. This needs a llama.cpp caller that can submit one non-causal

llama_batch containing the full multimodal sequence; the server command above

covers the text-query path. Splitting prefix, image, and suffix into separate

decode calls changes the page embeddings. Only the model's full-attention

blocks become bidirectional. The linear-attention blocks remain causal.

Score a query matrix Q against each page matrix P with late-interaction

MaxSim:

score = (Q @ P.T).max(axis=1).sum()

The page encoder must reproduce the Qwen3.5-VL image prompt and preprocessing:

32-pixel resize factor, 65,536 minimum pixels, 786,432 maximum pixels, and the

prompt below with the image tokens inserted between the vision markers.

<|im_start|>user
<|vision_start|><|image_pad|><|vision_end|>Describe the image.<|im_end|><|endoftext|>

Use the LoRA only for retrieval. Detach it before generation.