---

license: apache-2.0

base_model: unsloth/Qwen3.5-9B

tags:

• text-generation
• vision
• multimodal
• gguf
• llama.cpp
• qwen3_5
• image-to-text

---

Qwen3.5-9B-AltText-v4 - GGUF

This repository contains GGUF quantized models for Jakelolipopp/Qwen3.5-9B-AltText-v4-LORA, fine-tuned from base model unsloth/Qwen3.5-9B.

Full merged safetensors weights are available at Jakelolipopp/Qwen3.5-9B-AltText-v4-merged.

The models and multimodal vision projector (mmproj) were converted and quantized using llama.cpp.

Available Quantization Files

| File Name | Size |

| :--- | :--- |

| Qwen3.5-9B-AltText-v4-BF16.gguf | 16.69 GB |

| Qwen3.5-9B-AltText-v4-Q2_K.gguf | 3.56 GB |

| Qwen3.5-9B-AltText-v4-Q3_K_M.gguf | 4.31 GB |

| Qwen3.5-9B-AltText-v4-Q4_K_M.gguf | 5.24 GB |

| Qwen3.5-9B-AltText-v4-Q5_K_M.gguf | 6.02 GB |

| Qwen3.5-9B-AltText-v4-Q6_K.gguf | 6.85 GB |

| Qwen3.5-9B-AltText-v4-Q8_0.gguf | 8.87 GB |

| mmproj-Qwen3.5-9B-AltText-v4-BF16.gguf | 0.86 GB |

| mmproj-Qwen3.5-9B-AltText-v4-Q8_0.gguf | 0.58 GB |

---

Quantization Details

Base Models

• Q8_0: Near-lossless quality, highest precision (recommended for maximum accuracy).
• Q6_K: High quality, virtually indistinguishable from full precision with noticeable size reduction.
• Q5_K_M: Great balance between quality and speed/VRAM usage (Recommended default).
• Q4_K_M: Fast inference, compact size, great for lower memory devices.
• Q3_K_M / Q2_K: Ultra-compressed for minimal resource footprints.

Multimodal Projectors (mmproj)

• mmproj-...-BF16.gguf: Full bfloat16 vision projector for maximum visual fidelity.
• mmproj-...-Q8_0.gguf: 8-bit quantized vision projector with lower VRAM overhead.

---

How to Run with llama.cpp

Make sure you pass both the model (-m) and the vision projector (--mmproj):

# Run interactive CLI with an image
./llama-cli \
  -m Qwen3.5-9B-AltText-v4-Q5_K_M.gguf \
  --mmproj mmproj-Qwen3.5-9B-AltText-v4-BF16.gguf \
  --image /path/to/image.jpg \
  -p "Generate detailed alternative text describing this image:" \
  -c 4096 --temp 0.2

Running Server Mode

./llama-server \
  -m Qwen3.5-9B-AltText-v4-Q5_K_M.gguf \
  --mmproj mmproj-Qwen3.5-9B-AltText-v4-BF16.gguf \
  --port 8080 -c 4096

---

How to Run with Ollama

Create a Modelfile:

FROM ./Qwen3.5-9B-AltText-v4-Q5_K_M.gguf
MMPROJ ./mmproj-Qwen3.5-9B-AltText-v4-BF16.gguf

PARAMETER temperature 0.2
PARAMETER top_p 0.95

Build and run the model:

ollama create qwen3.5-9b-alttext-v4 -f Modelfile
ollama run qwen3.5-9b-alttext-v4 "Describe this image: /path/to/image.png"