---

library_name: gguf

tags:

- gguf

- quantized

- mixed-precision

- imatrix

- qwen

- qwen3

- moe

license: apache-2.0

base_model: Qwen/Qwen3.6-35B-A3B

base_model_relation: quantized

pipeline_tag: text-generation

language: en

---

Qwen3.6-35B-A3B — 26GB (GGUF)

Mixed-precision GGUF quantization of Qwen/Qwen3.6-35B-A3B, prepared by baa.ai.

Files

| File | Quant | Size |

|------|-------|------|

| Qwen3.6-35B-A3B-RAM-26GB.gguf | Mixed (Q3_K–F16) | 26 GB |

Metrics

| Metric | Value |

|---|---|

| Size on disk | 26.3 GB |

| Average bits per weight | 6.01 |

| Base type | Q4_K_M (per-tensor overrides via spec) |

| Framework | llama.cpp (GGUF) |

| Calibration | Importance matrix (wikitext-2 + MMLU-Pro, seed=99) |

| Source | Qwen/Qwen3.6-35B-A3B (BF16, 71.9 GB) |

Bit distribution (620 quantized tensors)

| Type | Count | Purpose |

|---|---|---|

| F16 | 363 | Attention (q/k/v/o), MoE routers, SSM projections, norms |

| Q8_0 | 93 | High-sensitivity MoE experts + shared-expert projections |

| Q6_K | 63 | Medium-sensitivity MoE experts |

| Q5_K | 77 | Lower-sensitivity MoE experts |

| Q4_K | 7 | Low-sensitivity SSM auxiliary tensors |

| Q3_K | 17 | SSM conv/gate tensors (lowest sensitivity) |

Benchmarks

Measured on MMLU-Pro (140 stratified questions, 10 per subject, seed=42, thinking mode on):

| Model | MMLU-Pro Accuracy | Size |

|---|---|---|

| MLX RAM sibling | 77.9% | 25.9 GB |

| F16 GGUF (unquantized ceiling) | 70.7% | 65 GB |

| This model | 69.3% | 26.3 GB |

Captures >97% of the GGUF format's achievable quality at ~40% of the F16 size. The gap between MLX and F16 GGUF reflects llama.cpp's handling of Qwen3.6's hybrid linear-attention + full-attention + SSM architecture, not quantization overhead.

Usage

Every option below is a complete start-from-zero recipe. Pick one.

> Disk space: leave ~30 GB free. The GGUF is 26.3 GB; some runtimes keep a working copy.

Option 1 — llama.cpp (CLI)

# 1. Install llama.cpp (macOS / Linux)
brew install llama.cpp huggingface-cli
# On Linux without brew:
#   pip install huggingface_hub
#   see https://github.com/ggml-org/llama.cpp for build instructions

# 2. Download this model (~26 GB)
hf download baa-ai/Qwen3.6-35B-A3B-RAM-26GB-GGUF \
    --include "*.gguf" \
    --local-dir ./qwen3.6-ram-26gb

# 3. Run interactively
llama-cli -m ./qwen3.6-ram-26gb/Qwen3.6-35B-A3B-RAM-26GB.gguf \
    -p "Explain quantum entanglement in one paragraph." \
    -n 512 --threads 8 -ngl 99

Option 2 — llama-server (OpenAI-compatible HTTP API)

# 1-2. Install + download (same as Option 1)
brew install llama.cpp huggingface-cli
hf download baa-ai/Qwen3.6-35B-A3B-RAM-26GB-GGUF \
    --include "*.gguf" \
    --local-dir ./qwen3.6-ram-26gb

# 3. Start the server
llama-server \
    -m ./qwen3.6-ram-26gb/Qwen3.6-35B-A3B-RAM-26GB.gguf \
    --port 8080 -ngl 99 --ctx-size 8192

# 4. Query it (in another terminal)
curl http://localhost:8080/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{
        "messages": [{"role": "user", "content": "Hello!"}],
        "max_tokens": 256,
        "temperature": 1.0
    }'

Option 3 — Ollama

Assumes Ollama is already installed and its daemon is running.

# 1. Download the GGUF into a working directory
mkdir -p qwen3.6-ram-26gb && cd qwen3.6-ram-26gb
hf download baa-ai/Qwen3.6-35B-A3B-RAM-26GB-GGUF \
    --include "*.gguf" \
    --local-dir .

# 2. Create a Modelfile next to the GGUF
cat > Modelfile <<'EOF'
FROM ./Qwen3.6-35B-A3B-RAM-26GB.gguf
PARAMETER temperature 1.0
PARAMETER top_p 0.95
PARAMETER top_k 40
PARAMETER repeat_penalty 1.1
PARAMETER num_ctx 8192
EOF

# 3. Import into Ollama and run
ollama create qwen3.6-ram-26gb -f Modelfile
ollama run qwen3.6-ram-26gb "Hello!"

Option 4 — LM Studio (GUI)

1. Install LM Studio and launch it
2. Search for baa-ai/Qwen3.6-35B-A3B-RAM-26GB-GGUF in the Discover tab (or paste the repo URL)
3. Click Download — LM Studio will fetch the GGUF into its local cache
4. Load the model from the My Models tab; use the Chat tab to converse

Option 5 — Python (llama-cpp-python)

# 1. Install bindings
pip install llama-cpp-python huggingface_hub

# 2. Download the GGUF
hf download baa-ai/Qwen3.6-35B-A3B-RAM-26GB-GGUF \
    --include "*.gguf" \
    --local-dir ./qwen3.6-ram-26gb

# 3. Use it
python - <<'PY'
from llama_cpp import Llama
llm = Llama(
    model_path="./qwen3.6-ram-26gb/Qwen3.6-35B-A3B-RAM-26GB.gguf",
    n_ctx=8192,
    n_gpu_layers=-1,   # all layers to GPU on Apple Silicon / CUDA
)
out = llm.create_chat_completion(
    messages=[{"role": "user", "content": "Hello!"}],
    max_tokens=256,
    temperature=1.0,
)
print(out["choices"][0]["message"]["content"])
PY

Recommended inference settings

temperature: 1.0
top_p: 0.95
top_k: 40
repetition_penalty: 1.1
max_tokens: 8192

Enable thinking mode in the chat template for best accuracy on reasoning tasks.

Quantization method

1. RAM probe allocator measures per-tensor sensitivity to quantization across bits 2–8
2. Structural-bounds + proxy prediction locates the global quality peak budget
3. Path B knapsack solver re-optimizes allocations in GGUF type space using GGUF-adjusted effective bits and actual GGUF byte costs
4. F16 kept for the most sensitive tensors (attention projections, MoE routers, SSM projections, norms)
5. Importance matrix computed from 100 chunks of wikitext-2 + 200 MMLU-Pro questions (seed=99, disjoint from evaluation seed)
6. llama-quantize applies the per-tensor spec with --imatrix

No fine-tuning, no calibration-dependent optimization — the probe runs on random Gaussian inputs only.

---

Quantized by baa.ai

---

Black Sheep AI Products

Black Sheep AI builds private AI with the Bounded AI Architecture (BAA): three layers, each bound to something you control, on infrastructure you control. Nothing leaves your boundary.

Ram (Reasoning Layer) — Full-capability frontier models, shrunk to fit smaller GPUs with reasoning kept and certified. They rely strictly on your Data Modules for knowledge, so you swap the model as better ones ship.

Paddock (Knowledge Plane) — Binds your Data Modules directly to the reasoning layer: one per topic, period, person, permission or tenant. Permission, freeze, hand back or delete each one on its own.

Watchman (GRC Layer) — Governance, risk and compliance in a single, complete audit log: every module permission, every bind, every model's provenance. Five certificates before anything goes live.

Shepherd — Takes a frontier model, compresses it into a Ram reasoning model, certifies it with Watchman, binds it to your Paddock, and deploys the Bounded AI Architecture to your infrastructure: your cloud, a Mac Studio, or an air-gapped facility.

Learn more at baa.ai — Private AI for enterprises whose data cannot leave the building.