---

language:

• en
• zh

license: apache-2.0

library_name: llama.cpp

base_model:

• JonathanColetti/Qwen3.8-27B-Uncensored
• Qwen/Qwen3.8-27B

base_model_relation: quantized

pipeline_tag: text-generation

tags:

• unsloth
• dynamic-3.0
• ud3
• gguf
• qwen
• uncensored
• mtp
• speculative-decoding
• 16gb-vram
• llama.cpp
• text-generation
• conversational

---

<div align="center">

<img src="https://huggingface.co/DavidrPatton/Qwen3.8-27B-Uncensored-UD3-GGUF/resolve/main/banner.png" width="450" alt="Qwen3.8-27B Uncensored UD3 GGUF" style="border-radius: 12px; box-shadow: 0 4px 20px rgba(0,0,0,0.3); margin-bottom: 20px;" />

</div>

Qwen3.8-27B-Uncensored (Unsloth Dynamic 3.0 UD3-GGUF)

An advanced, hybrid mixed-precision Unsloth Dynamic 3.0 (UD3) quantization of Qwen3.8-27B-Uncensored featuring a verified, pinned Q8_0 Multi-Token Prediction (MTP) draft head (Layer 64).

Engineered specifically to fit a complete 27-Billion parameter uncensored reasoning model plus a massive 128K context window directly inside consumer 16GB VRAM GPUs (such as the NVIDIA RTX 4070 Ti SUPER, RTX 4080, RTX 3090, and RTX 4090) as well as Apple Silicon Macs and Linux workstations.

---

Demystifying the "2-Bit" Label: The Mixed-Precision Architecture

> Why is this labeled "2-bit" on Hugging Face?

> Hugging Face automatically buckets this repository under its "2-bit" filter because the base ftype identifier is Q2_K (~2.7 bits per weight on non-critical MLP blocks).

>

> However, this is NOT a degraded uniform 2-bit model.

> Standard uniform 2-bit quants (IQ2_M at 10.6 GB) compress all 64 layers equally, resulting in severe degradation of reasoning and vocabulary.

>

> This build (UD-Q2_K_XL at 12.1 GB on disk) injects 1.5 GB of extra high-precision tensor data into the most critical neural paths, delivering near-4-bit reasoning accuracy with the memory footprint of a 2-bit model.

Precision Allocation Breakdown

| # | Precision | Target Layers & Tensors | Purpose |

| :-: | :--- | :--- | :--- |

| 1 | Q8_0 / F32 | Input embeddings (token_embd), Output logits, all 17 Layer-64 NextN draft heads | Zero vocabulary loss and intact MTP speculative speed |

| 2 | Q4_K | Core attention projections (Layers 18 to 28: attn_k, attn_v, attn_o) | Full 4-bit reasoning fidelity on deep logic layers |

| 3 | Q3_K | Intermediate self-attention scoring matrices | Optimal balance between memory and attention scoring |

| 4 | IQ2_M | Bulk feed-forward network (FFN/MLP) blocks | Maximum compression on noise-resilient weights |

---

Key Advantages Over Standard Quantizations

| Feature | Standard IQ2_M | Standard Q4_K_M | This Build: UD3-Q2_K_XL |

| :--- | :--- | :--- | :--- |

| File Size on Disk | 10.6 GB | 16.8 GB | 12.1 GB (11.23 GB raw) |

| Quantization Method | Uniform 2-bit | Uniform 4-bit | Dynamic Layer-Importance Mix |

| Token Vocabulary | Degraded | Baseline | Max Precision (Q8_0 / F32) |

| MTP Draft Head | Missing / Fused | Missing / Fused | Pinned Q8_0 (Layer 64 Verified) |

| 16GB VRAM + 128K Context | Fits (quality loss) | Out of Memory | Fits Comfortably (~14.2 GB) |

| Reasoning Quality | Degraded (+0.70 PPL) | Baseline (+0.02 PPL) | Near-4-Bit Quality (~0.12 PPL) |

---

🚀 Beginner-Friendly Setup Guides (Pick Your Operating System)

Click on your operating system below for an exact, step-by-step walkthrough.

<details>

<summary><b>🪟 Windows Setup Guide (Click to open)</b></summary>

<br>

Follow these steps on Windows 10 or Windows 11:

Step 1: Check Your Hard Drives & Free Space

Before downloading the 12.1 GB model, make sure you choose a drive with at least 20 GB of free space.

• Option 1 (Visual / File Explorer):

1. On your keyboard, press the Windows Key + E to open File Explorer.

2. Click This PC on the left menu.

3. Look under Devices and drives to find which drive (e.g. C:, D:, or G:) has enough free space.

• Option 2 (PowerShell Command):

1. Press Windows Key + R, type powershell, and press Enter.

2. Copy and paste this command and press Enter:

```powershell

Get-Volume | Select-Object DriveLetter, FileSystemLabel, @{Name="FreeSpaceGB";Expression={[math]::round($_.SizeRemaining/1GB,2)}}

```

3. Pick a drive letter that has at least 20 GB free.

---

Step 2: Open PowerShell & Create Your Folder

1. Press the Windows Key + R, type powershell, and press Enter.
2. If you are using your C: drive, run:

```powershell

mkdir C:\QwenModel; cd C:\QwenModel

```

1. (If you chose a different drive like D:, run D:, then mkdir QwenModel; cd QwenModel).

---

Step 3: Download the Model Files

Copy and paste these commands into PowerShell to download the model directly into your folder:

# 1. Install the official Hugging Face downloader
pip install -U huggingface_hub

# 2. Download model files
huggingface-cli download DavidrPatton/Qwen3.8-27B-Uncensored-UD3-GGUF Qwen3.8-27B-Uncensored-UD-Q2_K_XL.gguf --local-dir .
huggingface-cli download DavidrPatton/Qwen3.8-27B-Uncensored-UD3-GGUF mmproj-BF16.gguf --local-dir .

---

Step 4: Run the Server (Choose Option A or Option B)

Option A: 1-Click Launch with Docker (Recommended)

If you have Docker Desktop installed:

1. In your QwenModel folder, create a text file named docker-compose.yml.
2. Paste the following text into the file and save it (adjust C:/QwenModel if using drive D: or G:):

services:
  qwen38-server:
    image: ghcr.io/ggml-org/llama.cpp:server-cuda
    container_name: qwen38-server
    restart: unless-stopped
    ports:
      - "9999:9999"
    volumes:
      - "C:/QwenModel:/models:ro"
    command: >
      --model "/models/Qwen3.8-27B-Uncensored-UD-Q2_K_XL.gguf"
      --mmproj "/models/mmproj-BF16.gguf"
      --n-gpu-layers 99
      --ctx-size 131072
      --cache-type-k q8_0
      --cache-type-v q4_0
      --temp 1.0
      --top-p 0.95
      --top-k 20
      --min-p 0.05
      --dry-multiplier 0.6
      --dry-base 1.75
      --dry-allowed-length 2
      --xtc-probability 0.1
      --flash-attn on
      --spec-type draft-mtp
      --spec-draft-n-max 2
      -b 4096
      -ub 1024
      --cont-batching
      --parallel 1
      --reasoning-preserve
      --reasoning-budget 1024
      --metrics
      --host 0.0.0.0
      --port 9999
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]

> 💡 How to Change the Port in Docker:

> If port 9999 is already in use by another app, you can change it to any port you want (e.g. 8080).

> Simply update both occurrences in the file:

> 1. ports: - "8080:8080"

> 2. --port 8080

1. In PowerShell inside your folder, start the server:

docker compose up -d

Option B: Run Directly on Windows (No Docker)

1. Download the pre-built Windows CUDA ZIP from llama.cpp Releases (look for llama-b*-bin-win-cuda-cu12.4-x64.zip).
2. Extract the files into your QwenModel folder.
3. In PowerShell, run:

.\llama-server.exe `
  -m "Qwen3.8-27B-Uncensored-UD-Q2_K_XL.gguf" `
  --mmproj "mmproj-BF16.gguf" `
  -ngl 99 `
  -c 131072 `
  --cache-type-k q8_0 `
  --cache-type-v q4_0 `
  --temp 1.0 `
  --top-p 0.95 `
  --top-k 20 `
  --min-p 0.05 `
  --dry-multiplier 0.6 `
  --dry-base 1.75 `
  --dry-allowed-length 2 `
  --xtc-probability 0.1 `
  --flash-attn on `
  --spec-type draft-mtp `
  --spec-draft-n-max 2 `
  -b 4096 `
  -ub 1024 `
  --cont-batching `
  --reasoning-preserve `
  --reasoning-budget 1024 `
  --port 9999

> 💡 How to Change the Port in Native Mode:

> Simply change --port 9999 to your desired port number (for example --port 8080).

---

Step 5: Start Chatting!

Open your web browser (Chrome, Edge, Firefox) and go to:

👉 http://localhost:9999

You will see the interactive chat window. Type a message and watch the model reason inside <think> tags and generate code!

</details>

<details>

<summary><b>🍎 macOS Setup Guide (Apple Silicon M1 / M2 / M3 / M4) (Click to open)</b></summary>

<br>

Follow these steps on Apple Silicon Macs (16GB+ Unified Memory recommended):

Step 1: Check Available Storage

1. Click the Apple Menu () at the top left -> System Settings -> General -> Storage.
2. Make sure you have at least 20 GB of free space available.

Step 2: Open Terminal & Create Your Folder

1. On your Mac keyboard, press Command (⌘) + Space to open Spotlight.
2. Type Terminal and press Return.
3. Create your folder and enter it:

```bash

mkdir -p ~/QwenModel && cd ~/QwenModel

```

Step 3: Download the Model Files

Run these commands in Terminal:

curl -L -O https://huggingface.co/DavidrPatton/Qwen3.8-27B-Uncensored-UD3-GGUF/resolve/main/Qwen3.8-27B-Uncensored-UD-Q2_K_XL.gguf
curl -L -O https://huggingface.co/DavidrPatton/Qwen3.8-27B-Uncensored-UD3-GGUF/resolve/main/mmproj-BF16.gguf

Step 4: Install and Run llama.cpp with Apple Metal GPU

1. Install llama.cpp using Homebrew:

brew install llama.cpp

1. Start the server with full Apple Metal GPU acceleration on port 9999:

llama-server \
  -m ~/QwenModel/Qwen3.8-27B-Uncensored-UD-Q2_K_XL.gguf \
  --mmproj ~/QwenModel/mmproj-BF16.gguf \
  -ngl 99 \
  -c 32768 \
  --temp 1.0 \
  --top-p 0.95 \
  --top-k 20 \
  --min-p 0.05 \
  --flash-attn on \
  --spec-type draft-mtp \
  --spec-draft-n-max 2 \
  --reasoning-preserve \
  --reasoning-budget 1024 \
  --port 9999

> 💡 How to Change the Port on Mac:

> Change --port 9999 in the command above to any port number you prefer (such as --port 8080).

Step 5: Start Chatting!

Open Safari or Chrome and navigate to:

👉 http://localhost:9999

</details>

<details>

<summary><b>🐧 Linux Setup Guide (Ubuntu / Debian / Arch) (Click to open)</b></summary>

<br>

Follow these steps on Linux with an NVIDIA GPU:

Step 1: Open Terminal & Check Disk Space

1. Press Ctrl + Alt + T to open the terminal.
2. Check your free disk space by typing:

```bash

df -h /home

```

Make sure you have at least 20 GB available.

Step 2: Create Folder & Download Files

mkdir -p ~/QwenModel && cd ~/QwenModel

# Download model weights
pip install -U huggingface_hub
huggingface-cli download DavidrPatton/Qwen3.8-27B-Uncensored-UD3-GGUF Qwen3.8-27B-Uncensored-UD-Q2_K_XL.gguf --local-dir ~/QwenModel
huggingface-cli download DavidrPatton/Qwen3.8-27B-Uncensored-UD3-GGUF mmproj-BF16.gguf --local-dir ~/QwenModel

Step 3: Run with Docker Compose on Port 9999

1. Create docker-compose.yml in ~/QwenModel:

services:
  qwen38-server:
    image: ghcr.io/ggml-org/llama.cpp:server-cuda
    container_name: qwen38-server
    restart: unless-stopped
    ports:
      - "9999:9999"
    volumes:
      - "$HOME/QwenModel:/models:ro"
    command: >
      --model "/models/Qwen3.8-27B-Uncensored-UD-Q2_K_XL.gguf"
      --mmproj "/models/mmproj-BF16.gguf"
      --n-gpu-layers 99
      --ctx-size 131072
      --cache-type-k q8_0
      --cache-type-v q4_0
      --temp 1.0
      --top-p 0.95
      --top-k 20
      --min-p 0.05
      --dry-multiplier 0.6
      --dry-base 1.75
      --dry-allowed-length 2
      --xtc-probability 0.1
      --flash-attn on
      --spec-type draft-mtp
      --spec-draft-n-max 2
      -b 4096
      -ub 1024
      --cont-batching
      --parallel 1
      --reasoning-preserve
      --reasoning-budget 1024
      --metrics
      --host 0.0.0.0
      --port 9999
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]

> 💡 How to Change the Port on Linux:

> In docker-compose.yml, change ports: - "9999:9999" and --port 9999 to your desired port number.

1. Start the container:

docker compose up -d

Step 4: Start Chatting!

Open your browser and navigate to http://localhost:9999.

</details>

<details>

<summary><b>🔌 Connect to AI Coding Editors (ZCode, Cursor, Zed) (Click to open)</b></summary>

<br>

Because the server provides an OpenAI-compatible API endpoint at http://localhost:9999/v1, you can connect your favorite coding editors:

In ZCode

1. Open ZCode Settings -> Add Custom Model.
2. Provider: OpenAI-compatible.
3. Base URL: http://127.0.0.1:9999/v1
4. API Key: local (or leave blank).
5. Model Name: Qwen3.8-27B-Uncensored
6. Context Window: 131072

In Cursor

1. Open Cursor Settings -> Models.
2. Under OpenAI API Key, type: local.
3. Under Base URL, type: http://localhost:9999/v1.
4. Add Model Name: Qwen3.8-27B-Uncensored.

In Zed

1. In Zed Settings (settings.json), add a custom endpoint:

{
  "language_models": {
    "openai": {
      "api_url": "http://localhost:9999/v1",
      "available_models": [
        {
          "name": "Qwen3.8-27B-Uncensored",
          "max_tokens": 131072
        }
      ]
    }
  }
}

</details>

---

📊 Performance Benchmarks (Tested on RTX 4070 Ti SUPER 16GB)

• Generation Speed: ~58 to 64 tokens/second (with MTP speculative decoding enabled).
• Prompt Prefill Speed: 2,500+ tokens/second (-b 4096 -ub 1024 batch evaluation).
• VRAM Allocation:

* Base Model Weights: 11.8 GB

* 128K Context Buffer (q8_0 Keys / q4_0 Values): 2.4 GB

* Total GPU VRAM Footprint: ~14.2 GB (Cleanly fits inside 16GB VRAM with zero PCIe swapping).

---

🙏 Credits & Heritage

• Base Uncensored Weights: Created by Jonathan Coletti (JonathanColetti/Qwen3.8-27B-Uncensored) using Heretic orthogonal abliteration at BF16.
• Base Architecture: Developed by the Qwen Team / Alibaba Cloud (Qwen/Qwen3.8-27B).
• Dynamic 3.0 Quantization: Layer importance mapping, MTP draft pinning, and local quantization build engineered by David Patton.