nbeerbower/huihui-qwen3.5-27b-abliterated-athanorlite-orpo-v2-gguf overview

{
  "metadata": {},
  "card_data": {
    "library_name": "gguf",
    "tags": [
      "merlina",
      "grimoire",
      "text-generation",
      "orpo",
      "gguf"
    ],
    "base_model": [
      "nbeerbower/Huihui-Qwen3.5-27B-abliterated-Athanorlite-ORPO-v2"
    ],
    "frontmatter": {
      "library_name": "gguf",
      "tags": [
        "merlina",
        "grimoire",
        "text-generation",
        "orpo",
        "gguf"
      ],
      "base_model": [
        "nbeerbower/Huihui-Qwen3.5-27B-abliterated-Athanorlite-ORPO-v2"
      ]
    },
    "hero_image_url": "https://raw.githubusercontent.com/Schneewolf-Labs/Merlina/refs/heads/main/frontend/madewithmerlina_smol.png",
    "summary": "GGUF quantizations of nbeerbower/Huihui-Qwen3.5-27B-abliterated-Athanorlite-ORPO-v2. This is a multimodal (vision-language) model. You need **both** a text model GGUF and the mmproj file for full functionality.",
    "quick_links": [],
    "benchmark_table_html": "",
    "readme_markdown": "---\nlibrary_name: gguf\ntags:\n- merlina\n- grimoire\n- text-generation\n- orpo\n- gguf\nbase_model:\n- nbeerbower/Huihui-Qwen3.5-27B-abliterated-Athanorlite-ORPO-v2\n---\n\n# Huihui-Qwen3.5-27B-abliterated-Athanorlite-ORPO-v2-GGUF\n\nGGUF quantizations of [nbeerbower/Huihui-Qwen3.5-27B-abliterated-Athanorlite-ORPO-v2](https://huggingface.co/nbeerbower/Huihui-Qwen3.5-27B-abliterated-Athanorlite-ORPO-v2).\n\nThis is a multimodal (vision-language) model. You need **both** a text model GGUF and the mmproj file for full functionality.\n\n## Available Quantizations\n\n| Quant | Size | BPW | Description |\n|-------|------|-----|-------------|\n| [Q8_0](./Huihui-Qwen3.5-27B-abliterated-Athanorlite-ORPO-v2-Q8_0.gguf) | 27 GB | 8.50 | Best quality, near-lossless |\n| [Q6_K](./Huihui-Qwen3.5-27B-abliterated-Athanorlite-ORPO-v2-Q6_K.gguf) | 21 GB | 6.57 | Great quality, good size balance |\n| [Q4_K_M](./Huihui-Qwen3.5-27B-abliterated-Athanorlite-ORPO-v2-Q4_K_M.gguf) | 16 GB | 4.92 | Recommended default |\n| [Q3_K_M](./Huihui-Qwen3.5-27B-abliterated-Athanorlite-ORPO-v2-Q3_K_M.gguf) | 13 GB | 3.86 | For constrained VRAM |\n\n### Vision Projector (required for multimodal)\n\n| File | Size | Type |\n|------|------|------|\n| [mmproj-F16](./Huihui-Qwen3.5-27B-abliterated-Athanorlite-ORPO-v2-mmproj-F16.gguf) | 885 MB | F16 |\n\n## Hardware Recommendations\n\n| Setup | Recommended Quant |\n|-------|-------------------|\n| 1x 48 GB (A6000, RTX 6000 Ada) | Q8_0 |\n| 2x 24 GB (RTX 3090/4090) | Q8_0 split across GPUs |\n| 1x 24 GB (RTX 3090/4090) | Q6_K |\n| 2x 16 GB (RTX 4060 Ti) | Q4_K_M or Q6_K split |\n| 1x 16 GB (RTX 4060 Ti) | Q3_K_M |\n\n> VRAM usage = text model + mmproj (885 MB) + KV cache (varies with context length). Leave at least 2-4 GB headroom for KV cache and overhead.\n\n## Usage\n\n### llama.cpp CLI\n\n```bash\n# Text-only\nllama-cli -m Huihui-Qwen3.5-27B-abliterated-Athanorlite-ORPO-v2-Q4_K_M.gguf -p \"Hello!\"\n\n# With vision (image input)\nllama-mtmd-cli \\\n  -m Huihui-Qwen3.5-27B-abliterated-Athanorlite-ORPO-v2-Q4_K_M.gguf \\\n  --mmproj Huihui-Qwen3.5-27B-abliterated-Athanorlite-ORPO-v2-mmproj-F16.gguf \\\n  --image photo.jpg \\\n  -p \"Describe this image.\"\n```\n\n### llama.cpp Server\n\n```bash\nllama-server \\\n  -m Huihui-Qwen3.5-27B-abliterated-Athanorlite-ORPO-v2-Q4_K_M.gguf \\\n  --mmproj Huihui-Qwen3.5-27B-abliterated-Athanorlite-ORPO-v2-mmproj-F16.gguf \\\n  --port 8080\n```\n\n### Multi-GPU split\n\n```bash\n# Example: 2x 24GB GPUs\nllama-server \\\n  -m Huihui-Qwen3.5-27B-abliterated-Athanorlite-ORPO-v2-Q8_0.gguf \\\n  --mmproj Huihui-Qwen3.5-27B-abliterated-Athanorlite-ORPO-v2-mmproj-F16.gguf \\\n  -ngl 99 --tensor-split 1,1\n```\n\n## About\n\nORPO fine-tune of [huihui-ai/Huihui-Qwen3.5-27B-abliterated](https://huggingface.co/huihui-ai/Huihui-Qwen3.5-27B-abliterated) on [schneewolflabs/Athanorlite-DPO](https://huggingface.co/datasets/schneewolflabs/Athanorlite-DPO). The original upload had broken state dict keys from a PEFT merge bug; the v2 safetensors model has corrected key naming and restored multimodal/MTP weights. See the [v2 model card](https://huggingface.co/nbeerbower/Huihui-Qwen3.5-27B-abliterated-Athanorlite-ORPO-v2) for details.\n\n---\n\n![Trained with Merlina](https://raw.githubusercontent.com/Schneewolf-Labs/Merlina/refs/heads/main/frontend/madewithmerlina_smol.png)\n\n[Merlina on GitHub](https://github.com/Schneewolf-Labs/Merlina)\n",
    "related_quantizations": []
  },
  "tags": [
    "gguf",
    "merlina",
    "grimoire",
    "text-generation",
    "orpo",
    "base_model:nbeerbower/Huihui-Qwen3.5-27B-abliterated-Athanorlite-ORPO-v2",
    "base_model:quantized:nbeerbower/Huihui-Qwen3.5-27B-abliterated-Athanorlite-ORPO-v2",
    "endpoints_compatible",
    "region:us",
    "conversational"
  ],
  "likes": 0,
  "downloads": 383,
  "gated": false,
  "private": false,
  "last_modified": "2026-04-14T00:50:30.000Z",
  "created_at": "2026-04-13T23:09:52.000Z",
  "pipeline_tag": "text-generation",
  "library_name": "gguf"
}

{
  "_id": "69dd77c0b81d8f5302126ee2",
  "id": "nbeerbower/Huihui-Qwen3.5-27B-abliterated-Athanorlite-ORPO-v2-GGUF",
  "modelId": "nbeerbower/Huihui-Qwen3.5-27B-abliterated-Athanorlite-ORPO-v2-GGUF",
  "sha": "aebe43a11c03ef6fc42b07c183874a630eba505e",
  "createdAt": "2026-04-13T23:09:52.000Z",
  "lastModified": "2026-04-14T00:50:30.000Z",
  "author": "nbeerbower",
  "downloads": 383,
  "likes": 0,
  "gated": false,
  "private": false,
  "pipeline_tag": "text-generation",
  "library_name": "gguf",
  "siblings_count": 7
}