---

language:

• en

license: apache-2.0

base_model: inclusionAI/Ling-3.0-tiny

tags:

• gguf
• llama.cpp
• quantized
• moe
• bailingmoev3
• hybrid-model
• local-llm
• text-generation

pipeline_tag: text-generation

---

Ling-3.0-tiny-GGUF

GGUF quantizations of inclusionAI/Ling-3.0-tiny, converted for use with compatible llama.cpp-based runtimes.

This repository includes a complete selection of standard K-quants and importance-matrix (IQ) quantizations, so you can choose the best balance of model size, speed, and output quality for your hardware.

> Runtime compatibility: Ling-3.0-tiny uses the BailingMoeV3 / hybrid architecture. Use a runtime with explicit support for this architecture. Generic or older llama.cpp builds may not load these files correctly.

Available files

| Quantization | Best for |

|---|---|

| F16 | Highest-fidelity baseline; re-quantization and high-memory systems |

| Q8_0 | Near-F16 quality with substantially lower memory use |

| Q6_K | High-quality local inference |

| Q5_K_M | Strong quality-to-size balance |

| Q5_K_S | Slightly smaller alternative to Q5_K_M |

| Q5_0 | Legacy-style 5-bit option |

| Q4_K_M | Recommended default for most users |

| Q4_K_S | Smaller Q4 K-quant alternative |

| Q4_0 | Compact legacy-style 4-bit option |

| IQ4_NL | High-quality importance-matrix 4-bit option |

| IQ4_XS | Compact importance-matrix 4-bit option |

| Q3_K_L | Higher-quality 3-bit K-quant |

| Q3_K_M | Balanced 3-bit K-quant |

| Q3_K_S | Smaller 3-bit K-quant |

| IQ3_M | Strong quality-per-GB option for constrained systems |

| IQ3_S | Smaller 3-bit IQ option |

| IQ3_XS | Very compact IQ 3-bit option |

| IQ3_XXS | Extremely compact IQ 3-bit option |

| Q2_K | Low-memory K-quant option |

| IQ2_M | Compact IQ quant with better quality potential than very-low-bit options |

| IQ2_S | Low-memory IQ option |

| IQ2_XS | Very small IQ option |

| IQ2_XXS | Extremely small IQ option |

| IQ1_M | Experimental ultra-low-memory option |

| IQ1_S | Smallest experimental option |

Recommended downloads

| Your priority | Recommended file |

|---|---|

| Best quality | Ling-3.0-tiny-F16.gguf |

| Near-original quality | Ling-3.0-tiny-Q8_0.gguf |

| High quality with lower memory use | Ling-3.0-tiny-Q6_K.gguf |

| Best general-purpose choice | Ling-3.0-tiny-Q4_K_M.gguf |

| Small but capable | Ling-3.0-tiny-IQ3_M.gguf |

| Tight VRAM / RAM budget | Ling-3.0-tiny-IQ2_M.gguf |

| Experimental minimum size | Ling-3.0-tiny-IQ1_S.gguf |

For most users, start with Q4_K_M. If you have more RAM or VRAM, try Q5_K_M, Q6_K, or Q8_0. IQ quants can offer attractive quality-to-size trade-offs, but results and compatibility may vary by runtime and hardware.

Usage

Download one .gguf file, then run it with a compatible build of llama.cpp.

llama-cli \
  -m Ling-3.0-tiny-Q4_K_M.gguf \
  -ngl 99 \
  -c 4096 \
  -p "Write a concise explanation of retrieval-augmented generation."

-ngl 99 attempts to offload all supported layers to the GPU. Remove it or set -ngl 0 for CPU-only inference.

Important notes

• These files are quantized derivatives of the original model; output quality changes depending on the chosen quantization.
• Very low-bit quants, especially IQ1 and IQ2 variants, are intended for memory-constrained or experimental use and may noticeably reduce output quality.
• Use the original model’s license, terms, and usage requirements.
• Validate the selected quantization on your own workload before production use.

Conversion details

• Base model: inclusionAI/Ling-3.0-tiny
• Format: GGUF
• Conversion/runtime branch: BailingMoeV3-compatible llama.cpp fork
• Standard K-quants: generated from the F16 GGUF
• IQ quants: generated using an importance matrix calibrated on a text corpus

Credits

• Original model by inclusionAI
• GGUF conversion and quantization by NANI-Nithin
• GGUF tooling by the llama.cpp community

Disclaimer

This is a community GGUF conversion and is not an official release by inclusionAI. Please report conversion, loading, or compatibility issues in this repository’s Discussions section.

Reproducibility

This repository was generated with a BailingMoeV3-enabled llama.cpp fork.

The exact source checkout checkpoint is recorded below:

{
  "stage": "01_checkout_bailing_llama",
  "status": "complete",
  "timestamp_utc": "2026-08-11T10:36:13.114546+00:00",
  "model": "inclusionAI/Ling-3.0-tiny",
  "llama_repo": "https://github.com/aetherbird/llama.cpp.git",
  "llama_branch": "bailingmoe3-support",
  "repo_dir": "/mnt/ling/src/llama.cpp",
  "commit": "3a0124fa8c20356ed5e6bf0c0ebae1566d6f49c1"
}

Files

• F16: Conversion baseline.
• Q4_K_M: General local-inference default.
• Q5_K_M, Q6_K, Q8_0: Higher-fidelity variants.
• IQ*: Importance-matrix variants, generated only when supported by the pinned quantizer.

Use a Ling/BailingMoeV3-compatible runtime to load these files.