---

license: other

license_name: openmdw-1.1

license_link: https://openmdw.ai/license/1-1/

base_model: EryriLabs/Lightning-Herald-30B-A3B

tags:

- gguf

- llama.cpp

- hermes-agent

- agentic

- tool-calling

- nemotron

pipeline_tag: text-generation

---

Lightning-Herald-30B-A3B GGUF

<p align="center">

<img src="Lightning-Herald-30B-A3B.jpg" alt="Lightning-Herald" width="360">

</p>

GGUF builds of EryriLabs/Lightning-Herald-30B-A3B,

my Hermes-Agent tune of NVIDIA Nemotron 3.5 Lightning (30B total, 3B active).

The full model card, benchmarks and training details live on the main repo.

This one is just the files.

Files

| Quant | Size | Notes |

|---|---|---|

| IQ4_XS | 19 GB | The exact file used in the before/after benchmark. Best fit for one 24GB card with a big context. |

| Q4_K_M | 25 GB | Standard 4-bit. Needs two GPUs or some CPU offload on 24GB. |

| Q5_K_M | 27 GB | |

| Q6_K | 35 GB | |

| Q8_0 | 35 GB | Close to lossless. |

All made with llama.cpp b10413 (convert_hf_to_gguf.py then llama-quantize)

from the released BF16 weights. You need a llama.cpp build with Nemotron-H

support, so b10413 or newer.

Run

llama-server -m Lightning-Herald-30B-A3B-IQ4_XS.gguf --jinja -ngl 999 -c 32768 --temp 1.0 --top-p 0.95

--jinja matters: it makes llama.cpp render the chat template with your tools

and parse tool calls into structured tool_calls. The sampling above is

NVIDIA's recommendation for the base model and what the benchmark used.

Point Hermes Agent at http://127.0.0.1:8080/v1. A config preset is on the

main repo.

Licence

OpenMDW-1.1, inherited from the base. Training data was Apache 2.0.

---

Dwain Barnes, EryriLabs