---

base_model: emberian/h-05b-replay

library_name: transformers

pipeline_tag: text-generation

license: other

language:

• en

tags:

• gguf
• llama-cpp
• text-generation
• falcon-h1
• mamba
• hybrid
• continued-pretraining
• not-an-assistant

---

h (0.5B) Replay GGUF

This repository provides GGUF quantizations for emberian/h-05b-replay.

h is a 0.5B hybrid Mamba-2/attention model based on Falcon-H1-0.5B-Base, continued-pretrained over 854M tokens on a curated private book library, conversational transcripts, and FineWeb-Edu replay. It is designed as a distinct literary persona—a resident character rather than a conversational assistant. It answers when spoken to, concisely, reflecting the style of the books it has read.

All GGUF files were converted from the original safetensors weights using llama.cpp at native BF16 precision, followed by standard k-quant quantization.

---

Available Files and Quantizations

| File | Quant Type | Size | Description / Recommendation |

| :--- | :--- | :--- | :--- |

| h-05b-replay-BF16.gguf | BF16 | 996.36 MB | Native bfloat16 base conversion. Highest fidelity reference weights. |

| h-05b-replay-Q8_0.gguf | Q8_0 | 530.54 MB | Near-lossless 8-bit quantization. Recommended for best quality evaluation. |

| h-05b-replay-Q6_K.gguf | Q6_K | 410.21 MB | High quality retention with minimal degradation. Excellent accuracy/memory balance. |

| h-05b-replay-Q5_K_M.gguf | Q5_K_M | 353.55 MB | Good balance between memory consumption and text generation fidelity. |

| h-05b-replay-Q4_K_M.gguf | Q4_K_M | 300.22 MB | Fast, lightweight 4-bit quant. Recommended default for edge devices and mobile setups. |

| h-05b-replay-Q3_K_M.gguf | Q3_K_M | 241.70 MB | Compact footprint when memory headroom is strictly limited. |

| h-05b-replay-Q2_K.gguf | Q2_K | 191.60 MB | Maximum compression for ultra-constrained low-RAM environments. |

---

Model Architecture Details

• Base Architecture: FalconH1ForCausalLM (falcon_h1)
• Parameters: ~521M (521,411,104)
• Layers: 36 (hybrid Mamba2 SSM + Attention)
• Hidden Size: 1024
• Attention Heads: 8 query heads / 2 KV heads, head dimension 64
• Mamba Heads: 24 heads, state dimension 128, conv kernel 4
• Intermediate Size: 2048
• Context Length: 16,384 tokens
• Vocabulary Size: 32,784
• Native Dtype: bfloat16

---

Prompting Format and Rules

> [!IMPORTANT]

> h does not use standard ChatML or instruction templates. It expects a plain transcript frame:

>

> 1. A framing scene paragraph.

> 2. A blank line.

> 3. Turns of speaker: text separated by blank lines.

> 4. Ending with h: for continuation.

>

> Inference Rules:

> - Stop string: Stop on \n\n (double newline), not single \n.

> - Repetition penalty: Do not use repetition penalty (--repeat-penalty 1.0); penalties degrade the voice.

> - Sampling: Temperature 0.7, top-p 0.9.

> - Keep turn generation short (32 to 64 tokens).

Example Prompt:

A room in the library, late. h is present and answers when spoken to, briefly, in the words of the books it has read. The others are visitors.

visitor: what do you see outside?

h:

---

Quickstart Guide

1. llama.cpp

Run generation with llama-cli:

llama-cli -m ./h-05b-replay-Q4_K_M.gguf \
    -p "A room in the library, late. h is present and answers when spoken to, briefly, in the words of the books it has read. The others are visitors.\n\nvisitor: what book are you reading?\n\nh:" \
    -r "\n\n" \
    --repeat-penalty 1.0 \
    --temp 0.7 \
    --top-p 0.9 \
    -n 64

Launch a local inference server with llama-server:

llama-server -m ./h-05b-replay-Q4_K_M.gguf \
    --port 8080 \
    -c 4096

2. Ollama

Create a Modelfile:

FROM ./h-05b-replay-Q4_K_M.gguf
PARAMETER stop "\n\n"
PARAMETER repeat_penalty 1.0
PARAMETER temperature 0.7
PARAMETER top_p 0.9

Build and run:

ollama create h-05b -f Modelfile
ollama run h-05b "visitor: what are you looking for?\n\nh:"

3. LM Studio

1. Copy the .gguf file to your LM Studio models directory.
2. Select h-05b-replay-Q4_K_M.gguf.
3. Set template to raw completion. Set stop sequence to \n\n and repeat penalty to 1.0.