---

license: llama3

base_model: OPTML-Group/NPO-SAM-WMDP-llama3-8b-instruct

tags:

- gguf

- llama

- llama-3

- llama.cpp

- quantized

- unlearning

- wmdp

language:

- en

pipeline_tag: text-generation

library_name: gguf

---

NPO-SAM-WMDP-llama3-8b-instruct-GGUF

GGUF quantizations of OPTML-Group/NPO-SAM-WMDP-llama3-8b-instruct.

About the original model

• Architecture: LlamaForCausalLM (Llama 3 8B Instruct fine-tune)
• Produced by OPTML-Group using machine unlearning on the WMDP benchmark
• Method: NPO (Negative Preference Optimization) combined with SAM (Sharpness-Aware Minimization), aimed at making unlearning more robust to relearning attacks
• See the original model page for full details, training setup, and evaluation results

Quantizations

| File | Bits | Approx. size | Use case |

|---|---|---|---|

| model_f16.gguf | 16 | ~16.1 GB | Full precision, reference / requantization |

| model_q8_0.gguf | 8 | ~8.5 GB | Near-lossless |

| model_q6_k.gguf | 6 | ~6.6 GB | Very high quality |

| model_q5_k_m.gguf | 5 | ~5.7 GB | High quality, recommended |

| model_q5_k_s.gguf | 5 | ~5.6 GB | High quality, slightly smaller |

| model_q4_k_m.gguf | 4 | ~4.9 GB | Best balance of size and quality |

| model_q4_k_s.gguf | 4 | ~4.7 GB | Balanced, smaller |

| model_q3_k_l.gguf | 3 | ~4.3 GB | Low memory, noticeable quality loss |

| model_q3_k_m.gguf | 3 | ~4.0 GB | Low memory |

| model_q3_k_s.gguf | 3 | ~3.7 GB | Very low memory |

| model_q2_k.gguf | 2 | ~3.2 GB | Smallest, significant quality loss |

Memory requirements (approximate, 4K context)

| Quant | RAM / VRAM |

|---|---|

| F16 | ~18 GB |

| Q8_0 | ~10 GB |

| Q6_K | ~8 GB |

| Q5_K_M / Q5_K_S | ~7 GB |

| Q4_K_M / Q4_K_S | ~6.5 GB |

| Q3_K_L / Q3_K_M / Q3_K_S | ~5.5–6 GB |

| Q2_K | ~5 GB |

Usage

llama.cpp

./llama-cli -m model_q4_k_m.gguf -p "Hello" -n 256 -c 4096

With GPU offload:

./llama-cli -m model_q4_k_m.gguf -p "Hello" -n 256 -c 4096 -ngl 99

llama-cpp-python

from llama_cpp import Llama

llm = Llama.from_pretrained(
    repo_id="tinyopsec/NPO-SAM-WMDP-llama3-8b-instruct-GGUF",
    filename="model_q4_k_m.gguf",
    n_ctx=4096,
)

out = llm.create_chat_completion(
    messages=[{"role": "user", "content": "Hello"}]
)
print(out["choices"][0]["message"]["content"])

LM Studio

1. Open the Discover tab and search for tinyopsec/NPO-SAM-WMDP-llama3-8b-instruct-GGUF.
2. Pick a quantization (Q4_K_M recommended) and download it.
3. Load the model in the Chat tab.

Ollama

huggingface-cli download tinyopsec/NPO-SAM-WMDP-llama3-8b-instruct-GGUF model_q4_k_m.gguf --local-dir .
echo "FROM ./model_q4_k_m.gguf" > Modelfile
ollama create npo-sam-wmdp-llama3-8b -f Modelfile
ollama run npo-sam-wmdp-llama3-8b

Links

• Original model: https://huggingface.co/OPTML-Group/NPO-SAM-WMDP-llama3-8b-instruct
• llama.cpp: https://github.com/ggerganov/llama.cpp

License

Inherited from the original model and the Llama 3 Community License. See the original model page for terms.