mobilint/llama-3.1-8b-instruct-gguf overview

{
  "metadata": {},
  "card_data": {
    "license": "llama3.1",
    "license_link": "https://huggingface.co/meta-llama/Llama-3.1-8B-Instruct/blob/main/LICENSE",
    "pipeline_tag": "text-generation",
    "base_model": [
      "meta-llama/Llama-3.1-8B-Instruct"
    ],
    "base_model_relation": "quantized",
    "tags": [
      "mobilint",
      "llama",
      "llama-cpp",
      "gguf",
      "npu"
    ],
    "library_name": "llama-cpp",
    "frontmatter": {
      "license": "llama3.1",
      "license_link": "https://huggingface.co/meta-llama/Llama-3.1-8B-Instruct/blob/main/LICENSE",
      "pipeline_tag": "text-generation",
      "base_model": [
        "meta-llama/Llama-3.1-8B-Instruct"
      ],
      "base_model_relation": "quantized",
      "tags": [
        "mobilint",
        "llama",
        "llama-cpp",
        "gguf",
        "npu"
      ],
      "library_name": "llama-cpp"
    },
    "hero_image_url": "https://huggingface.co/mobilint/Llama-3.1-8B-Instruct/resolve/main/mobilint_logo.png",
    "summary": "This repository provides **Llama 3.1 8B Instruct** compiled and optimized for Mobilint NPU hardware, packaged for use with llama-cli-mblt.",
    "quick_links": [],
    "benchmark_table_html": "",
    "readme_markdown": "---\nlicense: llama3.1\nlicense_link: https://huggingface.co/meta-llama/Llama-3.1-8B-Instruct/blob/main/LICENSE\npipeline_tag: text-generation\nbase_model:\n- meta-llama/Llama-3.1-8B-Instruct\nbase_model_relation: quantized\ntags:\n- mobilint\n- llama\n- llama-cpp\n- gguf\n- npu\nlibrary_name: llama-cpp\n---\n\n<p align=\"center\">\n  <a href=\"https://mobilint.com\"><img src=\"https://huggingface.co/mobilint/Llama-3.1-8B-Instruct/resolve/main/mobilint_logo.png\" width=\"400\"/></a>\n</p>\n\n# Llama 3.1 8B Instruct — GGUF + MXQ for llama-cli-mblt\n\nThis repository provides **Llama 3.1 8B Instruct** compiled and optimized for Mobilint NPU hardware, packaged for use with [`llama-cli-mblt`](https://github.com/ggml-org/llama.cpp/tree/master/examples/mobilint).\n\n## Branches\n\n| Branch | Contents | Description |\n|--------|----------|-------------|\n| **`main`** | Body model only | Standard autoregressive decoding |\n| **`eagle3`** | Body + FC + Draft models | EAGLE3 speculative decoding (~2-4x faster) |\n\n## Files\n\n### main branch\n\n| File | Size | Description |\n|------|------|-------------|\n| `llama-3.1-8b-instruct-vocab.gguf` | 7.5 MB | Tokenizer (vocab-only GGUF) |\n| `target_emb.bin` | 2.0 GB | Body embedding weights (float32) |\n| `single_Body_Llama-3.1-8B-Instruct.mxq` | 3.7 GB | Body model for NPU (W4V8 quantized) |\n| `config.json` | — | Model configuration |\n\n### eagle3 branch (adds)\n\n| File | Size | Description |\n|------|------|-------------|\n| `single_Fc_Llama-3.1-8B-Instruct.mxq` | 49 MB | FC dimension converter model |\n| `Draft_Llama-3.1-8B-Instruct.mxq` | 181 MB | EAGLE3 draft model |\n| `draft_emb.bin` | 2.0 GB | Draft embedding weights |\n| `d2t.bin` | 250 KB | Draft-to-target vocabulary mapping |\n\n## Quick Start\n\n### Install\n\n```bash\n# Build llama-cli-mblt\ncd llama.cpp\ncmake -B build -DLLAMA_MOBILINT=ON -DLLAMA_MOBILINT_RUNTIME_DIR=/path/to/qbruntime -DCMAKE_BUILD_TYPE=Release\ncmake --build build --target llama-cli-mblt -j$(nproc)\n```\n\n### Simple decoding (main branch)\n\n```bash\n# Download model files\nhuggingface-cli download mobilint/Llama-3.1-8B-Instruct-GGUF --local-dir models/llama-8b\n\n# Run\n./build/bin/llama-cli-mblt \\\n    --gguf  models/llama-8b/llama-3.1-8b-instruct-vocab.gguf \\\n    --embd  models/llama-8b/target_emb.bin \\\n    --mxq   models/llama-8b/single_Body_Llama-3.1-8B-Instruct.mxq \\\n    --core-mode global4 --chat \\\n    -p \"What is the meaning of life?\" -n 256\n```\n\n### EAGLE3 speculative decoding (eagle3 branch)\n\n```bash\n# Download with eagle3 branch\nhuggingface-cli download mobilint/Llama-3.1-8B-Instruct-GGUF --revision eagle3 --local-dir models/llama-8b-eagle3\n\n# Run with ~2-4x speedup\n./build/bin/llama-cli-mblt \\\n    --gguf  models/llama-8b-eagle3/llama-3.1-8b-instruct-vocab.gguf \\\n    --embd  models/llama-8b-eagle3/target_emb.bin \\\n    --mxq   models/llama-8b-eagle3/single_Body_Llama-3.1-8B-Instruct.mxq \\\n    --mxq-fc    models/llama-8b-eagle3/single_Fc_Llama-3.1-8B-Instruct.mxq \\\n    --mxq-draft models/llama-8b-eagle3/Draft_Llama-3.1-8B-Instruct.mxq \\\n    --embd-draft models/llama-8b-eagle3/draft_emb.bin \\\n    --d2t   models/llama-8b-eagle3/d2t.bin \\\n    --core-mode global4 --n-draft 2 --tree-depth 6 --total-tokens 23 \\\n    --chat --temp 0.0 -p \"Explain quantum computing\" -n 200\n\n# Interactive chat\n./build/bin/llama-cli-mblt \\\n    --gguf  models/llama-8b-eagle3/llama-3.1-8b-instruct-vocab.gguf \\\n    --embd  models/llama-8b-eagle3/target_emb.bin \\\n    --mxq   models/llama-8b-eagle3/single_Body_Llama-3.1-8B-Instruct.mxq \\\n    --mxq-fc    models/llama-8b-eagle3/single_Fc_Llama-3.1-8B-Instruct.mxq \\\n    --mxq-draft models/llama-8b-eagle3/Draft_Llama-3.1-8B-Instruct.mxq \\\n    --embd-draft models/llama-8b-eagle3/draft_emb.bin \\\n    --d2t   models/llama-8b-eagle3/d2t.bin \\\n    --core-mode global4 --n-draft 2 --tree-depth 6 --total-tokens 23 \\\n    -i -n 256\n```\n\n## Performance\n\nTested on Mobilint Aries NPU with global4 core mode:\n\n| Mode | Prefill | Decode | Tokens/Step |\n|------|---------|--------|-------------|\n| Simple | ~330 t/s | ~10 t/s | 1.0 |\n| EAGLE3 | ~330 t/s | ~23 t/s | ~4.2 |\n\n## About\n\nThis model is compiled and optimized for [Mobilint NPU hardware](https://mobilint.com). It is intended to be used with `llama-cli-mblt` from llama.cpp's mobilint example.\n",
    "related_quantizations": []
  },
  "tags": [
    "llama-cpp",
    "gguf",
    "mobilint-llama",
    "mobilint",
    "llama",
    "npu",
    "text-generation",
    "base_model:meta-llama/Llama-3.1-8B-Instruct",
    "base_model:quantized:meta-llama/Llama-3.1-8B-Instruct",
    "license:llama3.1",
    "endpoints_compatible",
    "region:us",
    "conversational"
  ],
  "likes": 0,
  "downloads": 752,
  "gated": false,
  "private": false,
  "last_modified": "2026-04-08T23:47:36.000Z",
  "created_at": "2026-03-25T09:39:19.000Z",
  "pipeline_tag": "text-generation",
  "library_name": "llama-cpp"
}

{
  "_id": "69c3ad47d1f08eadea24e1cd",
  "id": "mobilint/Llama-3.1-8B-Instruct-GGUF",
  "modelId": "mobilint/Llama-3.1-8B-Instruct-GGUF",
  "sha": "f671a1fc53aacb7dd687b91d3e4037486c1d210d",
  "createdAt": "2026-03-25T09:39:19.000Z",
  "lastModified": "2026-04-08T23:47:36.000Z",
  "author": "mobilint",
  "downloads": 752,
  "likes": 0,
  "gated": false,
  "private": false,
  "pipeline_tag": "text-generation",
  "library_name": "llama-cpp",
  "siblings_count": 6
}