infohound/ha-voice-7b-gguf overview

{
  "metadata": {},
  "card_data": {
    "license": "apache-2.0",
    "base_model": "Qwen/Qwen2.5-7B-Instruct",
    "tags": [
      "gguf",
      "voice-assistant",
      "home-assistant",
      "tool-calling",
      "fine-tuned",
      "smart-home",
      "qlora"
    ],
    "model_name": "ha-voice-7b",
    "pipeline_tag": "text-generation",
    "frontmatter": {
      "license": "apache-2.0",
      "base_model": "Qwen/Qwen2.5-7B-Instruct",
      "tags": [
        "gguf",
        "voice-assistant",
        "home-assistant",
        "tool-calling",
        "fine-tuned",
        "smart-home",
        "qlora"
      ],
      "model_name": "ha-voice-7b",
      "pipeline_tag": "text-generation"
    },
    "hero_image_url": "",
    "summary": "A purpose-built 7B language model for Home Assistant voice pipelines, fine-tuned from Qwen2.5-7B-Instruct on 8,000 HA tool-calling conversations using QLoRA.",
    "quick_links": [],
    "benchmark_table_html": "",
    "readme_markdown": "---\nlicense: apache-2.0\nbase_model: Qwen/Qwen2.5-7B-Instruct\ntags:\n  - gguf\n  - voice-assistant\n  - home-assistant\n  - tool-calling\n  - fine-tuned\n  - smart-home\n  - qlora\nmodel_name: ha-voice-7b\npipeline_tag: text-generation\n---\n\n# ha-voice-7b — Fine-Tuned Voice Assistant for Home Assistant\n\nA purpose-built 7B language model for Home Assistant voice pipelines, fine-tuned from [Qwen2.5-7B-Instruct](https://huggingface.co/Qwen/Qwen2.5-7B-Instruct) on 8,000 HA tool-calling conversations using QLoRA.\n\n## Why This Model Exists\n\nHome Assistant's voice pipeline sends 50+ tool schemas to the LLM on every voice command. Generic models struggle with this — they use wrong parameters (`name` instead of `area`), call wrong tools, or return empty responses. This model was specifically trained to:\n\n1. **Select the correct tool** from 50+ concurrent tool schemas\n2. **Use the correct parameters** — `area` for rooms, `floor` for stories, `name` for specific devices, `domain` as arrays\n3. **Produce clean tool call JSON** in OpenAI-compatible format\n4. **Confirm actions** with brief text after tool execution\n5. **Generalize** across any entity naming convention — not hardcoded to one user's setup\n\n## Benchmark Results\n\nTested through Ollama with 33 diverse voice commands and 25+ tool schemas:\n\n| Model | Size | Tool Accuracy | Param Accuracy | Empty Responses | Latency |\n|-------|------|--------------|----------------|-----------------|---------|\n| cogito:14b Q4_K_M (baseline) | 9.0 GB | 88% | 58% | 0 | 2,856ms |\n| **ha-voice:7b Q4_K_M** | **4.5 GB** | **94%** | **85%** | **0** | **341ms** |\n\n- **94% tool selection accuracy** — correct tool called nearly every time\n- **85% full parameter accuracy** — correct params with proper formatting (uses `area` not `name`)\n- **Zero empty responses** — always produces a tool call or text\n- **8.4x faster** than cogito:14b baseline\n- **Half the size** (4.5 GB vs 9.0 GB)\n\n## Available Files\n\n| File | Quant | Size | Use Case |\n|------|-------|------|----------|\n| `ha-voice-7b-Q4_K_M.gguf` | Q4_K_M | 4.5 GB | **Recommended** — best balance of size and quality |\n\n## How to Use\n\n### With Ollama\n\n```bash\n# Download the GGUF\n# Create a Modelfile (use your preferred chat template)\ncat > Modelfile << 'EOF'\nFROM ./ha-voice-7b-Q4_K_M.gguf\nPARAMETER temperature 0.1\nPARAMETER num_ctx 4096\nEOF\n\nollama create ha-voice:7b -f Modelfile\n```\n\n### With Home Assistant\n\nConfigure the `local_openai` custom integration to point to your Ollama instance with this model. The model expects tools in standard OpenAI function calling format, which `local_openai` sends automatically.\n\n**Recommended LiteLLM configuration:**\n```yaml\n- model_name: local/voice\n  litellm_params:\n    model: ollama_chat/ha-voice:7b\n    api_base: http://localhost:11434\n    stream: false\n  model_info:\n    supports_function_calling: true\n```\n\n**Direct Ollama (without LiteLLM):**\nPoint `local_openai` to `http://your-ollama-host:11434/v1` and use model name `ha-voice:7b`.\n\n> **Note:** Use `stream: false` or non-streaming mode for reliable tool call responses. Streaming tool calls may drop tool call data depending on your proxy configuration.\n\n### With llama.cpp\n\n```bash\nllama-server \\\n  -m ha-voice-7b-Q4_K_M.gguf \\\n  -ngl 99 \\\n  -c 4096 \\\n  --port 8080\n```\n\n## What It Handles\n\nThe model was trained on these HA tool types with randomized entity names:\n\n- **HassTurnOn / HassTurnOff** — lights, switches, scripts, scenes (with `area`, `floor`, `name`, `domain` parameters)\n- **HassLightSet** — brightness (0-100), color, color temperature\n- **HassVacuumStart / HassVacuumReturnToBase** — vacuum control by name or area\n- **HassListAddItem / HassListRemoveItem / HassListCompleteItem** — shopping/todo lists\n- **HassBroadcast** — whole-home announcements\n- **HassCancelAllTimers** — timer management\n- **GetDateTime** — current date/time queries\n- **Camera scripts** — parameterless tools for showing camera feeds on TVs\n- **HassGetState / HassSetPosition / HassClimateSetTemperature** — state queries and climate control\n- **HassMediaPause / HassSetVolume** — media player control\n\nTraining data included 15-80 tools per conversation, so the model handles large tool sets without degradation.\n\n## Training Details\n\n- **Base model:** Qwen2.5-7B-Instruct (7.7B parameters)\n- **Method:** QLoRA (LoRA rank 64, alpha 128, 2.08% trainable parameters)\n- **Training data:** 8,000 conversations with:\n  - Randomized entity names (English, European, technical naming styles)\n  - Varied tool counts (15-80 tools per conversation)\n  - Multi-turn sequences (user → tool_call → tool_result → confirmation)\n  - Ollama Qwen2 chat template format\n- **Epochs:** 3\n- **Hardware:** NVIDIA DGX Spark (GB10 Grace Blackwell, 128 GB unified memory)\n- **Training time:** ~30 hours total\n\n### Why Qwen2.5-7B-Instruct\n\n- Same architecture family as cogito:14b (which was proven for HA voice)\n- No thinking/reasoning tokens (critical — `<think>` tokens cause empty responses via OpenAI API)\n- Strong baseline tool-calling ability\n- 7B is the sweet spot for consumer GPU deployment (fits in 8 GB VRAM)\n\n## Key Design Decisions\n\n### Generalization Over Memorization\n\nThe training data uses **randomized entity names** from multiple pools:\n- English: \"Living Room\", \"Kitchen\", \"Master Bedroom\"\n- European: \"Salon\", \"Cuisine\", \"Wohnzimmer\"\n- Technical: \"lr\", \"kit\", \"br1\"\n- Product names: \"Hue Lamp\", \"LIFX Strip\", \"Govee Strip\"\n\nThis means the model works with **any HA setup**, not just the one it was trained on.\n\n### Parameter Accuracy\n\nThe model learned the correct HA parameter semantics:\n- `area` for rooms: `{\"area\": \"Living Room\"}`\n- `floor` for stories: `{\"floor\": \"Upstairs\"}`\n- `name` for specific devices: `{\"name\": \"Desk LED Lights\"}`\n- `domain` as arrays: `{\"domain\": [\"light\"]}`\n\nGeneric models (including cogito:14b) consistently confuse these parameters.\n\n### No Thinking Tokens\n\nThe base model (Qwen2.5-7B-Instruct) does not produce `<think>` tokens. This is critical for HA voice pipelines — models with thinking tokens (qwen3, qwen3.5) return empty `content` via the OpenAI-compatible API, causing Voice PE devices to show error states.\n\n## Known Issues\n\n- **Post-tool confirmation text:** The model generates correct confirmations (\"Done.\", \"Lights are on.\") when tested directly, but HA's `local_openai` streaming pipeline may drop them. A fallback patch in `entity.py` injects \"Done.\" when this occurs. This is an HA integration issue, not a model issue.\n- Trained primarily on English voice commands\n- Does not include cover/lock/valve control in training data (coming in future versions)\n- \"Bedroom\" may be used instead of \"Master Bedroom\" for ambiguous area names (HA resolves this correctly)\n- Best results with the full HA system prompt and tool schemas (minimal prompts may produce empty responses)\n\n## Hardware Requirements\n\n| Hardware | VRAM | Performance |\n|----------|------|-------------|\n| RTX 3060 (12 GB) | ~5 GB | Good — primary target |\n| RTX 4060/4070 (8-12 GB) | ~5 GB | Excellent |\n| RTX 4080/4090 (16+ GB) | ~5 GB | Excellent |\n| CPU only (16+ GB RAM) | N/A | Usable but slower (~2-5s per call) |\n| Raspberry Pi 5 (8 GB) | N/A | Marginal — may work with CPU inference |\n\n## Acknowledgments\n\n- [Qwen Team](https://huggingface.co/Qwen) for the excellent Qwen2.5-7B-Instruct base model\n- [Home Assistant](https://www.home-assistant.io/) for the intent definitions used in training data generation\n- [llama.cpp](https://github.com/ggml-org/llama.cpp) for GGUF conversion tools\n- [Hugging Face TRL](https://github.com/huggingface/trl) for the SFT training framework\n- Built on an [NVIDIA DGX Spark](https://www.nvidia.com/en-us/products/workstations/dgx-spark/)\n\n## License\n\nApache 2.0 (same as the base model)\n",
    "related_quantizations": []
  },
  "tags": [
    "gguf",
    "voice-assistant",
    "home-assistant",
    "tool-calling",
    "fine-tuned",
    "smart-home",
    "qlora",
    "text-generation",
    "base_model:Qwen/Qwen2.5-7B-Instruct",
    "base_model:quantized:Qwen/Qwen2.5-7B-Instruct",
    "license:apache-2.0",
    "endpoints_compatible",
    "region:us",
    "conversational"
  ],
  "likes": 0,
  "downloads": 110,
  "gated": false,
  "private": false,
  "last_modified": "2026-04-04T04:42:54.000Z",
  "created_at": "2026-04-04T04:00:31.000Z",
  "pipeline_tag": "text-generation",
  "library_name": ""
}

{
  "_id": "69d08cdffa94ee49460ab067",
  "id": "infohound/ha-voice-7b-GGUF",
  "modelId": "infohound/ha-voice-7b-GGUF",
  "sha": "63740e11993f1223376df3918214223a00740eb6",
  "createdAt": "2026-04-04T04:00:31.000Z",
  "lastModified": "2026-04-04T04:42:54.000Z",
  "author": "infohound",
  "downloads": 110,
  "likes": 0,
  "gated": false,
  "private": false,
  "pipeline_tag": "text-generation",
  "library_name": "",
  "siblings_count": 3
}