manojb/qwen3-4b-toolcall-gguf-llamacpp-codex overview

{
  "metadata": {},
  "card_data": {
    "license": "mit",
    "base_model": "Qwen/Qwen3-4B-Instruct-2507",
    "datasets": [
      "Salesforce/xlam-function-calling-60k"
    ],
    "language": [
      "en"
    ],
    "pipeline_tag": "text-generation",
    "quantized_by": "Manojb",
    "tags": [
      "function-calling",
      "tool-calling",
      "codex",
      "local-llm",
      "gguf",
      "4gb-vram",
      "llama-cpp",
      "code-assistant",
      "api-tools",
      "openai-alternative",
      "qwen3",
      "qwen",
      "instruct"
    ],
    "frontmatter": {
      "license": "mit",
      "base_model": "Qwen/Qwen3-4B-Instruct-2507",
      "datasets": [
        "Salesforce/xlam-function-calling-60k"
      ],
      "language": [
        "en"
      ],
      "pipeline_tag": "text-generation",
      "quantized_by": "Manojb",
      "tags": [
        "function-calling",
        "tool-calling",
        "codex",
        "local-llm",
        "gguf",
        "4gb-vram",
        "llama-cpp",
        "code-assistant",
        "api-tools",
        "openai-alternative",
        "qwen3",
        "qwen",
        "instruct"
      ]
    },
    "hero_image_url": "",
    "summary": "A specialized 4B parameter model fine-tuned for function calling and tool usage, optimized for local deployment with llama-cpp-python.",
    "quick_links": [],
    "benchmark_table_html": "",
    "readme_markdown": "---\nlicense: mit\nbase_model: Qwen/Qwen3-4B-Instruct-2507\ndatasets:\n- Salesforce/xlam-function-calling-60k\nlanguage:\n- en\npipeline_tag: text-generation\nquantized_by: Manojb\ntags:\n- function-calling\n- tool-calling\n- codex\n- local-llm\n- gguf\n- 4gb-vram\n- llama-cpp\n- code-assistant\n- api-tools\n- openai-alternative\n- qwen3\n- qwen\n- instruct\n---\n\n# Qwen3-4B Tool Calling with llama-cpp-python\n\nA specialized 4B parameter model fine-tuned for function calling and tool usage, optimized for local deployment with llama-cpp-python.\n\n## Features\n\n- **4B Parameters** - Sweet spot for local deployment\n- **Function Calling** - Fine-tuned on 60K function calling examples\n- **GGUF Format** - Optimized for CPU/GPU inference\n- **3.99GB Download** - Fits on any modern system\n- **262K Context** - Large context window for complex tasks\n- **VRAM** - Full context within 6GB!\n  \n## Model Details\n\n- **Base Model**: Qwen3-4B-Instruct-2507\n- **Fine-tuning**: LoRA on Salesforce xlam-function-calling-60k dataset\n- **Quantization**: Q8_0 (8-bit) for optimal performance/size ratio\n- **Architecture**: Qwen3 with specialized tool calling tokens\n- **License**: Apache 2.0\n\n## Installation\n\n### Quick Install\n\n```bash\n# Clone the repository\ngit clone https://huggingface.co/Manojb/qwen3-4b-toolcall-gguf-llamacpp-codex\ncd qwen3-4b-toolcall-llamacpp-codex\n\n# Run the installation script\n./install.sh\n```\n\n### Manual Installation\n\n#### Prerequisites\n\n- Python 3.8+\n- 6GB+ RAM (8GB+ recommended)\n- 5GB+ free disk space\n\n#### Install Dependencies\n\n```bash\npip install -r requirements.txt\n```\n\n#### Download Model\n\n```bash\n# Download the model file\nhuggingface-cli download Manojb/qwen3-4b-toolcall-gguf-llamacpp-codex Qwen3-4B-Function-Calling-Pro.gguf\n```\n\n### Alternative: Install with specific llama-cpp-python build\n\nFor better performance, you can install llama-cpp-python with specific optimizations:\n\n```bash\n# For CPU-only (default)\npip install llama-cpp-python\n\n# For CUDA support (if you have NVIDIA GPU)\nCMAKE_ARGS=\"-DLLAMA_CUBLAS=on\" pip install llama-cpp-python\n\n# For OpenBLAS support\nCMAKE_ARGS=\"-DLLAMA_BLAS=on -DLLAMA_BLAS_VENDOR=OpenBLAS\" pip install llama-cpp-python\n```\n\n## Quick Start\n\n### Option 1: Using the Run Script\n\n```bash\n# Interactive mode (default)\n./run_model.sh\n# or\nsource ./run_model.sh\n\n# Start Codex server\n./run_model.sh server\n# or\nsource ./run_model.sh server\n\n# Show help\n./run_model.sh help\n# or\nsource ./run_model.sh help\n```\n\n### Option 2: Direct Python Usage\n\n```python\nfrom llama_cpp import Llama\n\n# Load the model\nllm = Llama(\n    model_path=\"Qwen3-4B-Function-Calling-Pro.gguf\",\n    n_ctx=2048,\n    n_threads=8,\n    temperature=0.7\n)\n\n# Simple chat\nresponse = llm(\"What's the weather like in London?\", max_tokens=200)\nprint(response['choices'][0]['text'])\n```\n\n### Option 3: Quick Start Demo\n\n```bash\npython3 quick_start.py\n```\n\n### Tool Calling Example\n\n```python\nimport json\nimport re\nfrom llama_cpp import Llama\n\ndef extract_tool_calls(text):\n    \"\"\"Extract tool calls from model response\"\"\"\n    tool_calls = []\n    json_pattern = r'\\[.*?\\]'\n    matches = re.findall(json_pattern, text)\n    \n    for match in matches:\n        try:\n            parsed = json.loads(match)\n            if isinstance(parsed, list):\n                for item in parsed:\n                    if isinstance(item, dict) and 'name' in item:\n                        tool_calls.append(item)\n        except json.JSONDecodeError:\n            continue\n    return tool_calls\n\n# Initialize model\nllm = Llama(\n    model_path=\"Qwen3-4B-Function-Calling-Pro.gguf\",\n    n_ctx=2048,\n    temperature=0.7\n)\n\n# Chat with tool calling\nprompt = \"Get the weather for New York\"\nformatted_prompt = f\"<|im_start|>user\\n{prompt}<|im_end|>\\n<|im_start|>assistant\\n\"\n\nresponse = llm(formatted_prompt, max_tokens=200, stop=[\"<|im_end|>\", \"<|im_start|>\"])\nresponse_text = response['choices'][0]['text']\n\n# Extract tool calls\ntool_calls = extract_tool_calls(response_text)\nprint(f\"Tool calls: {tool_calls}\")\n```\n\n## Examples\n\n### 1. Weather Tool Calling\n\n```python\n# The model will generate:\n# [{\"name\": \"get_weather\", \"arguments\": {\"q\": \"London\"}}]\n```\n\n### 2. Hotel Search\n\n```python\n# The model will generate:\n# [{\"name\": \"search_stays\", \"arguments\": {\"check_in\": \"2023-04-01\", \"check_out\": \"2023-04-08\", \"city\": \"Paris\"}}]\n```\n\n### 3. Flight Booking\n\n```python\n# The model will generate:\n# [{\"name\": \"flights_search\", \"arguments\": {\"q\": \"New York to Tokyo\"}}]\n```\n\n### 4. News Search\n\n```python\n# The model will generate:\n# [{\"name\": \"search_news\", \"arguments\": {\"q\": \"AI\", \"gl\": \"us\"}}]\n```\n\n## Codex Integration\n\n### Setting up Codex Server\n\nTo use this model with Codex, you need to run a local server that Codex can connect to:\n\n#### 1. Install llama-cpp-python with server support\n\n```bash\npip install llama-cpp-python[server]\n```\n\n#### 2. Start the Codex-compatible server\n\n```bash\npython -m llama_cpp.server \\\n    --model Qwen3-4B-Function-Calling-Pro.gguf \\\n    --host 0.0.0.0 \\\n    --port 8000 \\\n    --n_ctx 2048 \\\n    --n_threads 8 \\\n    --temperature 0.7\n```\n\n#### 3. Configure Codex to use the local server\n\nIn your Codex configuration, set:\n- **Server URL**: `http://localhost:8000`\n- **API Key**: (not required for local server)\n- **Model**: `Qwen3-4B-Function-Calling-Pro`\n\n### Codex Integration Example\n\n```python\n# codex_integration.py\nimport requests\nimport json\n\nclass CodexClient:\n    def __init__(self, base_url=\"http://localhost:8000\"):\n        self.base_url = base_url\n        self.session = requests.Session()\n    \n    def chat_completion(self, messages, tools=None, temperature=0.7):\n        \"\"\"Send chat completion request to Codex\"\"\"\n        payload = {\n            \"model\": \"Qwen3-4B-Function-Calling-Pro\",\n            \"messages\": messages,\n            \"temperature\": temperature,\n            \"max_tokens\": 512,\n            \"stop\": [\"<|im_end|>\", \"<|im_start|>\"]\n        }\n        \n        if tools:\n            payload[\"tools\"] = tools\n        \n        response = self.session.post(\n            f\"{self.base_url}/v1/chat/completions\",\n            json=payload,\n            headers={\"Content-Type\": \"application/json\"}\n        )\n        \n        return response.json()\n    \n    def extract_tool_calls(self, response):\n        \"\"\"Extract tool calls from Codex response\"\"\"\n        tool_calls = []\n        if \"choices\" in response and len(response[\"choices\"]) > 0:\n            message = response[\"choices\"][0][\"message\"]\n            if \"tool_calls\" in message:\n                tool_calls = message[\"tool_calls\"]\n        return tool_calls\n\n# Usage with Codex\ncodex = CodexClient()\n\n# Define tools for Codex\ntools = [\n    {\n        \"type\": \"function\",\n        \"function\": {\n            \"name\": \"get_weather\",\n            \"description\": \"Get current weather for a location\",\n            \"parameters\": {\n                \"type\": \"object\",\n                \"properties\": {\n                    \"location\": {\n                        \"type\": \"string\",\n                        \"description\": \"City name\"\n                    }\n                },\n                \"required\": [\"location\"]\n            }\n        }\n    }\n]\n\n# Send request\nmessages = [{\"role\": \"user\", \"content\": \"What's the weather in London?\"}]\nresponse = codex.chat_completion(messages, tools=tools)\ntool_calls = codex.extract_tool_calls(response)\n\nprint(f\"Response: {response}\")\nprint(f\"Tool calls: {tool_calls}\")\n```\n\n### Docker Setup for Codex\n\nCreate a `Dockerfile` for easy deployment:\n\n```dockerfile\nFROM python:3.11-slim\n\nWORKDIR /app\n\n# Install dependencies\nCOPY requirements.txt .\nRUN pip install -r requirements.txt\n\n# Install llama-cpp-python with server support\nRUN pip install llama-cpp-python[server]\n\n# Copy model and scripts\nCOPY . .\n\n# Expose port\nEXPOSE 8000\n\n# Start server\nCMD [\"python\", \"-m\", \"llama_cpp.server\", \\\n     \"--model\", \"Qwen3-4B-Function-Calling-Pro.gguf\", \\\n     \"--host\", \"0.0.0.0\", \\\n     \"--port\", \"8000\", \\\n     \"--n_ctx\", \"2048\"]\n```\n\nBuild and run:\n```bash\ndocker build -t qwen3-codex-server .\ndocker run -p 8000:8000 qwen3-codex-server\n```\n\n## Advanced Usage\n\n### Custom Tool Calling Class\n\n```python\nclass Qwen3ToolCalling:\n    def __init__(self, model_path):\n        self.llm = Llama(\n            model_path=model_path,\n            n_ctx=2048,\n            n_threads=8,\n            temperature=0.7,\n            verbose=False\n        )\n    \n    def chat(self, message, system_message=None):\n        # Build prompt with proper formatting\n        prompt_parts = []\n        if system_message:\n            prompt_parts.append(f\"<|im_start|>system\\n{system_message}<|im_end|>\")\n        prompt_parts.append(f\"<|im_start|>user\\n{message}<|im_end|>\")\n        prompt_parts.append(\"<|im_start|>assistant\\n\")\n        \n        formatted_prompt = \"\\n\".join(prompt_parts)\n        \n        # Generate response\n        response = self.llm(\n            formatted_prompt,\n            max_tokens=512,\n            stop=[\"<|im_end|>\", \"<|im_start|>\"],\n            temperature=0.7\n        )\n        \n        response_text = response['choices'][0]['text']\n        tool_calls = self.extract_tool_calls(response_text)\n        \n        return {\n            'response': response_text,\n            'tool_calls': tool_calls\n        }\n```\n\n## Performance\n\n### System Requirements\n\n| Component | Minimum | Recommended |\n|-----------|---------|-------------|\n| RAM | 6GB | 8GB+ |\n| Storage | 5GB | 10GB+ |\n| CPU | 4 cores | 8+ cores |\n| GPU | Optional | NVIDIA RTX 3060+ |\n\n### Benchmarks\n\n- **Inference Speed**: ~75-100 tokens/second (CPU)\n- **Memory Usage**: ~4GB RAM\n- **Model Size**: 3.99GB (Q8_0 quantized)\n- **Context Length**: 262K tokens\n- **Function Call Accuracy**: 94%+ on test set\n\n## Use Cases\n\n- **AI Agents** - Building intelligent agents that can use tools\n- **Local Coding Assistants** - Function calling without cloud dependencies\n- **API Integration** - Seamless tool orchestration\n- **Privacy-Sensitive Development** - 100% local processing\n- **Learning Function Calling** - Educational purposes\n\n## Model Architecture\n\n### Special Tokens\n\nThe model includes specialized tokens for tool calling:\n\n- `<tool_call>` - Start of tool call\n- `</tool_call>` - End of tool call\n- `<tool_response>` - Start of tool response\n- `</tool_response>` - End of tool response\n\n### Chat Template\n\nThe model uses a custom chat template optimized for tool calling:\n\n```\n<|im_start|>system\n{system_message}<|im_end|>\n<|im_start|>user\n{user_message}<|im_end|>\n<|im_start|>assistant\n{assistant_response}<|im_end|>\n```\n\n## Repository Structure\n\n```\nqwen3-4b-toolcall-llamacpp/\n├── Qwen3-4B-Function-Calling-Pro.gguf    # Main model file\n├── qwen3_toolcalling_example.py          # Complete example\n├── quick_start.py                        # Quick start demo\n├── codex_integration.py                  # Codex integration example\n├── run_model.sh                          # Run script for llama-cpp\n├── install.sh                            # Installation script\n├── requirements.txt                      # Python dependencies\n├── README.md                             # This file\n├── config.json                           # Model configuration\n├── tokenizer_config.json                 # Tokenizer configuration\n├── special_tokens_map.json               # Special tokens mapping\n├── added_tokens.json                     # Added tokens\n├── chat_template.jinja                   # Chat template\n├── Dockerfile                            # Docker configuration\n├── docker-compose.yml                    # Docker Compose setup\n└── .gitignore                            # Git ignore file\n```\n\n\n```bibtex\n@model{Manojb/Qwen3-4b-toolcall-gguf-llamacpp-codex,\n  title={Qwen3-4B-toolcalling-gguf-codex: Local Function Calling},\n  author={Manojb},\n  year={2025},\n  url={https://huggingface.co/Manojb/Qwen3-4b-toolcall-gguf-llamacpp-codex}\n}\n```\n\n## License\n\nThis project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.\n\n## Related Projects\n\n- [llama-cpp-python](https://github.com/abetlen/llama-cpp-python) - Python bindings for llama.cpp\n- [Qwen3](https://huggingface.co/Qwen/Qwen3-4B-Instruct-2507) - Base model\n- [xlam-function-calling-60k](https://huggingface.co/datasets/Salesforce/xlam-function-calling-60k) - Training dataset\n\n---\n\n**Built with ❤️ for the developer community**",
    "related_quantizations": []
  },
  "tags": [
    "safetensors",
    "gguf",
    "qwen3",
    "function-calling",
    "tool-calling",
    "codex",
    "local-llm",
    "4gb-vram",
    "llama-cpp",
    "code-assistant",
    "api-tools",
    "openai-alternative",
    "qwen",
    "instruct",
    "text-generation",
    "conversational",
    "custom_code",
    "en",
    "dataset:Salesforce/xlam-function-calling-60k",
    "base_model:Qwen/Qwen3-4B-Instruct-2507",
    "base_model:quantized:Qwen/Qwen3-4B-Instruct-2507",
    "license:mit",
    "endpoints_compatible",
    "8-bit",
    "region:us"
  ],
  "likes": 6,
  "downloads": 801,
  "gated": false,
  "private": false,
  "last_modified": "2025-09-25T23:40:47.000Z",
  "created_at": "2025-09-21T23:56:19.000Z",
  "pipeline_tag": "text-generation",
  "library_name": ""
}

{
  "_id": "68d090a304335a4bdd69b7d2",
  "id": "Manojb/Qwen3-4b-toolcall-gguf-llamacpp-codex",
  "modelId": "Manojb/Qwen3-4b-toolcall-gguf-llamacpp-codex",
  "sha": "abbca8f452c1289341b19ffbeea8da47f166adb4",
  "createdAt": "2025-09-21T23:56:19.000Z",
  "lastModified": "2025-09-25T23:40:47.000Z",
  "author": "Manojb",
  "downloads": 801,
  "likes": 6,
  "gated": false,
  "private": false,
  "pipeline_tag": "text-generation",
  "library_name": "",
  "siblings_count": 26
}