jeney/qwen3-4b-instruct-2507-gguf overview

{
  "metadata": {},
  "card_data": {
    "library_name": "transformers",
    "license": "apache-2.0",
    "license_link": "https://huggingface.co/Qwen/Qwen3-4B-Instruct-2507/blob/main/LICENSE",
    "base_model": [
      "Qwen/Qwen3-4B-Instruct-2507"
    ],
    "tags": [
      "qwen",
      "qwen3",
      "unsloth"
    ],
    "frontmatter": {
      "library_name": "transformers",
      "license": "apache-2.0",
      "license_link": "https://huggingface.co/Qwen/Qwen3-4B-Instruct-2507/blob/main/LICENSE",
      "base_model": [
        "Qwen/Qwen3-4B-Instruct-2507"
      ],
      "tags": [
        "qwen",
        "qwen3",
        "unsloth"
      ]
    },
    "hero_image_url": "https://github.com/unslothai/unsloth/raw/main/images/unsloth%20new%20logo.png",
    "summary": "",
    "quick_links": [],
    "benchmark_table_html": "",
    "readme_markdown": "---\nlibrary_name: transformers\nlicense: apache-2.0\nlicense_link: https://huggingface.co/Qwen/Qwen3-4B-Instruct-2507/blob/main/LICENSE\nbase_model:\n- Qwen/Qwen3-4B-Instruct-2507\ntags:\n- qwen\n- qwen3\n- unsloth\n---\n<div>\n  <p style=\"margin-bottom: 0; margin-top: 0;\">\n    <strong>See <a href=\"https://huggingface.co/collections/unsloth/qwen3-680edabfb790c8c34a242f95\">our collection</a> for all versions of Qwen3 including GGUF, 4-bit & 16-bit formats.</strong>\n  </p>\n  <p style=\"margin-bottom: 0;\">\n    <em>Learn to run Qwen3-2507 correctly - <a href=\"https://docs.unsloth.ai/basics/qwen3-2507\">Read our Guide</a>.</em>\n  </p>\n<p style=\"margin-top: 0;margin-bottom: 0;\">\n    <em><a href=\"https://docs.unsloth.ai/basics/unsloth-dynamic-v2.0-gguf\">Unsloth Dynamic 2.0</a> achieves superior accuracy & outperforms other leading quants.</em>\n  </p>\n  <div style=\"display: flex; gap: 5px; align-items: center; \">\n    <a href=\"https://github.com/unslothai/unsloth/\">\n      <img src=\"https://github.com/unslothai/unsloth/raw/main/images/unsloth%20new%20logo.png\" width=\"133\">\n    </a>\n    <a href=\"https://discord.gg/unsloth\">\n      <img src=\"https://github.com/unslothai/unsloth/raw/main/images/Discord%20button.png\" width=\"173\">\n    </a>\n    <a href=\"https://docs.unsloth.ai/basics/qwen3-2507\">\n      <img src=\"https://raw.githubusercontent.com/unslothai/unsloth/refs/heads/main/images/documentation%20green%20button.png\" width=\"143\">\n    </a>\n  </div>\n<h1 style=\"margin-top: 0rem;\">✨ Read our Qwen3-2507 Guide <a href=\"https://docs.unsloth.ai/basics/qwen3-2507\">here</a>!</h1>\n</div>\n\n- Fine-tune Qwen3 (14B) for free using our Google [Colab notebook here](https://docs.unsloth.ai/get-started/unsloth-notebooks)!\n- Read our Blog about Qwen3 support: [unsloth.ai/blog/qwen3](https://unsloth.ai/blog/qwen3)\n- View the rest of our notebooks in our [docs here](https://docs.unsloth.ai/get-started/unsloth-notebooks).\n- Run & export your fine-tuned model to Ollama, llama.cpp or HF.\n\n| Unsloth supports          |    Free Notebooks                                                                                           | Performance | Memory use |\n|-----------------|--------------------------------------------------------------------------------------------------------------------------|-------------|----------|\n| **Qwen3 (14B)**      | [▶️ Start on Colab](https://docs.unsloth.ai/get-started/unsloth-notebooks)               | 3x faster | 70% less |\n| **GRPO with Qwen3 (8B)**      | [▶️ Start on Colab](https://docs.unsloth.ai/get-started/unsloth-notebooks)               | 3x faster | 80% less |\n| **Llama-3.2 (3B)**      | [▶️ Start on Colab](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Llama3.2_(1B_and_3B)-Conversational.ipynb)               | 2.4x faster | 58% less |\n| **Llama-3.2 (11B vision)**      | [▶️ Start on Colab](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Llama3.2_(11B)-Vision.ipynb)               | 2x faster | 60% less |\n| **Qwen2.5 (7B)**      | [▶️ Start on Colab](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Qwen2.5_(7B)-Alpaca.ipynb)               | 2x faster | 60% less |\n\n# Qwen3-4B-Instruct-2507\n<a href=\"https://chat.qwen.ai\" target=\"_blank\" style=\"margin: 2px;\">\n    <img alt=\"Chat\" src=\"https://img.shields.io/badge/%F0%9F%92%9C%EF%B8%8F%20Qwen%20Chat%20-536af5\" style=\"display: inline-block; vertical-align: middle;\"/>\n</a>\n\n## Highlights\n\nWe introduce the updated version of the **Qwen3-4B non-thinking mode**, named **Qwen3-4B-Instruct-2507**, featuring the following key enhancements:\n\n- **Significant improvements** in general capabilities, including **instruction following, logical reasoning, text comprehension, mathematics, science, coding and tool usage**.\n- **Substantial gains** in long-tail knowledge coverage across **multiple languages**.\n- **Markedly better alignment** with user preferences in **subjective and open-ended tasks**, enabling more helpful responses and higher-quality text generation.\n- **Enhanced capabilities** in **256K long-context understanding**.\n\n![image/jpeg](https://qianwen-res.oss-accelerate.aliyuncs.com/Qwen3-2507/Qwen3-4B-Instruct.001.jpeg)\n\n## Model Overview\n\n**Qwen3-4B-Instruct-2507** has the following features:\n- Type: Causal Language Models\n- Training Stage: Pretraining & Post-training\n- Number of Parameters: 4.0B\n- Number of Paramaters (Non-Embedding): 3.6B\n- Number of Layers: 36\n- Number of Attention Heads (GQA): 32 for Q and 8 for KV\n- Context Length: **262,144 natively**. \n\n**NOTE: This model supports only non-thinking mode and does not generate ``<think></think>`` blocks in its output. Meanwhile, specifying `enable_thinking=False` is no longer required.**\n\nFor more details, including benchmark evaluation, hardware requirements, and inference performance, please refer to our [blog](https://qwenlm.github.io/blog/qwen3/), [GitHub](https://github.com/QwenLM/Qwen3), and [Documentation](https://qwen.readthedocs.io/en/latest/).\n\n\n## Performance\n\n|  | GPT-4.1-nano-2025-04-14 | Qwen3-30B-A3B Non-Thinking | Qwen3-4B Non-Thinking | Qwen3-4B-Instruct-2507 |\n|--- | --- | --- | --- | --- |\n| **Knowledge** | | | |\n| MMLU-Pro | 62.8 | 69.1 | 58.0 | **69.6** |\n| MMLU-Redux | 80.2 | 84.1 | 77.3 | **84.2** |\n| GPQA | 50.3 | 54.8 | 41.7 | **62.0** |\n| SuperGPQA | 32.2 | 42.2 | 32.0 | **42.8** |\n| **Reasoning** | | | |\n| AIME25 | 22.7 | 21.6 | 19.1 | **47.4** |\n| HMMT25 | 9.7 | 12.0 | 12.1 | **31.0** |\n| ZebraLogic | 14.8 | 33.2 | 35.2 | **80.2** |\n| LiveBench 20241125 | 41.5 | 59.4 | 48.4 | **63.0** |\n| **Coding** | | | |\n| LiveCodeBench v6 (25.02-25.05) | 31.5 | 29.0 | 26.4 | **35.1** |\n| MultiPL-E | 76.3 | 74.6 | 66.6 | **76.8** |\n| Aider-Polyglot |  9.8 | **24.4** | 13.8 | 12.9 |\n| **Alignment** | | | |\n| IFEval | 74.5 | **83.7** | 81.2 | 83.4 |\n| Arena-Hard v2* | 15.9 | 24.8 | 9.5 | **43.4** |\n| Creative Writing v3 | 72.7 | 68.1 | 53.6 | **83.5** |\n| WritingBench | 66.9 | 72.2 | 68.5 | **83.4** |\n| **Agent** | | | |\n| BFCL-v3 | 53.0 | 58.6 | 57.6 | **61.9** |\n| TAU1-Retail | 23.5 | 38.3 | 24.3 | **48.7** |\n| TAU1-Airline | 14.0 | 18.0 | 16.0 | **32.0** |\n| TAU2-Retail | - | 31.6 | 28.1 | **40.4** |\n| TAU2-Airline | - | 18.0 | 12.0 | **24.0** |\n| TAU2-Telecom | - | **18.4** | 17.5 | 13.2 |\n| **Multilingualism** | | | |\n| MultiIF | 60.7 | **70.8** | 61.3 | 69.0 |\n| MMLU-ProX | 56.2 | **65.1** | 49.6 | 61.6 |\n| INCLUDE | 58.6 | **67.8** | 53.8 | 60.1 |\n| PolyMATH | 15.6 | 23.3 | 16.6 | **31.1** |\n\n*: For reproducibility, we report the win rates evaluated by GPT-4.1.\n\n\n## Quickstart\n\nThe code of Qwen3 has been in the latest Hugging Face `transformers` and we advise you to use the latest version of `transformers`.\n\nWith `transformers<4.51.0`, you will encounter the following error:\n```\nKeyError: 'qwen3'\n```\n\nThe following contains a code snippet illustrating how to use the model generate content based on given inputs. \n```python\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\n\nmodel_name = \"Qwen/Qwen3-4B-Instruct-2507\"\n\n# load the tokenizer and the model\ntokenizer = AutoTokenizer.from_pretrained(model_name)\nmodel = AutoModelForCausalLM.from_pretrained(\n    model_name,\n    torch_dtype=\"auto\",\n    device_map=\"auto\"\n)\n\n# prepare the model input\nprompt = \"Give me a short introduction to large language model.\"\nmessages = [\n    {\"role\": \"user\", \"content\": prompt}\n]\ntext = tokenizer.apply_chat_template(\n    messages,\n    tokenize=False,\n    add_generation_prompt=True,\n)\nmodel_inputs = tokenizer([text], return_tensors=\"pt\").to(model.device)\n\n# conduct text completion\ngenerated_ids = model.generate(\n    **model_inputs,\n    max_new_tokens=16384\n)\noutput_ids = generated_ids[0][len(model_inputs.input_ids[0]):].tolist() \n\ncontent = tokenizer.decode(output_ids, skip_special_tokens=True)\n\nprint(\"content:\", content)\n```\n\nFor deployment, you can use `sglang>=0.4.6.post1` or `vllm>=0.8.5` or to create an OpenAI-compatible API endpoint:\n- SGLang:\n    ```shell\n    python -m sglang.launch_server --model-path Qwen/Qwen3-4B-Instruct-2507 --context-length 262144\n    ```\n- vLLM:\n    ```shell\n    vllm serve Qwen/Qwen3-4B-Instruct-2507 --max-model-len 262144\n    ```\n\n**Note: If you encounter out-of-memory (OOM) issues, consider reducing the context length to a shorter value, such as `32,768`.**\n\nFor local use, applications such as Ollama, LMStudio, MLX-LM, llama.cpp, and KTransformers have also supported Qwen3.\n\n## Agentic Use\n\nQwen3 excels in tool calling capabilities. We recommend using [Qwen-Agent](https://github.com/QwenLM/Qwen-Agent) to make the best use of agentic ability of Qwen3. Qwen-Agent encapsulates tool-calling templates and tool-calling parsers internally, greatly reducing coding complexity.\n\nTo define the available tools, you can use the MCP configuration file, use the integrated tool of Qwen-Agent, or integrate other tools by yourself.\n```python\nfrom qwen_agent.agents import Assistant\n\n# Define LLM\nllm_cfg = {\n    'model': 'Qwen3-4B-Instruct-2507',\n\n    # Use a custom endpoint compatible with OpenAI API:\n    'model_server': 'http://localhost:8000/v1',  # api_base\n    'api_key': 'EMPTY',\n}\n\n# Define Tools\ntools = [\n    {'mcpServers': {  # You can specify the MCP configuration file\n            'time': {\n                'command': 'uvx',\n                'args': ['mcp-server-time', '--local-timezone=Asia/Shanghai']\n            },\n            \"fetch\": {\n                \"command\": \"uvx\",\n                \"args\": [\"mcp-server-fetch\"]\n            }\n        }\n    },\n  'code_interpreter',  # Built-in tools\n]\n\n# Define Agent\nbot = Assistant(llm=llm_cfg, function_list=tools)\n\n# Streaming generation\nmessages = [{'role': 'user', 'content': 'https://qwenlm.github.io/blog/ Introduce the latest developments of Qwen'}]\nfor responses in bot.run(messages=messages):\n    pass\nprint(responses)\n```\n\n## Best Practices\n\nTo achieve optimal performance, we recommend the following settings:\n\n1. **Sampling Parameters**:\n   - We suggest using `Temperature=0.7`, `TopP=0.8`, `TopK=20`, and `MinP=0`.\n   - For supported frameworks, you can adjust the `presence_penalty` parameter between 0 and 2 to reduce endless repetitions. However, using a higher value may occasionally result in language mixing and a slight decrease in model performance.\n\n2. **Adequate Output Length**: We recommend using an output length of 16,384 tokens for most queries, which is adequate for instruct models.\n\n3. **Standardize Output Format**: We recommend using prompts to standardize model outputs when benchmarking.\n   - **Math Problems**: Include \"Please reason step by step, and put your final answer within \\boxed{}.\" in the prompt.\n   - **Multiple-Choice Questions**: Add the following JSON structure to the prompt to standardize responses: \"Please show your choice in the `answer` field with only the choice letter, e.g., `\"answer\": \"C\"`.\"\n\n### Citation\n\nIf you find our work helpful, feel free to give us a cite.\n\n```\n@misc{qwen3technicalreport,\n      title={Qwen3 Technical Report}, \n      author={Qwen Team},\n      year={2025},\n      eprint={2505.09388},\n      archivePrefix={arXiv},\n      primaryClass={cs.CL},\n      url={https://arxiv.org/abs/2505.09388}, \n}\n```",
    "related_quantizations": []
  },
  "tags": [
    "transformers",
    "gguf",
    "qwen",
    "qwen3",
    "unsloth",
    "arxiv:2505.09388",
    "base_model:Qwen/Qwen3-4B-Instruct-2507",
    "base_model:quantized:Qwen/Qwen3-4B-Instruct-2507",
    "license:apache-2.0",
    "endpoints_compatible",
    "region:us",
    "conversational"
  ],
  "likes": 0,
  "downloads": 993,
  "gated": false,
  "private": false,
  "last_modified": "2026-03-26T04:45:06.000Z",
  "created_at": "2026-03-26T04:43:16.000Z",
  "pipeline_tag": "",
  "library_name": "transformers"
}

{
  "_id": "69c4b964a0da18fff119e0cc",
  "id": "Jeney/Qwen3-4B-Instruct-2507-GGUF",
  "modelId": "Jeney/Qwen3-4B-Instruct-2507-GGUF",
  "sha": "b4fc3bb18a3ad825c65cfe774df6cd78ad4033b9",
  "createdAt": "2026-03-26T04:43:16.000Z",
  "lastModified": "2026-03-26T04:45:06.000Z",
  "author": "Jeney",
  "downloads": 993,
  "likes": 0,
  "gated": false,
  "private": false,
  "pipeline_tag": "",
  "library_name": "transformers",
  "siblings_count": 32
}