---

language:

• en
• pl

license: apache-2.0

tags:

• polish
• chatgpt
• text-generation
• conversational
• llama.cpp
• gguf
• ollama
• lm-studio
• jan-ai
• qwen2.5
• code

base_model: Qwen/Qwen2.5-7B-Instruct

pipeline_tag: text-generation

inference: false

---

<div align="center">

⚡ GPT-neo-Custom (7B) GGUF ⚡

Next-Generation Polish & English AI Assistant







<p align="center">

<b>Trained with passion by Szymon-Wachelko</b><br>

<i>Fine-tuned for deep logical reasoning, coding excellence, and native Polish eloquence.</i>

</p>

</div>

---

> [!TIP]

> 💬 Found a bug or have an idea to improve the model?

> Join the conversation in the Community / Discussions tab! All feedback, bug reports, and feature requests are actively monitored and incorporated into future releases.

---

📖 Model Overview

> GPT-neo-Custom 7B is a high-performance conversational language model designed to bridge the gap in high-quality Polish language generation while maintaining top-tier software engineering and mathematical problem-solving skills.

Built upon the state-of-the-art Qwen 2.5 7B Instruct architecture, this model underwent a multi-stage QLoRA fine-tuning process specifically balanced for conversational coherence and zero-shot reasoning.

✨ Key Strengths

• 🇵🇱 Natural Polish Fluency: Full mastery of Polish grammar, complex sentence structures, declensions, and cultural nuances without translation artifacts.
• 💻 Software Engineering: Proficient in writing, debugging, and explaining code in Python, C++, JavaScript, TypeScript, SQL, and shell scripting.
• 🧮 Structured Reasoning: Excels at chain-of-thought mathematical reasoning, logic puzzles, and technical documentation.
• 🚀 Universal Compatibility: High-performance GGUF quants tailored for local execution from low-end laptops to dedicated RTX workstations.

---

📊 Quantization Matrix

| File Name | Precision | File Size | Recommended VRAM | Recommended Setup |

| :--- | :---: | :---: | :---: | :--- |

| GPT-neo-Custom [7B] [F16].gguf | 16-bit Float | ~15.2 GB | 16 GB+ | Reference studio grade; 100% lossless precision. |

| GPT-neo-Custom [7B] [Q8_0].gguf | 8-bit | ~7.7 GB | 10–12 GB | 🏆 Gold Standard: Perfect for RTX 3060 (12GB). Identical to FP16. |

| GPT-neo-Custom [7B] [Q6_K].gguf | 6-bit | ~6.3 GB | 8–10 GB | High accuracy with blazing fast generation speeds. |

| GPT-neo-Custom [7B] [Q5_K_M].gguf | 5-bit Medium | ~5.4 GB | 8 GB | Optimal balance between intelligence and resource usage. |

| GPT-neo-Custom [7B] [Q5_K_S].gguf | 5-bit Small | ~5.1 GB | 8 GB | Compact 5-bit version for balanced performance. |

| GPT-neo-Custom [7B] [Q4_K_M].gguf | 4-bit Medium | ~4.68 GB | 6 GB | 🌟 Most Popular: Recommended for everyday laptop and CPU inference. |

| GPT-neo-Custom [7B] [Q4_K_S].gguf | 4-bit Small | ~4.3 GB | 6 GB | Lighter 4-bit variant for memory-constrained systems. |

| GPT-neo-Custom [7B] [Q3_K_L].gguf | 3-bit Large | ~3.8 GB | 4–6 GB | Preserved attention heads for high quality at lower bitrates. |

| GPT-neo-Custom [7B] [Q3_K_M].gguf | 3-bit Medium | ~3.5 GB | 4 GB | Ultra-lightweight footprint for mobile and edge devices. |

---

🚀 Quickstart Deployment Guide

🦙 1. Ollama (Instant Run)

Run directly from Hugging Face:

ollama run hf.co/Szymon-Wachelko/GPT-neo-Custom-7B-GGUF

Or configure locally with a Modelfile:

FROM ./"GPT-neo-Custom [7B] [Q8_0].gguf"
PARAMETER temperature 0.7
PARAMETER top_p 0.9
PARAMETER stop "<|im_end|>"
SYSTEM "Jesteś polskim asystentem AI GPT-neo-Custom. Zawsze odpowiadaj po polsku."

ollama create gpt-neo-custom -f Modelfile
ollama run gpt-neo-custom

---

⚡ 2. llama.cpp

Interactive Terminal Chat

llama-cli -m "GPT-neo-Custom [7B] [Q8_0].gguf" -ngl 99 -cnv --color -p "Jesteś polskim asystentem AI GPT-neo-Custom. Zawsze odpowiadaj po polsku."

Standalone ChatGPT-like Web UI

llama-server -m "GPT-neo-Custom [7B] [Q8_0].gguf" -ngl 99 --port 8080 -c 4096

> Open your browser at http://localhost:8080 for a full offline web interface.

---

🖥️ 3. LM Studio

1. Install LM Studio.
2. Download any .gguf file from the Files tab above.
3. Drag and drop the .gguf file into LM Studio.
4. Set GPU Acceleration to MAX in the right panel.
5. Click New Chat and start conversation!

---

🐍 4. Python Integration

from llama_cpp import Llama

# Load model with full GPU offloading
llm = Llama(
    model_path="GPT-neo-Custom [7B] [Q8_0].gguf",
    n_gpu_layers=-1, # -1 offloads all layers to GPU
    n_ctx=4096
)

response = llm.create_chat_completion(
    messages=[
        {"role": "system", "content": "Jesteś asystentem AI GPT-neo-Custom. Zawsze odpowiadaj po polsku."},
        {"role": "user", "content": "Cześć! W czym możesz mi dzisiaj pomóc?"}
    ]
)

print(response["choices"][0]["message"]["content"])

---

💬 Prompt Template (ChatML)

The model is trained on the standard ChatML format:

<|im_start|>system
Jesteś asystentem AI GPT-neo-Custom. Zawsze odpowiadasz w języku polskim.<|im_end|>
<|im_start|>user
Napisz skrypt w Pythonie do pobierania kursów walut z API NBP.<|im_end|>
<|im_start|>assistant

---

💡 Community, Bug Reports & Feature Requests

The GPT-neo-Custom project is developed with an open-source mindset, and community feedback is vital for future iterations!

🛠️ How to contribute feedback or report issues:

1. Navigate to the **[Community / Discussions](https://huggingface.