---

language:

• en

tags:

• gemma-3
• gemma-3-27b
• gguf
• code-generation
• debugging
• python
• cpp
• java
• javascript
• qlora
• fine-tuned
• merged

license: gemma

base_model: google/gemma-3-27b-it

pipeline_tag: text-generation

datasets:

• bigcode/self-oss-instruct-sc2-exec-filter-50k
• Rtian/DebugBench
• sahil2801/CodeAlpaca-20k

---

Forge — Gemma 3 27B Code Assistant (Q4_K_M GGUF)

Forge is a QLoRA fine-tune of Google's Gemma 3 27B-IT base model, conditioned

for structured code generation, multi-language debugging, and algorithmic problem solving

across Python, JavaScript, Java, C++, C and SQL.

This repository contains the Q4_K_M GGUF weights for local inference on consumer

hardware. The unquantized bfloat16 merge is ~56 GB; Q4_K_M compresses it to ~16 GB,

runnable on a single 24 GB GPU or with CPU offloading on 12 GB VRAM cards.

🔗 Links & Resources

• Training Code & Custom Eval Harness: GitHub Repo Link

Related Tools

EvalSpy repo link — a CLI that audits

LLM eval pipelines for the exact bugs encountered building this model

(0% HumanEval, 9% MBPP, 0 DebugBench samples).

\```bash

pip install evalspy

evalspy check your_eval.py

\```

> Author is open to LLM engineering roles (India onsite / worldwide remote). Contact: kaustubhkubitkar@gmail.com

Here is the working demo of the model:

(Note: The generation phase in the attached video is speed up 8x. Real-time inference on the 3060 took ~4 minutes.)

this is the link to the video : https://youtu.be/3acwPjRmo74

---

Model Details

| Field | Value |

|---|---|

| Base model | google/gemma-3-27b-it |

| Fine-tune method | QLoRA (bfloat16 compute, NF4 4-bit base weights) |

| LoRA rank / alpha | r=16 / α=32 |

| LoRA target | all-linear |

| Training steps | 1 000 (≈ 0.8 epochs over 31 K samples) |

| Effective batch size | 16 (per_device=2 × grad_accum=8) |

| Learning rate | 2e-4 · cosine decay · 3% warmup |

| Sequence length | 2 048 tokens |

| Hardware | 1× H100 80 GB SXM |

| Training time | ~3 h 48 min |

| GGUF quant | Q4_K_M via llama.cpp convert_hf_to_gguf.py |

⚡ Local Inference Performance (RTX 3060 12GB)

Tested using llama.cpp with Q4_K_M quantization and partial GPU offloading.

| Metric | Performance |

|---|---|

| Hardware | Nvidia RTX 3060 (12GB VRAM) + System RAM |

| GPU Offload | 25 layers (out of full model) |

| VRAM Usage | ~11.5 GB |

| Time to First Token (TTFT) | ~2.5 seconds |

| Generation Speed | ~12 to 15 tokens/second (estimated) |

---

Training Data

Three execution-verified sources, filtered and deduplicated before training:

| Source | Samples kept | Description |

|---|---|---|

| bigcode/self-oss-instruct-sc2-exec-filter-50k | 20 000 | Execution-verified Python instruction-response pairs |

| sahil2801/CodeAlpaca-20k | 13 141 | Instruction-style multi-language coding tasks |

| Rtian/DebugBench | ~0* | Multi-language buggy→fixed code pairs |

> *DebugBench was included in the pipeline but contributed 0 training samples due to a

> field-name mismatch (solution vs fixed_code) discovered during evaluation. Training

> effectively ran on ~33 K samples from the first two sources only.

Filtering applied: minimum 80-character responses, maximum 6 000-character sequences,

removal of placeholder patterns (TODO, NotImplementedError, trailing ellipsis), Jaccard

deduplication at 0.85 threshold.

Chat format: ChatML with a fixed Forge system prompt injected at every turn.

---

Evaluation Results

Evaluated on the merged bfloat16 model (no quantization) on an H100 80 GB using

HuggingFace transformers inference.

| Benchmark | Score | Notes |

|---|---|---|

| HumanEval pass@1 | Re-run pending | Earlier figure withdrawn — see note below |

| MBPP pass@1 | 73% | 100-problem sanitized test split, execution-based |

| DebugBench | 78% (39/50) | Token-overlap proxy, directional only |

| Spot checks (format) | 91% | 5 hand-crafted keyword-match prompts |

Base model reference (Google, Gemma 3 Technical Report, Gemma 3 27B-IT,

Google's own evaluation setup): HumanEval 87.8%, MBPP 74.4%.

HumanEval — withdrawn and being re-run

An earlier version of this card reported 98.78%. While auditing the evaluation

harness I found that it concatenated the model's code with HumanEval's test block

but never called the check() function, so the assertions never executed and any

code that ran without error was counted as a pass. The number has been withdrawn.

A corrected run, comparing this model and the base Gemma 3 27B-IT on the same

harness, is in progress.

MBPP — an eval bug found and fixed

The first MBPP run scored 9% because the test assertions hardcode the expected

function name and the prompt gave no name hint, so correct solutions failed with

NameError. After injecting the expected name, MBPP pass@1 is 73%, in line with

Google's reported 74.4% for the base model, so the fine-tune changed response

style and structure without losing capability.

DebugBench — metric limitation

Scored as the share of the reference fix's first 30 words that appear in the

model's response (pass if > 50%). This is a directional signal, not an

execution-based debugging metric.

System Prompt

Forge is conditioned on this system prompt at inference time. For best results, inject it

at the start of every conversation:

You are Forge, an elite precision coding assistant with deep expertise in Python,
JavaScript, Java, C++, C and SQL.

RESPONSE STRUCTURE:
1. One-sentence summary.
2. Complete code in a fenced block with language tag.
3. Brief explanation (3-6 bullet points).
4. Edge cases — at least 2.

When writing algorithms: state time and space complexity. Show a worked example.
When debugging: root cause in one sentence. Corrected code with # FIXED: comments.
Never write TODO, placeholder code, or incomplete functions.

---

How to Run Locally

llama.cpp (recommended)

# 12 GB VRAM card — offload as many layers as fit
./llama-cli \
  -m gemma3-forge-Q4_K_M.gguf \
  -n 2048 \
  --n-gpu-layers 20 \
  -c 4096 \
  --temp 0.1 \
  --top-k 40 \
  --top-p 0.95 \
  --repeat-penalty 1.1 \
  -p "<start_of_turn>user\nImplement a binary search tree in Python<end_of_turn>\n<start_of_turn>model\n"

> Note: Use llama-cli not main — the main binary was renamed in llama.cpp

> after build b3000. If your build is older, replace llama-cli with ./main.

llama-cpp-python

from llama_cpp import Llama

llm = Llama(
    model_path="gemma3-forge-Q4_K_M.gguf",
    n_gpu_layers=20,    # increase if you have more VRAM
    n_ctx=4096,
)

system = "You are Forge, an elite precision coding assistant..."
prompt = f"<start_of_turn>user\n{system}\n\nWrite a Python function to find all prime numbers up to n using the Sieve of Eratosthenes.<end_of_turn>\n<start_of_turn>model\n"

output = llm(
    prompt,
    max_tokens=1024,
    temperature=0.1,
    top_k=40,
    top_p=0.95,
    repeat_penalty=1.1,
    stop=["<end_of_turn>"],
)
print(output["choices"][0]["text"])

VRAM requirements

| Configuration | VRAM needed |

|---|---|

| Full offload (all layers GPU) | ~17 GB |

| Partial offload (--n-gpu-layers 20) | ~8–10 GB |

| CPU-only | 32 GB RAM recommended |

---

Intended Use

• Local code generation and completion (Python, JS, Java, C++, C, SQL)
• Multi-language debugging assistance
• Algorithmic problem solving with complexity analysis
• Offline/air-gapped environments where cloud LLM APIs are unavailable

Out of Scope / Limitations

• Not intended for general conversation — Forge's system prompt conditions it heavily

toward structured code responses; open-ended chat quality is degraded.

• Vision inputs ignored — vision tower weights were pruned during export; multimodal

inputs will not work.

• Context window — trained at 2 048 tokens; performance on very long contexts (>4 K)

is untested.

• Benchmark inflation — HumanEval scores should be read alongside MBPP and HumanEval+

results (pending) for a full generalization picture.

• DebugBench metric is weak — 78% is a directional number, not a rigorous debugging

accuracy claim.

---

License

This model is derived from google/gemma-3-27b-it and is therefore subject to the

Gemma Terms of Use: https://ai.google.dev/gemma/terms

Fine-tuning code, eval harness, and the FastAPI server are released separately under MIT.

---

Citation

If you use Forge in research or projects, please cite the base model:

@misc{gemmateam2024gemma3,
  title  = {Gemma 3 Technical Report},
  author = {Gemma Team, Google DeepMind},
  year   = {2024},
  url    = {https://ai.google.dev/gemma}
}

---

license: apache-2.0