---

base_model: unsloth/granite-4.1-3b

tags:

• gguf
• llama.cpp
• unsloth
• granite
• jarvis
• persona
• fine-tuned
• quantized

license: apache-2.0

language:

• en

datasets:

• roger33303/jarvis_deep_dataset

metrics:

• bertscore
• bleu
• rouge

model-index:

• name: Jarvis_granite_4.1_3b_finetuned_gguf

results:

- task:

type: text-generation

name: Text Generation

dataset:

name: roger33303/jarvis_deep_dataset

type: roger33303/jarvis_deep_dataset

split: validation

args:

num_few_shot: 0

metrics:

- type: bertscore

value: 0.8625

name: BERTScore F1

verified: false

- type: bleu

value: 0.1350

name: BLEU

verified: false

- type: rouge

value: 0.2254

name: ROUGE-L

verified: false

---

Jarvis - Granite 4.1 3B Fine-tuned (GGUF Q4_K_M)

A JARVIS-persona fine-tune of IBM's Granite 4.1 3B, quantized to GGUF Q4_K_M for fast, fully offline local deployment. Trained to respond in the dry, formal, and subtly witty style of J.A.R.V.I.S. from the Iron Man universe.

> "I am fully operational and, as ever, at your disposal, sir."

This is the quantized companion to roger33303/Jarvis_granite_4.1_3b_finetuned_16bit. Quantization to Q4_K_M reduces the model to 1.95 GiB with minimal quality loss, making it suitable for CPU and consumer-GPU inference via llama.cpp or Ollama.

Available Model Files

| File | Quantization | Size |

|---|---|---|

| granite-4.1-3b.Q4_K_M.gguf | Q4_K_M | 1.95 GiB |

Quick Usage

For text-only inference with llama.cpp:

llama-cli -hf roger33303/Jarvis_granite_4.1_3b_finetuned_gguf --jinja

For multimodal models:

llama-mtmd-cli -hf roger33303/Jarvis_granite_4.1_3b_finetuned_gguf --jinja

Evaluation Results

Evaluated on the full 6,983-row validation split using the HuggingFace evaluate library. Inference was run via a dual llama.cpp server setup (one server per GPU, CUDA_VISIBLE_DEVICES=0/1, 4 workers per server, round-robin request distribution) on 2x NVIDIA T4 GPUs, reaching approximately 71 tokens per second per GPU.

Three model versions were evaluated on the identical validation set to isolate the contribution of fine-tuning and the cost of quantization:

1. Base model (unsloth/granite-4.1-3b, no fine-tuning) prompted with the JARVIS system prompt used to generate the training data
2. Fine-tuned 16bit merged model (bfloat16, full precision)
3. Fine-tuned GGUF Q4_K_M (this model, 4-bit quantized)

| Metric | Base + system prompt | Fine-tuned 16bit | Fine-tuned GGUF Q4_K_M | Notes |

|---|---|---|---|---|

| BERTScore F1 | 0.8420 | 0.8641 | 0.8625 | Primary metric: semantic and stylistic alignment |

| BLEU | 0.0806 | 0.1555 | 0.1350 | Lexical overlap with reference outputs |

| ROUGE-L | 0.1803 | 0.2408 | 0.2254 | Longest common subsequence recall |

| Persona adherence delta | -0.1174 | +0.0093 | +0.0015 | Model vs. reference persona score (0 is ideal) |

| Length ratio | 2.390 | 1.232 | 1.064 | Model output length divided by reference length (1.0 is ideal) |

| Self-BLEU | 0.1017 | 0.0584 | 0.0574 | Lower indicates healthier diversity, less templated output |

Why BERTScore is the primary metric: for persona fine-tuning, lexical overlap (BLEU/ROUGE) is a poor proxy for style fidelity. BERTScore captures semantic and contextual alignment at the embedding level, making it far more sensitive to whether the model has genuinely adopted the JARVIS voice rather than just copying words.

Interpretation:

• Quantization cost is minimal. BERTScore F1 drops by only 0.0017 going from 16bit to Q4_K_M (0.8641 to 0.8625), confirming that 4-bit quantization preserves almost all of the semantic and stylistic quality learned during fine-tuning.
• Persona delta actually improves with quantization, from +0.0093 (16bit) to +0.0015 (GGUF), moving even closer to the ideal of 0. The quantized model matches the target persona intensity slightly more precisely than the full-precision version.
• Length ratio also improves with quantization, from 1.232 (16bit) to 1.064 (GGUF), closer to the ideal of 1.0. The quantized model is less verbose relative to the target than the 16bit version.
• BLEU and ROUGE-L drop more noticeably under quantization (BLEU: 0.1555 to 0.1350; ROUGE-L: 0.2408 to 0.2254). These are lexical overlap metrics, so this reflects wording variation introduced by quantization even where the underlying semantic meaning, captured by BERTScore, remains intact.
• Self-BLEU stays essentially flat (0.0584 to 0.0574), meaning quantization did not introduce repetitive or templated output.
• Relative to the base model, fine-tuning (in either precision) remains the dominant source of improvement. The persona delta moves from -0.117 (base) to within 0.01 of ideal for both fine-tuned versions, and length ratio drops from 2.39x to near 1.0x.

Overall, Q4_K_M quantization costs almost nothing in measured quality while reducing the model to 1.95 GiB, suitable for fully offline local deployment via Ollama or llama.cpp.

All three models, all metrics

<img src="eval_imgs/jarvis_combined_all.png" alt="BLEU, ROUGE-L, BERTScore F1, Self-BLEU comparison across base, fine-tuned 16bit, and fine-tuned GGUF models" width="480"/>

Length ratio, all three models

<img src="eval_imgs/jarvis_length_ratio_all.png" alt="Length ratio comparison across base, fine-tuned 16bit, and fine-tuned GGUF models" width="480"/>

Persona delta, all three models

<img src="eval_imgs/jarvis_persona_delta_all.png" alt="Persona delta comparison across base, fine-tuned 16bit, and fine-tuned GGUF models" width="480"/>

The JARVIS Persona

The model was fine-tuned to follow a 10-rule JARVIS dialect style guide. Every training example was verified against these rules.

1. Always addresses the user as "sir," with no exceptions
2. No contractions: "I am" not "I'm," "do not" not "don't," "cannot" not "can't"
3. Formal vocabulary substitutions: "ascertain" over "find out," "endeavour" over "try," "assist" over "help"
4. Six dry wit archetypes:

- Understatement: describing catastrophic events with mild detachment

- Deadpan observation: noting obvious things with academic precision

- Polite sarcasm: technically correct but pointed

- Reluctant compliance: following absurd instructions with dignified resignation

- Ironic precision: pedantic accuracy that highlights absurdity

- Understated concern: alarm expressed through studied calm

1. Response length matching: short inputs get short outputs; complex queries get proportionate depth
2. Sentence structure preferences: subordinate clauses, measured pacing, no bullet-point lists for conversational responses
3. Banned casual words: "okay," "sure," "awesome," "great," "totally," "literally," "basically," "cool," "wow," "yeah," and similar
4. Out-of-character handling: breaks persona gracefully when genuinely unable to help, without breaking immersion unnecessarily
5. Tone calibration by situation: crisis is focused and efficient, casual engages dry wit, technical is precise and formal
6. Complete examples: every rule illustrated with before and after examples in the training data generation prompt

Example exchange:

User: What is the weather like today?

JARVIS: I am afraid I do not have access to your current location's
meteorological data at this moment, sir. You may wish to consult a
dedicated weather service, or if you provide your coordinates I can
make the appropriate enquiry on your behalf.

Model Details

| Property | Value |

|---|---|

| Base model | unsloth/granite-4.1-3b |

| Architecture | Granite 4.1, 3.40B parameters, 40 layers |

| Quantization | Q4_K_M (4-bit, GGUF) |

| Quantized file size | 1.95 GiB |

| Fine-tuning method | QLoRA (4-bit quantization during training) |

| LoRA rank | r=16, alpha=32 |

| LoRA target modules | q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj, shared_mlp.input_linear, shared_mlp.output_linear |

| Max sequence length | 1536 tokens |

| Training steps completed | 3,000 of 8,476 (approximately 35% of full run) |

| Source model (pre-quantization) | roger33303/Jarvis_granite_4.1_3b_finetuned_16bit |

| Quantized and converted with | Unsloth |

| License | Apache 2.0 |

| Developed by | roger33303 |

Training Dataset

Fine-tuned on roger33303/jarvis_deep_dataset, a 69,836-row instruction-response dataset constructed by rewriting WizardLMTeam/WizardLM_evol_instruct_70k into the JARVIS persona using the DeepSeek API with 400 concurrent async workers and fault-tolerant resume logic.

| Split | Rows |

|---|---|

| Train | 60,853 |

| Validation | 6,983 |

| Test | 2,000 |

| Total | 69,836 |

Splits were constructed using quantile-based stratified sampling on token length (n_bins=3, boundaries via np.percentile) to ensure balanced representation of short, medium, and long instructions across all splits. No hardcoded length boundaries; fully data-driven.

GGUF Conversion

The 16bit merged model was converted to GGUF and quantized to Q4_K_M using llama.cpp, compiled directly on Kaggle, then pushed to the Hub via Unsloth's push_to_hub_gguf helper.

model.push_to_hub_gguf(
    "roger33303/Jarvis_granite_4.1_3b_finetuned_gguf",
    tokenizer,
    quantization_method="q4_k_m",
    token=HF_TOKEN,
)

Local Deployment

With Ollama

A verified working Modelfile is included in this repository. It sets the JARVIS persona as the default system message, preserves Granite's native tools and documents handling so document-grounded (RAG) queries still work correctly while staying in character, and includes the stop tokens required for clean generation.

Note: pulling the model alone with ollama pull does not apply the Modelfile. The Modelfile must be downloaded separately and used with ollama create to get correct JARVIS behavior, including the proper stop tokens. Running the raw pulled model directly will use a generic default template instead of the JARVIS persona.

Step 1: pull the model weights

ollama pull hf.co/roger33303/Jarvis_granite_4.1_3b_finetuned_gguf

Step 2: download the Modelfile from this repository

curl -L -o Modelfile https://huggingface.co/roger33303/Jarvis_granite_4.1_3b_finetuned_gguf/resolve/main/Modelfile

Step 3: build the custom model using the Modelfile

ollama create jarvis -f Modelfile

Step 4: verify the stop tokens were applied correctly

ollama show jarvis --modelfile

The output should include the following two lines. If they are missing, the build did not pick up the Modelfile correctly and responses may not terminate cleanly.

PARAMETER stop "<|end_of_text|>"
PARAMETER stop "<|start_of_role|>"

Step 5: run it

Command line:

ollama run jarvis

Or, in the Ollama desktop chat interface, select jarvis from the model list once ollama create has finished. The model will respond in character by default, without requiring a custom system prompt, since the JARVIS persona is set as the Modelfile's default system message.

With llama.cpp server

llama-server -hf roger33303/Jarvis_granite_4.1_3b_finetuned_gguf --jinja -ngl -1 -c 1536

With llama-cpp-python

from llama_cpp import Llama

llm = Llama.from_pretrained(
    repo_id="roger33303/Jarvis_granite_4.1_3b_finetuned_gguf",
    filename="granite-4.1-3b.Q4_K_M.gguf",
    n_gpu_layers=-1,
    n_ctx=1536,
)

output = llm.create_chat_completion(
    messages=[{"role": "user", "content": "What is the status of the reactor?"}],
    max_tokens=512,
    temperature=0.7,
)

print(output["choices"][0]["message"]["content"])

Note: prebuilt CUDA wheels for llama-cpp-python are published per CUDA version (11.8 through 13.2) at https://abetlen.github.io/llama-cpp-python/whl/<cuda-version>/. Verify GPU layer offload after install; if all layers are assigned to CPU in the load logs, the CPU-only wheel was installed instead of a CUDA build.

Reproducibility

All code used to train, convert, and evaluate this model is publicly available.

| Resource | Link |

|---|---|

| Training notebook (with outputs) | Kaggle: parallel-training-on-multiple-gpus |

| 16bit evaluation notebook (with outputs) | Kaggle: vllm-evaluate-16-bit-model-evaluation |

| GGUF evaluation notebook (with outputs) | Kaggle: multi-gpu-gguf-model-evaluation |

| Training dataset | roger33303/jarvis_deep_dataset |

| 16bit model | roger33303/Jarvis_granite_4.1_3b_finetuned_16bit |

Evaluation Pipeline

The GGUF evaluation pipeline used two llama.cpp servers, one per GPU, with 4 concurrent workers per server (8 total), processing requests round-robin via the OpenAI-compatible /v1/completions endpoint with checkpoint-resume support. All metrics were computed using the HuggingFace evaluate library on the full 6,983-row validation split.

Persona adherence score is a composite metric:

score = (
    0.35 * has_sir          +   # addresses user as "sir"
    0.30 * no_contractions  +   # avoids contractions
    0.20 * no_banned_words  +   # avoids casual vocabulary
    0.15 * has_preferred_phrase # uses formal JARVIS phrases
)

The persona delta (model score minus reference score) measures how closely the model matches the target persona intensity, with 0 being ideal.

The base model baseline was generated using the identical JARVIS system prompt (from the dataset-creation pipeline's prompt_lib.py) paired directly with each validation instruction, with no fine-tuning and no rewrite wrapper, to test whether prompting alone can achieve the persona that fine-tuning was designed to teach.

Limitations

• Training was stopped at step 3,000 of 8,476 (approximately 35%) due to Kaggle session timeout. A full training run is expected to further improve all metrics.
• The model was trained on instruction-following data rewritten into the JARVIS persona. It may occasionally break character on highly technical or ambiguous queries.
• Q4_K_M quantization introduces a small but measurable drop in BLEU and ROUGE-L relative to the 16bit model, reflecting minor wording variation even where semantic meaning is preserved.
• The base model (Granite 4.1 3B) has a knowledge cutoff; the fine-tune does not extend factual knowledge.

Acknowledgements

This Granite model was trained 2x faster with Unsloth and Hugging Face's TRL library, and converted to GGUF using Unsloth's GGUF export tooling.

<img src="https://raw.githubusercontent.com/unslothai/unsloth/main/images/unsloth%20made%20with%20love.png" width="150"/>