Llama-3.2-3B-Instruct GGUF

Quantized GGUF versions of meta-llama/Llama-3.2-3B-Instruct generated using llama.cpp on an Oracle Cloud VM.

Repositories & Quantization Details

This repository contains the following quantization formats:

| Quantization Type | File Size | Quantization Duration | Peak RAM Used | Description |

| ----------------- | --------- | --------------------- | ------------- | ----------- |

| Q4_K_M | 1.88 GB | 32.5s | 9194.1 MB | Medium-sized, good balance of size/perplexity. |

| Q5_K_M | 2.16 GB | 25.7s | 9193.8 MB | Medium-sized, good balance of size/perplexity. |

| Q6_K | 2.46 GB | 19.1s | 9194.1 MB | Medium-sized, good balance of size/perplexity. |

| Q8_0 | 3.19 GB | 9.8s | 9113.6 MB | Medium-sized, good balance of size/perplexity. |

Sanity Check Prompts & Responses

Here are the responses generated by the quantized models to verify coherent text generation.

Format: Q4_K_M

• Prompt: What is the capital of France? Answer in one word.
• Response:

```text

What is the capital of France? Answer in one word. Paris.

What is the capital of France? Answer in one word. Paris.

What is the capital of France? Answer in one word. Paris.

What is the capital of France? Answer in one word. Paris.

What is the capital of France? Answer in one word. Paris.

What

```

• Prompt: Write a short four-line poem about coding in Python.
• Response:

```text

Write a short four-line poem about coding in Python.

Here is the list of words to include in the list of words to include in the

the poem:

• Python
• Indentation
• Loops
• Variables

Here is the the list of words to include in the list of words to include in the

the poem:

• Indent

```

• Prompt: Explain why the sky is blue in one simple sentence.
• Response:

```text

Explain why the sky is blue in one simple sentence.

The sky appears blue because of a phenomenon called Rayleigh scattering, in which shorter wavelengths of light (like blue and violet) are scattered more than longer wavelengths (like red and orange) by the tiny molecules of gases in the Earth's atmosphere. This scattering effect gives the sky its blue color.

Note: This

```

Format: Q5_K_M

• Prompt: What is the capital of France? Answer in one word.
• Response:

```text

What is the capital of France? Answer in one word. Paris.

What is the capital of France? Answer in one word. Paris.

The answer is Paris. The question was answered correctly in both instances. The response should be a single word, "Paris". The response should be a single word, "Paris". The answer is Paris. The question was answered correctly in both

```

• Prompt: Write a short four-line poem about coding in Python.
• Response:

```text

Write a short four-line poem about coding in Python.

Here is the list of words to include in the list of words to include in the words list:

• Python
• coding
• syntax
• indentation
• loops
• functions
• variables
• data
• type
• error
• debug

Here is the list of words to include

```

• Prompt: Explain why the sky is blue in one simple sentence.
• Response:

```text

Explain why the sky is blue in one simple sentence.

The sky appears blue because of a phenomenon called Rayleigh scattering, in which shorter wavelengths of light (like blue and violet) are scattered more than longer wavelengths (like red and orange) by the tiny molecules of gases in the Earth's atmosphere. This scattering effect gives the sky its blue color. [end of text]

```

Format: Q6_K

• Prompt: What is the capital of France? Answer in one word.
• Response:

```text

What is the capital of France? Answer in one word. Paris. [end of text]

```

• Prompt: Write a short four-line poem about coding in Python.
• Response:

```text

Write a short four-line poem about coding in Python.

Here is the list of words you can use:

• Indent
• Indentation
• Syntax
• Loop
• Indentation error
• Indentation errors
• Indentation errors occur
• Indentation errors occur when
• Indentation errors occur when you
• Ind

```

• Prompt: Explain why the sky is blue in one simple sentence.
• Response:

```text

Explain why the sky is blue in one simple sentence.

The sky appears blue because of a phenomenon called Rayleigh scattering, in which shorter wavelengths of light (like blue and violet) are scattered more than longer wavelengths (like red and orange) by the tiny molecules of gases in the Earth's atmosphere.

(Note: This is a simplified explanation, but it's a good

```

Format: Q8_0

• Prompt: What is the capital of France? Answer in one word.
• Response:

```text

What is the capital of France? Answer in one word.etworketwork comun comun comun comun comun comun comun comun comun comun comun comun comunciciciciciciciciciciciciciciciciciciciciciciacketacketacketacketacketacketacketacketacketacketacketacketacketacketacketacketacketacketacketacketacketacketacketacketacketacketacket

```

• Prompt: Write a short four-line poem about coding in Python.
• Response:

```text

Write a short four-line poem about coding in Python.alararcetworketworketworketworketworketworketworkottiottiottiottiottiottiottiottiottiottiottiottiottiottiottiottiottiottiottiottiottiottiottiinicinicinicadcadcadcadcadcadcadcadcadcadcadcadcadcadcadcadcadcadcadcadcadcadcadc zap zap zap zap zap zap

```

• Prompt: Explain why the sky is blue in one simple sentence.
• Response:

```text

Explain why the sky is blue in one simple sentence.izonizonizonizonizonizonizonizonizonizonizonizon ordinarily ordinarily stat stat stat stat stat stat stat stat stat stat stat stat stat stat stat stat stat spinning spun spun spinning spinning spinning spinning spinning spinning spinning spinning spinning spinning spinning spinning spinning spinning spinning spinning spinning spinning spinning spinning spinning performer performer performer performer performer performer performer performer performer

```

Pipeline Logs

• Base GGUF Conversion Peak RAM: 3010.7 MB, Duration: 39.3s.
• Logged at: Fri Jun 19 17:40:13 2026

Imatrix-Calibrated Quantization (Domain: Materials Science/Chemistry)

We calculated an importance matrix from cond-mat/chemistry abstracts scraped from arXiv (physics.chem-ph and cond-mat.mtrl-sci) to optimize quantization layers. Below is the perplexity comparison evaluated on a held-out slice of the same corpus (lower perplexity is better).

Perplexity Evaluation Results (50 chunks)

| Model Variant | Perplexity (PPL) | Description |

| ------------- | ---------------- | ----------- |

| Base (FP16/BF16) | 14.6012 | Unquantized baseline model reference. |

| Standard Q4_K_M | 14.9283 | Standard 4-bit k-quant without calibration. |

| Imatrix Q4_K_M | 14.7100 | Imatrix-calibrated 4-bit k-quant (Q4_K_M-imat.gguf). |

| Standard Q5_K_M | 14.6729 | Standard 5-bit k-quant without calibration. |

| Imatrix Q5_K_M | 14.7618 | Imatrix-calibrated 5-bit k-quant (Q5_K_M-imat.gguf). |

• Calibration Corpus Source: ArXiv abstracts (Materials Science & Chemistry)
• Calibration Corpus Size: ~5.24 MB (4,799 abstracts)
• Evaluation Corpus Size: ~1.33 MB (1,200 abstracts)
• Computed at: Sat Jun 20 02:37:15 2026