---

language:

• en

license: apache-2.0

library_name: llama-cpp

pipeline_tag: text-generation

base_model: Qwen/Qwen3.5-0.8B

datasets:

• Ionio-ai/ecommerce-search-extraction

tags:

• gguf
• llama-cpp
• ecommerce
• information-extraction
• structured-output
• lora
• trl

---

Qwen3.5-0.8B Ecommerce Extraction GGUF

Five GGUF quantizations of a LoRA-rank-32 fine-tune for schema-conditioned e-commerce query extraction. Training used two epochs, a cosine learning-rate schedule, and assistant-only loss. No F16 GGUF is shipped because llama.cpp can dequantize dynamically at inference time.

The source model was converted with --no-mtp; these are text-only extraction exports.

Required prompt format

Inference must use the same chat structure and prompt contract used for training and evaluation. Reasoning was disabled; temperature was 0, top_p was 1, and the output allowance was 4096 tokens.

System message:

You extract structured filters from e-commerce search queries.
Return only one valid JSON object, with no markdown or explanation.
Your output must validate against the supplied JSON Schema: include every required key, preserve nesting, do not add keys, and keep arrays as arrays.
Fill values only when stated or clearly implied by the query. Use JSON null for a required scalar field whose value is not available in the query.
Preserve the exact spelling and capitalization of every JSON key.

User message:

E-commerce query:
{query}

JSON Schema:
{compact_json_schema}

The schema is value-free: it contains keys and allowed JSON types, but never target values. Every object key is required, additional properties are forbidden, and scalar types permit JSON null. If the query lacks a required scalar value, emit JSON null (not Python None, a missing key, or the string "null"). The dataset's meta_prompt column is not used.

Apply the base model's chat template; do not concatenate raw text outside that template. For Qwen3.5, disable thinking/reasoning.

Full held-out GGUF evaluation

All rows use the same 1,095-example held-out split, exact prompt construction, llama.cpp CUDA backend, 64 concurrent requests, temperature 0, and 4096 maximum generated tokens. Quant runs were concurrent, so wall-clock latency is workload-dependent and should not be interpreted as an isolated speed benchmark.

| Quant | Size MiB | Strict JSON | Schema valid | Exact | Leaf F1 | Key F1 | Null accuracy | Truncated |

|---|---:|---:|---:|---:|---:|---:|---:|---:|

| Q8_0 | 774.2 | 99.91% | 99.73% | 30.96% | 88.49% | 99.46% | 99.82% | 1 |

| Q6_K | 600.6 | 100.00% | 99.91% | 29.95% | 88.56% | 99.57% | 99.82% | 0 |

| Q5_K_M | 551.2 | 99.91% | 99.73% | 30.32% | 88.35% | 99.38% | 99.73% | 0 |

| Q4_K_M | 504.8 | 99.82% | 99.45% | 29.41% | 88.04% | 99.33% | 99.63% | 1 |

| Q3_K_M | 444.6 | 95.53% | 94.16% | 23.93% | 82.17% | 94.62% | 95.05% | 17 |

| merged BF16 reference | — | 99.91% | 99.73% | 30.32% | 88.52% | — | — | — |

Q3_K_M prioritizes size and may materially damage instruction/schema adherence. Select using schema validity and Leaf F1, not file size alone.

Metric definitions

• Strict JSON: the complete response parses as exactly one JSON value without code fences, commentary, or repair.
• Schema valid: the parsed object satisfies the per-example Draft 2020-12 schema: required keys, types, nesting, arrays, and no extra keys.
• Exact: case-sensitive equality to the full gold JSON; array order is significant.
• Leaf F1: macro average of per-example precision/recall F1 over flattened (JSON path, typed value) pairs. This gives partial credit while still checking keys, types, and case-sensitive values.
• Key F1: macro F1 over flattened JSON paths, ignoring values; it measures structural adherence.
• Null accuracy: macro per-example accuracy on gold-null paths; non-null examples contribute 100% by definition, so interpret it with the other metrics.
• Truncated: responses that hit the 4096-token limit. No repaired or “recoverable” JSON is counted as strict JSON.

Machine-readable results are in evaluation-results.json. The accompanying PDF includes methodology and analysis.

End-to-end example

User query:

men's Nike running shoes in red under $100

JSON Schema supplied in the user message:

{"type":"object","properties":{"product_type":{"type":["string","null"]},"brand":{"type":["string","null"]},"color":{"type":["string","null"]},"price_max":{"type":["number","null"]}},"required":["product_type","brand","color","price_max"],"additionalProperties":false}

Expected assistant output:

{"product_type":"running shoes","brand":"Nike","color":"red","price_max":100}

llama.cpp example

llama-cli -m qwen3.5-0.8b-ecommerce-Q4_K_M.gguf -ngl 99 --temp 0 --top-p 1 --reasoning-budget 0 -n 4096 -cnv

Then send the system and user messages above through chat mode. For production, validate every response against the supplied schema and retry or reject failures.

Limitations

This model extracts only fields represented by the supplied schema and reflects the source dataset's annotation quality. Exact match is deliberately harsh: capitalization or a single value error fails the entire example. Results apply to this held-out dataset and prompt; other schemas, languages, llama.cpp versions, chat templates, or sampling settings may behave differently. Do not treat inferred attributes as verified product facts.