---

license: cc-by-nc-4.0

language:

• en

library_name: gguf

base_model: startlux-models/StartLux-Decision-v1.1-4B-Instant

base_model_relation: quantized

tags:

• gguf
• llama.cpp
• decision-model
• typed-decisions
• classification

---

StartLux-Decision-v1.1-4B-Instant-GGUF

StartLux-Decision-v1.1-4B-Instant as GGUF files for llama.cpp, in Q8_0 (8-bit) and Q4_K_M (4-bit). StartLux-Decision-v1.1-4B-Instant is a 4B decision model. Given a state (text or JSON) and typed questions about it (pick an option, yes or no, or a level on a scale of 2 to 10), it returns a probability for every option of every question from one forward pass, without generating text. Requests and responses use the TypeSafe /v1/systemone format.

Formats: BF16 · GGUF · MLX 8-bit · MLX 4-bit · Code: StartLuxLabs/StartLux-Decision · Paper: arXiv 2610.17309

<p align="center"><img src="https://huggingface.co/startlux-models/StartLux-Decision-v1.1-4B-Instant-GGUF/resolve/main/figures/score_vs_size.png" width="860" alt="Decision Index 0.3 public score against model size"></p>

<p align="center"><em>Decision Index 0.3, public subset, as of 11 October 2026: the StartLux-Decision Instant models against other decision models of each size. Figure 1 of the <a href="https://arxiv.org/abs/2610.17309">technical report</a>.</em></p>

Files

| file | size | DecisionPivot |

|---|---|---|

| StartLux-Decision-v1.1-4B-Instant-Q8_0.gguf | 4.48 GB | 46.2 |

| StartLux-Decision-v1.1-4B-Instant-Q4_K_M.gguf | 2.71 GB | 43.7 |

| mmproj-StartLux-Decision-v1.1-4B-Instant-F16.gguf | 0.67 GB | |

| BF16 weights on vLLM, for comparison | | 47.5 |

The DecisionPivot column is the mean chance-corrected accuracy over the seven layers of the benchmark's scorer on its 7,800 core items, from the report's measurements: the GGUF files on llama.cpp (release b11549), the BF16 weights on vLLM. Q8_0 is the recommended file; Q4_K_M lost up to 3.8 points across the sizes.

The Q8_0 and Q4_K_M files hold the text decoder only, without the multi-token-prediction layer of the BF16 checkpoint. The vision encoder is the separate F16 file mmproj-StartLux-Decision-v1.1-4B-Instant-F16.gguf (672 MB), which lets llama-server read images (section Images). Plain chat with a GGUF file does not give these decisions, because the prompt format, the option-letter read-out and its temperatures are in the package, which takes the tokenizer, config.json and decision_config.json from the BF16 repository.

Run

Instant models read their prompt bidirectionally in the full-attention layers, and these files carry llama.cpp's non-causal attention flag for that. llama-server must then evaluate every prompt alone and in one piece. A stock llama-server does so with one slot, no context checkpoints and a micro-batch that holds the longest prompt:

hf download startlux-models/StartLux-Decision-v1.1-4B-Instant-GGUF StartLux-Decision-v1.1-4B-Instant-Q8_0.gguf --local-dir ~/models/StartLux-Decision-v1.1-4B-Instant-GGUF
hf download startlux-models/StartLux-Decision-v1.1-4B-Instant --exclude "*.safetensors" --local-dir ~/models/StartLux-Decision-v1.1-4B-Instant   # tokenizer and configs
git clone https://github.com/StartLuxLabs/StartLux-Decision && pip install -e ./StartLux-Decision
llama-server -m ~/models/StartLux-Decision-v1.1-4B-Instant-GGUF/StartLux-Decision-v1.1-4B-Instant-Q8_0.gguf -ngl 99 -c 20480 \
    --parallel 1 --ctx-checkpoints 0 -b 16384 -ub 16384 --port 8081 &
STARTLUX_LLAMACPP_BIDIR=server startlux-decide serve --model ~/models/StartLux-Decision-v1.1-4B-Instant \
    --backend llamacpp --llamacpp-url http://127.0.0.1:8081 --port 8090

-ub 16384 covers prompts of up to 16,384 tokens, and -b must be at least as large, because llama.cpp caps the micro-batch at the batch size. With the patch patches/llama.cpp/0001-server-bidirectional-prompts.patch of the repository applied to llama.cpp, any --parallel is safe; leave STARTLUX_LLAMACPP_BIDIR unset then, and the package asks for a bidirectional prompt with every read-out and checks that the server applied it (on a server that also has patch 0003 it relies on the non-causal flag of these files instead, see Images). With its default settings a stock server batches prompts and splits off their last tokens, which cost the 4B Instant model 7.7 DecisionPivot points; read causally, it loses 27 of its 48 points.

For Q4_K_M, replace Q8_0 in both places. -ngl 99 puts every layer on the GPU (CUDA or Metal); leave it out on a machine without one. Tested with llama.cpp b11549.

curl -s localhost:8090/v1/systemone -H 'Content-Type: application/json' -d '{
  "state": {"ticket": "I was charged twice for order #4411 and the app still shows it as unpaid."},
  "questions": {
    "team": {"type": "choice", "instructions": "Which team should handle this ticket?",
             "criteria": {"billing": "Payments and refunds", "shipping": "Delivery", "technical": "App and login"}},
    "urgent": {"type": "noul", "instructions": "Should this ticket be answered today?"}
  }
}'

Images

mmproj-StartLux-Decision-v1.1-4B-Instant-F16.gguf (672 MB) is the vision encoder of this model in one F16 file; the encoder is not quantized, so Q8_0 and Q4_K_M use the same file. Download it next to the language model and give it to llama-server with --mmproj. The decision engine (startlux-decide, --backend llamacpp) sends the images to llama-server, resizes them to multiples of 32 pixels (65,536 to 1,048,576 pixels, 64 to 1,024 tokens per image) and checks the image token count of every call.

hf download startlux-models/StartLux-Decision-v1.1-4B-Instant-GGUF mmproj-StartLux-Decision-v1.1-4B-Instant-F16.gguf --local-dir ~/models/StartLux-Decision-v1.1-4B-Instant-GGUF
llama-server -m ~/models/StartLux-Decision-v1.1-4B-Instant-GGUF/StartLux-Decision-v1.1-4B-Instant-Q8_0.gguf \
    --mmproj ~/models/StartLux-Decision-v1.1-4B-Instant-GGUF/mmproj-StartLux-Decision-v1.1-4B-Instant-F16.gguf \
    -ngl 99 -fa on --parallel 1 -c 20480 -b 16384 -ub 16384 --ctx-checkpoints 0 --cache-ram 0 \
    --image-min-tokens 64 --image-max-tokens 1024 --port 8081 &
STARTLUX_LLAMACPP_BIDIR=server startlux-decide serve --model ~/models/StartLux-Decision-v1.1-4B-Instant \
    --backend llamacpp --llamacpp-url http://127.0.0.1:8081 --port 8090

An Instant model reads the text and the image of a prompt together in one micro-batch, with the non-causal attention these files carry. Stock llama.cpp (b11549) decodes the text before an image, the image and the text after it as three separate micro-batches, which cuts that attention; images still work, but on 32 image questions the 4B Instant Q8_0 read-outs then differ from BF16 by 0.022 on average, against 0.0055 with a server that has patch patches/llama.cpp/0003-server-mtmd-prompt-whole-on-non-causal-context.patch of the repository, which reads the prompt whole. -ub must be at least the longest prompt including its image tokens: a longer prompt is refused, never cut. With patch 0003 the package recognises the server from its /props and STARTLUX_LLAMACPP_BIDIR can stay unset.

License

This is a quantized version of StartLux-Decision-v1.1-4B-Instant. The weights are released under CC BY-NC 4.0, as the BF16 weights are: free for research and other non-commercial use, with attribution. Commercial use requires a separate license from StartLux Labs (contact@startlux.com). The inference package startlux-decide is Apache-2.0. See LICENSE and NOTICE.