---

base_model: rekstar/omega-attention-0.5b

library_name: peft

pipeline_tag: text-generation

tags:

• lora
• transformers
• unsloth

license: mit

language:

• en

---

GGUF Files for omega-attention-0.5b

These are the GGUF files for rekstar/omega-attention-0.5b.

Downloads

| GGUF Link | Quantization | Description |

| ---- | ----- | ----------- |

| Download | Q2_K | Lowest quality |

| Download | Q3_K_S | |

| Download | IQ3_S | Integer quant, preferable over Q3_K_S |

| Download | IQ3_M | Integer quant |

| Download | Q3_K_M | |

| Download | Q3_K_L | |

| Download | IQ4_XS | Integer quant |

| Download | Q4_K_S | Fast with good performance |

| Download | Q4_K_M | Recommended: Perfect mix of speed and performance |

| Download | Q5_K_S | |

| Download | Q5_K_M | |

| Download | Q6_K | Very good quality |

| Download | Q8_0 | Best quality |

| Download | f16 | Full precision, don't bother; use a quant |

Note from Flexan

I provide GGUFs and quantizations of publicly available models that do not have a GGUF equivalent available yet,

usually for models I deem interesting and wish to try out.

If there are some quants missing that you'd like me to add, you may request one in the community tab.

If you want to request a public model to be converted, you can also request that in the community tab.

If you have questions regarding this model, please refer to the original model repo.

You can find more info about me and what I do here.

Model Card for Model ID

<!-- Provide a quick summary of what the model is/does. -->

Omega Attention chooses between five attention operations:

ATTEND — bring something into active attention

MAINTAIN — continue attending to the current focus

SWITCH — move attention to a more important candidate

SUPPRESS — actively suppress a distracting or conflicting candidate

IGNORE — take no attentional action

The output contract is deliberately narrow.

{"operation":"SWITCH","target":"observation_4","confidence":0.91,"reason_code":"HIGHER_PRIORITY_INTERRUPT"}

The model is not intended to chat, explain its reasoning, or generate general-purpose text.

Its job is simply:

Given the current cognitive state, what should receive attention?

Model Details

Property Value

Base model unsloth/Qwen2.5-0.5B-Instruct-bnb-4bit

Base parameters ~502M

Fine-tuning LoRA

LoRA rank 16

LoRA alpha 16

Trainable parameters ~8.8M

Trainable percentage ~1.75%

Training epochs 3

Training examples 5,000

Evaluation examples 500

Training platform AMD Radeon 8060S / ROCm

Training framework Unsloth

Output Compact JSON

Model Description

<!-- Provide a longer summary of what this model is. -->

Omega Attention is a small language model fine-tuned specifically for attention management inside an artificial cognitive architecture.

Rather than using a general-purpose LLM to repeatedly decide what an agent should focus on, Omega Attention is designed as a lightweight specialist cognitive node.

Given a cognitive workspace containing competing goals, memories, observations, system events, and tool activity, the model selects an attention operation and target.

It is part of the broader Omega artificial cognitive architecture experiment.

• Developed by: [Derek Robertson]
• Model type: [Attention Processor]
• Language(s) (NLP): [English]
• License: [MIT]
• Finetuned from model [optional]: [unsloth/Qwen2.5-0.5B-Instruct-bnb-4bit]

Uses

<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->

Intended use

Omega Attention is intended for experimentation with:

artificial cognitive architectures

autonomous agents

cognitive workspaces

attention allocation

persistent agent loops

multi-model agent systems

specialised neural cognitive modules

It is primarily intended to operate as a component inside another system, rather than as a standalone conversational model.

Not intended for

Omega Attention is not intended to be:

a general-purpose assistant

a reasoning model

a chatbot

a factual knowledge model

a safety classifier

a replacement for human decision-making

It has been trained for a deliberately narrow experimental task.

Downstream Use [optional]

<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->

About Omega

Omega is an experimental artificial cognitive architecture exploring the idea that an artificial mind does not need to be synonymous with a single large language model.

Instead, language models can act as specialised neural components within a larger cognitive system incorporating mechanisms such as attention, working memory, long-term memory, goal management, reasoning, planning, and action.

Omega Attention is the first specialised cognitive model produced as part of that experiment.

STATUS

Experimental — v0.1

This is the first public iteration of Omega Attention.

The major question addressed by v0.1 was:

Can a very small language model learn the attention-selection policy required by Omega?

The initial result is encouraging.

The next question is more important:

Does that learned policy generalise beyond the synthetic distribution on which it was trained?

Future work will focus on harder evaluation rather than immediately increasing model size.

Recommendations

<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->

Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.

How to Get Started with the Model

Use the code below to get started with the model.

[More Information Needed]

Training Details

Training Data

<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->

[More Information Needed]

Training Procedure

<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->

Training Hyperparameters

• Training regime: [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->

Evaluation

<!-- This section describes the evaluation protocols and provides the results. -->

Testing Data, Factors & Metrics

Testing Data

<!-- This should link to a Dataset Card if possible. -->

[More Information Needed]

Factors

<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->

[More Information Needed]

Metrics

<!-- These are the evaluation metrics being used, ideally with a description of why. -->

Evaluation was performed against 500 held-out synthetic scenarios.

Metric 200 steps 1,875 steps / 3 epochs

Valid JSON 100% 100%

Operation accuracy 74.2% 96.0%

Target accuracy 45.2% 92.2%

Results

The architecture, dataset, LoRA configuration, and model size were unchanged between these runs.

The primary difference was training duration.

Target accuracy increased from 45.2% to 92.2% without increasing model size or inference cost.

Summary

These results should not be interpreted as demonstrating 96% accuracy on arbitrary real-world attention-management tasks.

The evaluation examples are held out from training, but they are generated by the same synthetic scenario-generation system used to produce the training dataset.

The results therefore demonstrate that a 0.5B model can successfully learn the attention policy represented by this dataset.

Generalisation to different distributions, adversarial scenarios, and real Omega cognitive traces remains to be evaluated.

More Information [optional]

Derek Robertson

Model Card Authors [optional]

Derek Robertson

Model Card Contact

Derek Robertson

Framework versions

• PEFT 0.20.0