---

license: apache-2.0

language:

- pt

- en

pipeline_tag: text-generation

tags:

- qwen

- lora

- peft

- legal

- portuguese

- instruct

- brazilian-portuguese

- lm-evaluation-harness

- oab

- enem

base_model: unsloth/Qwen3.5-9B-Base

widget:

- text: "Explique o princípio da presunção de inocência no direito brasileiro."

example_title: "Legal Reasoning"

---

Athenas-Symbiote-9B

<div align="center">

<img src="image.png" alt="Athenas-Symbiote Logo" width="300"/>

</div>

Model Details

Athenas-Symbiote-9B is a large language model (LLM) fine-tuned for advanced reasoning, complex text interpretation, and legal/academic domain expertise, primarily optimized for Brazilian Portuguese (PT-BR).

Built upon the Qwen3.5-9B architecture, this model leverages Parameter-Efficient Fine-Tuning (PEFT) to adapt its robust multilingual baseline specifically to the nuances, terminology, and logical structures required by the Brazilian legal and educational systems.

📄 Technical Report: Available on Zenodo — full methodology, evaluation protocol, and per-edition analysis.

Model Description

• Developed by: Bruno Kodjaoglanian Cardoso Tulux (Independent AI Researcher)
• Model Type: Causal Language Model (Transformer)
• Base Model: unsloth/Qwen3.5-9B-Base
• Parameter Count: 9 Billion
• Context Length: 32,768 tokens
• Language(s) (NLP): Portuguese (Primary), English
• License: Apache 2.0
• Finetuning Approach: LoRA (Low-Rank Adaptation)

---

Evaluation & Benchmarks

The model was comprehensively evaluated using the lm-evaluation-harness framework. Inference was conducted on high-performance infrastructure (1x NVIDIA H100 80GB) utilizing native bfloat16 precision to ensure metric fidelity and prevent quantization degradation.

[Insert your consolidated performance bar/radar chart here]

National Benchmarks (Brazil)

Evaluations targeting the Brazilian cultural, legal, and educational framework.

| Benchmark | Metric | Score (Zero/Few-Shot) |

| :--- | :--- | :---: |

| OAB Exams (Brazilian Bar Association) | Accuracy | 63.23% (3-shot) |

| ENEM Challenge (National High School Exam) | Accuracy | 80.90% (3-shot) |

> Benchmark Datasets:

> - OAB Exams: maritaca-ai/oab-bench

> - ENEM Challenge: maritaca-ai/enem

ENEM Challenge — Detailed Breakdown by Exam

| Exam | Accuracy | Std. Error |

| :--- | :---: | :---: |

| 2009 | 84.35% | ±1.95% |

| 2010 | 83.76% | ±1.97% |

| 2011 | 85.47% | ±1.88% |

| 2012 | 83.62% | ±1.98% |

| 2013 | 79.63% | ±2.24% |

| 2014 | 81.65% | ±2.15% |

| 2015 | 81.51% | ±2.05% |

| 2016 | 77.69% | ±2.17% |

| 2016 (2ª Aplicação) | 80.49% | ±2.07% |

| 2017 | 82.76% | ±2.03% |

| 2022 | 71.43% | ±2.26% |

| 2023 | 80.00% | ±2.00% |

| Overall | 80.90% | ±0.60% |

OAB Exams — Detailed Breakdown by Exam

| Exam | Accuracy | Std. Error |

| :--- | :---: | :---: |

| 2010-01 | 50.59% | ±3.13% |

| 2010-02 | 68.00% | ±2.69% |

| 2011-03 | 56.57% | ±2.89% |

| 2011-04 | 56.25% | ±3.21% |

| 2011-05 | 65.00% | ±3.08% |

| 2012-06 | 65.00% | ±3.08% |

| 2012-06a | 72.50% | ±2.88% |

| 2012-07 | 57.50% | ±3.20% |

| 2012-08 | 65.00% | ±3.07% |

| 2012-09 | 54.55% | ±3.26% |

| 2013-10 | 63.75% | ±3.11% |

| 2013-11 | 58.75% | ±3.20% |

| 2013-12 | 71.25% | ±2.92% |

| 2014-13 | 53.75% | ±3.22% |

| 2014-14 | 75.00% | ±2.80% |

| 2014-15 | 71.79% | ±2.94% |

| 2015-16 | 58.75% | ±3.19% |

| 2015-17 | 71.79% | ±2.95% |

| 2015-18 | 62.50% | ±3.11% |

| 2016-19 | 62.82% | ±3.17% |

| 2016-20 | 61.25% | ±3.14% |

| 2016-20a | 61.25% | ±3.12% |

| 2016-21 | 58.75% | ±3.18% |

| 2017-22 | 72.50% | ±2.88% |

| 2017-23 | 65.00% | ±3.08% |

| 2017-24 | 68.75% | ±3.00% |

| 2018-25 | 60.00% | ±3.17% |

| Overall | 63.23% | ±0.60% |

Global Benchmarks (Translated / Multilingual)

Frontier evaluations assessing general intelligence and mathematical reasoning.

| Benchmark | Description | Score |

| :--- | :--- | :---: |

| MMLU | Massive Multitask Language Understanding (57 subjects) | — |

| GSM8K | Grade School Math (Chain-of-Thought Reasoning) | — |

| HellaSwag | Commonsense NLI | — |

| ARC Challenge | Advanced Scientific Reasoning | — |

> Note: Global benchmark evaluations are pending. Scores will be updated upon completion.

---

How to Get Started with the Model

Use the code below to get started with the model. As this is a base model fine-tuned with LoRA, there is no built-in chat template. The ChatML format is recommended for optimal inference with instruction-style prompts.

Requirements

pip install transformers accelerate bitsandbytes

Python Inference

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model_id = "Kodjaoglanian/Athenas-Symbiote-9B"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    device_map="auto",
    torch_dtype=torch.bfloat16
)

messages = [
    {"role": "system", "content": "Você é a Athenas, uma assistente virtual altamente inteligente e precisa."},
    {"role": "user", "content": "Explique a diferença entre dolo e culpa no código penal brasileiro."}
]

text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer([text], return_tensors="pt").to(model.device)

outputs = model.generate(
    **inputs, 
    max_new_tokens=512, 
    temperature=0.3,
    do_sample=True
)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

---

Training Details

Training Data

The model was fine-tuned on the ClassiCC-Corpus/ClassiCC-PT dataset, consumed in streaming mode with a strict quality filter (edu_score > 0.8). This heuristic ensured the model ingested only high-rigor educational, literary, and academic content. The training corpus includes:

• Brazilian jurisprudence, legislation, and legal proceedings.
• Academic literature and technical documentation in Portuguese.
• Chain-of-Thought (CoT) reasoning structures.

Data Pipeline: Streaming → edu_score > 0.8 filter → EOS concatenation → Sequence packing (packing=True)

Training Pipeline

graph TD
    A["Load Qwen3.5-9B-Base<br/>(4-bit, Unsloth)"] --> B["Configure PEFT<br/>LoRA r=32, α=16, rsLoRA"]
    B --> C["Dataset: ClassiCC-PT<br/>(streaming, edu_score > 0.8)"]
    C --> D["SFTTrainer<br/>1,200 steps, LR 5e-5<br/>adamw_8bit, packing=True"]
    D --> E{"Training OK?"}
    E -- No --> F["Auto-retry<br/>(5x, resume checkpoint)"]
    F --> D
    E -- Yes --> G["Push to HF Hub<br/>(Safetensors + Tokenizer)"]
    G --> H["Evaluation<br/>lm-eval-harness<br/>H100, BF16, 3-shot"]

    style A fill:#1B3A5C,color:#fff
    style B fill:#1B3A5C,color:#fff
    style C fill:#1B3A5C,color:#fff
    style D fill:#1B3A5C,color:#fff
    style F fill:#C5A059,color:#fff
    style G fill:#C5A059,color:#fff
    style H fill:#C5A059,color:#fff

Training: Kaggle T4 (16GB) | Evaluation: Lightning AI H100 (80GB)

Training Procedure

The fine-tuning process prioritized structural knowledge retention and computational efficiency.

• Hardware: 1x NVIDIA T4 (16GB VRAM) via Kaggle
• Quantization: 4-bit (NF4 via bitsandbytes)
• Precision: FP16 (T4 does not support BF16)
• Framework: Unsloth + TRL + PEFT + bitsandbytes
• Optimization: LoRA adapters applied to all linear projections
• LoRA Rank: 32
• LoRA Alpha: 16
• Scaling Strategy: rsLoRA (use_rslora=True)
• LoRA Dropout: 0.0
• Target Modules: q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
• Learning Rate: 5e-5
• LR Scheduler: Linear
• Warmup Steps: 50
• Training Steps: 1,200 (Dataset via Streaming)
• Batch Size: 1 (Effective Batch Size: 8 via Gradient Accumulation)
• Optimizer: adamw_8bit
• Weight Decay: 0.01
• Packing: True
• Gradient Checkpointing: Unsloth
• Seed: 3407
• Max Sequence Length (Training): 512 tokens

Evaluation Hardware

• Infrastructure: Lightning AI platform
• GPU: 1x NVIDIA H100 (80GB VRAM)
• Precision: bfloat16 (native, no quantization)
• Framework: lm-evaluation-harness (EleutherAI)
• Few-Shot: 3-shot
• Bootstrap Iterations: 100,000
• Execution: Sequential evaluation to isolate memory overhead.

---

Intended Use & Limitations

Intended Use

• Primary Use Case: Research and application in Natural Language Processing (NLP) specifically for the Portuguese language.
• Secondary Use Case: Automation of complex text analysis, summarization, and logical reasoning within legal and academic contexts (assisted by human review).

Out-of-Scope Use

The model is not designed to operate autonomously in high-stakes environments. It should not be used for:

• Automated legal counsel or medical diagnosis without human-in-the-loop validation.
• Generation of deceptive, malicious, or highly biased content.

Bias, Risks, and Limitations

Like all statistical language models, Athenas-Symbiote-9B is susceptible to generating hallucinations—plausible-sounding but factually incorrect information. Furthermore, the model may inherit and amplify biases present in its pre-training data and fine-tuning corpus. Users must exercise critical judgment and implement rigorous verification protocols before utilizing the model's outputs in production environments.

---

Citation

If you use this model in your research, please cite it as follows:

@misc{athenas_symbiote_9b,
  author       = {Bruno Kodjaoglanian Cardoso Tulux},
  title        = {Athenas-Symbiote-9B: A Parameter-Efficient Fine-Tuned LLM for Brazilian Portuguese Legal and Academic Domains},
  year         = {2026},
  publisher    = {Zenodo},
  doi          = {10.5281/zenodo.21418168},
  url          = {https://zenodo.org/records/21418168},
  note         = {Model weights: https://huggingface.co/Kodjaoglanian/Athenas-Symbiote-9B}
}