---

license: apache-2.0

language:

• en

library_name: transformers

pipeline_tag: text-generation

datasets:

• HuggingFaceFW/fineweb-edu
• epfml/FineWeb-HQ
• HuggingFaceTB/smollm-corpus

tags:

• lm
• language-model
• causal-lm
• causal-language-model
• decoder-only
• base-model
• pretraining
• small-language-model
• applemind
• applemind10
• applemind10-mini
• fineweb-edu
• fineweb-hq
• smollm-corpus
• cosmopedia-v2
• pytorch
• safetensors
• custom-code
• custom-architecture
• trust-remote-code

base_model:

• AppleMind-AI/AppleMind-1.0-Mini

---

gguf version incase anyone wants it :)

sadly the model is too small for a q8 or under gguf :(

so only fp32, fp16, and bf16 are avaliable for gguf

AppleMind-1.0-Mini

!Banner

AppleMind-1.0-Mini is a compact decoder-only causal language model trained from scratch by AppleMind on a 300M-token curriculum.

The model has 1,020,480 parameters, a 256-token context window, and a 50,263-token digit-aware byte-level BPE tokenizer (GPT-2 + special tokens).

Model Details

| Field | Value |

| -------------------- | ------------------------------------------------------: |

| Parameters | 1,020,480 |

| Architecture | AppleMind 1.0 Mini decoder-only Transformer |

| Layers | 4 |

| Hidden size | 128 |

| Intermediate size | 512 |

| Attention heads | 4 |

| KV heads | 4 |

| Head dim | 32 |

| Attention style | Multi-head causal self-attention |

| MLP | GELU |

| Position embeddings | Learned positional embeddings |

| Normalization | LayerNorm |

| Vocabulary size | 50,263 |

| Context length | 256 |

| Embeddings | Tied input/output embeddings |

| Tokenizer | Digit-aware byte-level BPE (GPT-2 + special tokens) |

| Weight format | safetensors |

| HF architecture | GPT2LMHeadModel |

| HF model type | gpt2 |

| Final training steps | 2,288 |

| Tokens seen | 299,892,736 |

| Tokens/parameter | 293.87:1 |

| Training data | FineWeb-Edu + FineWeb-HQ + SmolLM-Corpus |

| Training mixture | 100M + 100M + 100M tokens |

| Precision | BF16 |

| Final model | AppleMind 1.0 Mini |

Credits to BananaMind for inspiring me to make AppleMind.

Tokenizer

AppleMind 1.0 Mini uses a 50,263-token digit-aware byte-level BPE tokenizer based on the GPT-2 tokenizer, with 3 additional special tokens. Digits are handled individually rather than being collapsed into large number tokens.

| Special token | ID |

| ------------- | --: |

| <&#124;pad&#124;> | 50,260 |

| <&#124;bos&#124;> | 50,261 |

| <&#124;eos&#124;> | 50,262 |

Training Data

| Dataset | Target Tokens | Share |

| ------------- | ------------: | ----: |

| FineWeb-Edu | 100M | 33.33% |

| FineWeb-HQ | 100M | 33.33% |

| SmolLM-Corpus | 100M | 33.33% |

| Total | 300M | 100% |

The training run used an equal mixture of FineWeb-Edu, FineWeb-HQ, and SmolLM-Corpus, with 100M tokens sampled from each dataset.

Training Setup

| Field | Value |

| --------------------------- | ------------: |

| Sequence length | 256 |

| Micro batch | 512 sequences |

| Gradient accumulation | 1 |

| Effective batch | 512 sequences |

| Tokens per optimizer step | 131,072 |

| Final optimizer step | 2,288 |

| Configured optimizer steps | 2,289 |

| Optimizer | AdamW |

| Peak learning rate | 0.0001 |

| Learning rate at final step | 3.114e-06 |

| LR schedule | Cosine decay |

| Gradient clipping | 1 |

| Weight format | safetensors |

| Training tokens | 299,892,736 |

| Target tokens | 300,000,000 |

| Tokens/parameter | 293.87:1 |

Evaluation

AppleMind 1.0 Mini has not been formally evaluated with lm_eval yet. No benchmark scores are currently reported.

| Benchmark | Score | Metric |

| ------------- | ------: | --------------- |

| Average | N/A | mean |

| ARC Easy | N/A | acc_norm,none |

| PIQA | N/A | acc_norm,none |

| ARC Challenge | N/A | acc_norm,none |

| HellaSwag | N/A | acc_norm,none |

The model's current generation quality has been checked with basic text-generation prompts, including:

• Once upon a time
• The little boy
• In the forest

Prompt: Once upon a time

------------------------------------------------------------

Once upon a time It the several times- nowThis, does to form provide from find another times work not atl The couldWhen on all be way H It lives among times always, worked

its G during work used after several There and at there

b known came be very that It thought It betweenIn course does. case other It 5?: or often I at's the: enough could in many

------------------------------------------------------------

Prompt: The little boy

------------------------------------------------------------

The little boy's form among. I and H be from� used. still all- G, lives take same always often its find amonged provide- number because: another now? then there among use course not thought case work usel result It between 5 It well atWhen new.: thatThis work on think 3 interestB then It does could do the among Ire use among now does to and many

------------------------------------------------------------

Prompt: In the forest

------------------------------------------------------------

In the forest often

form and times on during severall find, several The then number between: well work take there provide, times among anotherWhen same Ged but lives could the to times its- course same enough and same I used

to same other not thought There H think� same 2 I body nowe cameb 5- do among's and several? same after- still,This- interest but

------------------------------------------------------------

Usage

AppleMind 1.0 Mini uses custom architecture code, so load it with trust_remote_code=True.

pip install -U transformers safetensors torch

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "AppleMind-AI/AppleMind-1.0-Mini"

tokenizer = AutoTokenizer.from_pretrained(
    model_id,
    trust_remote_code=True,
)

device = "cuda" if torch.cuda.is_available() else "cpu"

dtype = (
    torch.bfloat16
    if torch.cuda.is_available() and torch.cuda.is_bf16_supported()
    else torch.float32
)

model = AutoModelForCausalLM.from_pretrained(
    model_id,
    trust_remote_code=True,
    torch_dtype=dtype,
).to(device).eval()

prompt = "The color of the sky is"

inputs = tokenizer(
    prompt,
    return_tensors="pt",
).to(device)

with torch.no_grad():
    output = model.generate(
        **inputs,
        max_new_tokens=96,
        do_sample=True,
        temperature=0.7,
        top_p=0.9,
        repetition_penalty=1.1,
        pad_token_id=tokenizer.eos_token_id,
        eos_token_id=tokenizer.eos_token_id,
    )

print(
    tokenizer.decode(
        output[0],
        skip_special_tokens=True,
    )
)

Note: AppleMind 1.0 Mini has a 256-token context window, so the prompt plus generated tokens should stay within that limit.

License

Apache 2.0