KIMI-K3-GGUF

This repository contains the GGUF format conversion of moonshotai/Kimi-K3 , a powerful large language model developed by Moonshot AI. The conversion has been performed using the specialized llama.cpp fork that includes native support for the Kimi-K3 architecture.

Original Model Information

Model Name: Kimi-K3

Developed by: Moonshot AI

Architecture: Custom architecture with specialized encoding_k3 module for efficient tokenization and processing

Context Length: [Insert the original model's context length]

Description: Kimi-K3 is a state-of-the-art large language model designed for complex reasoning, long-context understanding, and a wide variety of natural language processing tasks. For detailed information about capabilities, training data, and limitations, please refer to the original model card.

GGUF Conversion Details

Prerequisites

The conversion was performed using a specific environment with the following dependencies:

bash

Clone the llama.cpp repository with Kimi-K3 support

git clone https://github.com/ggml-org/llama.cpp.git

cd llama.cpp

git fetch origin pull/26185/head:pwilkin/kimi-k3-text

git checkout pwilkin/kimi-k3-text

Environment Setup

bash

Create and activate conda environment

conda create -n kimi-convert python=3.10

conda activate kimi-convert

Install required packages

pip install huggingface-hub==0.17.3

pip install accelerate==0.24.1

pip install numpy==1.24.4

pip install transformers==4.31.0

Conversion Command

The model was converted using the following command:

bash

Set environment variables

export PYTHONPATH="/mnt/sdb/models--moonshotai--Kimi-K3/snapshots/9f62e4e9fffbd0a83ddd60e1c209d828994b3569:$PYTHONPATH"

export NUMEXPR_MAX_THREADS=128

Verify the custom tokenizer imports correctly

python -c "import encoding_k3; print('✓ encoding_k3 imported successfully!')"

Perform the conversion

python convert_hf_to_gguf.py /mnt/sdb/models--moonshotai--Kimi-K3/snapshots/9f62e4e9fffbd0a83ddd60e1c209d828994b3569 \

--outfile /mnt/sdb/KIMI-K3.GGUF \

--outtype auto

Key Points about the Conversion:

The --outtype auto flag automatically selects the optimal output type (typically F16 for the base model)

The PYTHONPATH modification ensures the custom encoding_k3 module (Kimi-K3's specialized tokenizer) is accessible

The conversion leverages the specific PR branch that adds native Kimi-K3 support to llama.cpp

High-Performance Computing (HPC) cluster

This KIMI-K3 base model powers our EASA-compliant High-Performance Computing (HPC) cluster, specifically architected for Aviation Safety and Compliance workflows at masi-ai.com.

KHM