steampunque/HY-MT1.5-7B-MP-GGUF overview

---

license: apache-2.0

base_model: tencent/Hy-MT1.5-7B

base_model_relation: quantized

tags:

• tencent Hy-MT
• Hy-MT1.5-7B
• GGUF
• mixed precision quantized
• 6-bit

---

Mixed Precision GGUF layer quantization of Hy-MT1.5-7B by tencent

Original model: https://huggingface.co/tencent/Hy-MT1.5-7B

The hybrid quant employs different quantization levels on a per layer basis to increased

flexibility of trading off performance vs file size. Less parameter bits are used at deep layers

and more bits at cortex layers to simultaneously optimize quantized size and model performance.

An extended layer definition E quant Q6_E_H for the model is defined as follows:

   LAYER_TYPES='[
   ["A","attn","Q","attn_q","K","attn_k","V","attn_v","O","attn_o","S","ssm","F","ffn","G","ffn_g","U","ffn_u","D","ffn_d"],
   ["MAP","VOD","0","QN_K","2","Q2_K","3","Q3_K","4","Q4_K","5","Q5_K","6","Q6_K","8","Q8_0","h","F16","f","F32"],
   [0 ,"Q6_K_888"], [1 ,"Q6_K_866"], [2 ,"Q6_K_666"], [3 ,"Q6_K_666"], [4 ,"Q5_K_666"], [5 ,"Q5_K_666"], [6 ,"Q5_K_666"], [7 ,"Q5_K_666"],
   [8 ,"Q5_K_666"], [9 ,"Q5_K_666"], [10,"Q5_K_666"], [11,"Q5_K_666"], [12,"Q5_K_666"], [13,"Q5_K_666"], [14,"Q5_K_666"], [15,"Q5_K_666"],
   [16,"Q6_K_666"], [17,"Q6_K_666"], [18,"Q6_K_666"], [19,"Q6_K_666"], [20,"Q6_K_666"], [21,"Q6_K_666"], [22,"Q6_K_666"], [23,"Q6_K_666"],
   [24,"Q6_K_866"], [25,"Q6_K_866"], [26,"Q6_K_866"], [27,"Q6_K_866"], [28,"Q6_K_888"], [29,"Q6_K_888"], [30,"Q6_K_888"], [31,"Q6_K_888"]
   ]'
   FLAGS="--token-embedding-type Q6_K --output-tensor-type Q6_K --layer-types-high"

The MP quant was optimized to ~ Q6_K bpw efficiency.

Comparison:

Quant | size | PPL | Comment

---------|---------|------|-----------

Q6_K | 6.2e9 | 22.2 | modified PPL

Q6_E_H | 6.0e9 | 22.0 | modified PPL

Usage:

Hy MT1.5 7B is an auto-id (it automatically detects source lang) machine translation model supporting 33 languages. See its model page for the list of languages it supports (same as Hy-MT2) : https://huggingface.co/tencent/Hy-MT2-7B#supported-languages

Translation prompt format:

In order for the model to work correctly for translations, a prompt template can be used as follows:

User prompt: <|startoftext|>Translate the following segment into {TL}, without additional explanation.\n\nText to be translated
Assistant prompt: <|extra_0|>

Where {TL} is from a set of language codes supported by the model, i.e {TL}=DE for tranlating english to german. The instructions

for the model state that full language names instead of codes should be used, however empirical tests with the model show that it

will produce identical translations when using either the full language name or the upper cased standard language code (not de, but DE). If using

lower case 2 letter language codes the model will mix up german and dutch target, but using upper case 2 letter langauge codes

was found to work across all tested languages.

Example using a downstream llama.cpp server with an autotokenizer implementing the above prompt template:

translate()

{
   phrase=$1
   SOURCE=$2
   TARGET=$3
   # use uppercase lang codes for Hy MT
   echo -n $(SL=${SOURCE^^} TL=${TARGET^^} lm "$phrase")
}

langs=(de es fr ru ja zh en)

phrase='The secret code to open the safe is "dead mens chest"'
SOURCE=en
echo "${SOURCE}: $phrase"
for TARGET in ${langs[@]}; do
   phrase=$(translate "$phrase" $SOURCE $TARGET)
   # Remove lead and trail \n delimiters
   phrase="${phrase#\\n}"
   phrase="${phrase%\\n}"
   echo "${TARGET}: $phrase"
   SOURCE=$TARGET
done

Running this script produces:

en: The secret code to open the safe is "dead mens chest"
de: Der geheime Code, um den Tresor zu öffnen, lautet „Toter Mannes Brust“.
es: El código secreto para abrir el baúl es «Pecho de hombre muerto».
fr: Le code secret pour ouvrir ce coffre est « Poitrine d’homme mort ».
ru: Секретный код для открытия этого сундука — «Грудь мертвеца».
ja: この箱を開けるための秘密の暗号は「死者の胸」です。
zh: 打开这个盒子的秘密密码是“死者的胸膛”。
en: The secret code to open this box is “the chest of the dead”.

Perplexity issue:

The model cannot compute valid perplexity on raw wikitext strings, which will result in very high values useless for comparative purposes. To work around this problem

a modifed perplexity is computed by overwriting the beginning of the perplexity chunk contents with the following prefix:

# chunk is a string of raw text to eval perplexity on

injects='<|startoftext|><|extra_0|>'

chunk="${injects}${chunk:${#injects}}"

logprobs are skipped over the beginning part of the perplexity prompt using a modified llama.cpp downstream server to compute the modified perplexity.

MT benchmarks for the model are given here: https://huggingface.co/spaces/steampunque/benchlm

Download the file from below:

| Link | Type | Size/e9 B | Notes |

|------|------|-----------|-------|

| HY-MT1.5-7B.Q6_E_H.gguf | Q6_E_H | 6.0e9 B | ~0.2B smaller than Q6_K |

A discussion thread about the hybrid layer quant approach can be found here on the llama.cpp git repository:

https://github.com/ggml-org/llama.cpp/discussions/13040