ddh0/meta-llama-3-8b-instruct-bf16-gguf overview

{
  "metadata": {},
  "card_data": {
    "license": "llama3",
    "pipeline_tag": "text-generation",
    "frontmatter": {
      "license": "llama3",
      "pipeline_tag": "text-generation"
    },
    "hero_image_url": "",
    "summary": "This is meta-llama/Meta-Llama-3-8B-Instruct, converted to GGUF without changing tensor data type. Moreover, the new correct pre-tokenizer llama-bpe is used (ref), and the EOS token is correctly set to  (ref). The llama.cpp output for this model is shown below for reference. `` Log start main: build = 2842 (18e43766) main: built with cc (Debian 12.2.0-14) 12.2.0 for x86_64-linux-gnu main: seed  = 1715355914 llama_model_loader: loaded meta data with 22 key-value pairs and 291 tensors from /media/dylan/SanDisk/LLMs/Meta-Llama-3-8B-Instruct-bf16.gguf (version GGUF V3 (latest)) llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output. llama_model_loader: - kv   0:                       general.architecture str              = llama llama_model_loader: - kv   1:                               general.name str              = Meta-Llama-3-8B-Instruct llama_model_loader: - kv   2:                          llama.block_count u32              = 32 llama_model_loader: - kv   3:                       llama.context_length u32              = 8192 llama_model_loader: - kv   4:                     llama.embedding_length u32              = 4096 llama_model_loader: - kv   5:                  llama.feed_forward_length u32              = 14336 llama_model_loader: - kv   6:                 llama.attention.head_count u32              = 32 llama_model_loader: - kv   7:              llama.attention.head_count_kv u32              = 8 llama_model_loader: - kv   8:                       llama.rope.freq_base f32              = 500000.000000 llama_model_loader: - kv   9:     llama.attention.layer_norm_rms_epsilon f32              = 0.000010 llama_model_loader: - kv  10:                          general.file_type u32              = 32 llama_model_loader: - kv  11:                           llama.vocab_size u32              = 128256 llama_model_loader: - kv  12:                 llama.rope.dimension_count u32              = 128 llama_model_loader: - kv  13:                       tokenizer.ggml.model str              = gpt2 llama_model_loader: - kv  14:                         tokenizer.ggml.pre str              = llama-bpe llama_model_loader: - kv  15:                      tokenizer.ggml.tokens arr[str,128256]  = [\"!\", \"\\\"\", \"#\", \"$\", \"%\", \"&\", \"'\", ... llama_model_loader: - kv  16:                  tokenizer.ggml.token_type arr[i32,128256]  = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ... llama_model_loader: - kv  17:                      tokenizer.ggml.merges arr[str,280147]  = [\"Ġ Ġ\", \"Ġ ĠĠĠ\", \"ĠĠ ĠĠ\", \"... llama_model_loader: - kv  18:                tokenizer.ggml.bos_token_id u32              = 128000 llama_model_loader: - kv  19:                tokenizer.ggml.eos_token_id u32              = 128009 llama_model_loader: - kv  20:                    tokenizer.chat_template str              = {% set loop_messages = messages %}{% ... llama_model_loader: - kv  21:               general.quantization_version u32              = 2 llama_model_loader: - type  f32:   65 tensors llama_model_loader: - type bf16:  226 tensors llm_load_vocab: special tokens definition check successful ( 256/128256 ). llm_load_print_meta: format           = GGUF V3 (latest) llm_load_print_meta: arch             = llama llm_load_print_meta: vocab type       = BPE llm_load_print_meta: n_vocab          = 128256 llm_load_print_meta: n_merges         = 280147 llm_load_print_meta: n_ctx_train      = 8192 llm_load_print_meta: n_embd           = 4096 llm_load_print_meta: n_head           = 32 llm_load_print_meta: n_head_kv        = 8 llm_load_print_meta: n_layer          = 32 llm_load_print_meta: n_rot            = 128 llm_load_print_meta: n_embd_head_k    = 128 llm_load_print_meta: n_embd_head_v    = 128 llm_load_print_meta: n_gqa            = 4 llm_load_print_meta: n_embd_k_gqa     = 1024 llm_load_print_meta: n_embd_v_gqa     = 1024 llm_load_print_meta: f_norm_eps       = 0.0e+00 llm_load_print_meta: f_norm_rms_eps   = 1.0e-05 llm_load_print_meta: f_clamp_kqv      = 0.0e+00 llm_load_print_meta: f_max_alibi_bias = 0.0e+00 llm_load_print_meta: f_logit_scale    = 0.0e+00 llm_load_print_meta: n_ff             = 14336 llm_load_print_meta: n_expert         = 0 llm_load_print_meta: n_expert_used    = 0 llm_load_print_meta: causal attn      = 1 llm_load_print_meta: pooling type     = 0 llm_load_print_meta: rope type        = 0 llm_load_print_meta: rope scaling     = linear llm_load_print_meta: freq_base_train  = 500000.0 llm_load_print_meta: freq_scale_train = 1 llm_load_print_meta: n_yarn_orig_ctx  = 8192 llm_load_print_meta: rope_finetuned   = unknown llm_load_print_meta: ssm_d_conv       = 0 llm_load_print_meta: ssm_d_inner      = 0 llm_load_print_meta: ssm_d_state      = 0 llm_load_print_meta: ssm_dt_rank      = 0 llm_load_print_meta: model type       = 8B llm_load_print_meta: model ftype      = BF16 llm_load_print_meta: model params     = 8.03 B llm_load_print_meta: model size       = 14.96 GiB (16.00 BPW) llm_load_print_meta: general.name     = Meta-Llama-3-8B-Instruct llm_load_print_meta: BOS token        = 128000 '' llm_load_print_meta: EOS token        = 128009 '' llm_load_print_meta: LF token         = 128 'Ä' llm_load_print_meta: EOT token        = 128009 '' ``",
    "quick_links": [],
    "benchmark_table_html": "",
    "readme_markdown": "---\nlicense: llama3\npipeline_tag: text-generation\n---\n\n# Meta-Llama-3-8B-Instruct-bf16-GGUF\n\nThis is [meta-llama/Meta-Llama-3-8B-Instruct](https://huggingface.co/meta-llama/Meta-Llama-3-8B-Instruct), converted to GGUF without changing tensor data type. Moreover, the new correct pre-tokenizer `llama-bpe` is used ([ref](https://github.com/ggerganov/llama.cpp/pull/6745#issuecomment-2094991999)), and the EOS token is correctly set to `<|eot_id|>` ([ref](https://huggingface.co/meta-llama/Meta-Llama-3-8B-Instruct/commit/a8977699a3d0820e80129fb3c93c20fbd9972c41)).\n\nThe `llama.cpp` output for this model is shown below for reference.\n\n```\nLog start\nmain: build = 2842 (18e43766)\nmain: built with cc (Debian 12.2.0-14) 12.2.0 for x86_64-linux-gnu\nmain: seed  = 1715355914\nllama_model_loader: loaded meta data with 22 key-value pairs and 291 tensors from /media/dylan/SanDisk/LLMs/Meta-Llama-3-8B-Instruct-bf16.gguf (version GGUF V3 (latest))\nllama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.\nllama_model_loader: - kv   0:                       general.architecture str              = llama\nllama_model_loader: - kv   1:                               general.name str              = Meta-Llama-3-8B-Instruct\nllama_model_loader: - kv   2:                          llama.block_count u32              = 32\nllama_model_loader: - kv   3:                       llama.context_length u32              = 8192\nllama_model_loader: - kv   4:                     llama.embedding_length u32              = 4096\nllama_model_loader: - kv   5:                  llama.feed_forward_length u32              = 14336\nllama_model_loader: - kv   6:                 llama.attention.head_count u32              = 32\nllama_model_loader: - kv   7:              llama.attention.head_count_kv u32              = 8\nllama_model_loader: - kv   8:                       llama.rope.freq_base f32              = 500000.000000\nllama_model_loader: - kv   9:     llama.attention.layer_norm_rms_epsilon f32              = 0.000010\nllama_model_loader: - kv  10:                          general.file_type u32              = 32\nllama_model_loader: - kv  11:                           llama.vocab_size u32              = 128256\nllama_model_loader: - kv  12:                 llama.rope.dimension_count u32              = 128\nllama_model_loader: - kv  13:                       tokenizer.ggml.model str              = gpt2\nllama_model_loader: - kv  14:                         tokenizer.ggml.pre str              = llama-bpe\nllama_model_loader: - kv  15:                      tokenizer.ggml.tokens arr[str,128256]  = [\"!\", \"\\\"\", \"#\", \"$\", \"%\", \"&\", \"'\", ...\nllama_model_loader: - kv  16:                  tokenizer.ggml.token_type arr[i32,128256]  = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...\nllama_model_loader: - kv  17:                      tokenizer.ggml.merges arr[str,280147]  = [\"Ġ Ġ\", \"Ġ ĠĠĠ\", \"ĠĠ ĠĠ\", \"...\nllama_model_loader: - kv  18:                tokenizer.ggml.bos_token_id u32              = 128000\nllama_model_loader: - kv  19:                tokenizer.ggml.eos_token_id u32              = 128009\nllama_model_loader: - kv  20:                    tokenizer.chat_template str              = {% set loop_messages = messages %}{% ...\nllama_model_loader: - kv  21:               general.quantization_version u32              = 2\nllama_model_loader: - type  f32:   65 tensors\nllama_model_loader: - type bf16:  226 tensors\nllm_load_vocab: special tokens definition check successful ( 256/128256 ).\nllm_load_print_meta: format           = GGUF V3 (latest)\nllm_load_print_meta: arch             = llama\nllm_load_print_meta: vocab type       = BPE\nllm_load_print_meta: n_vocab          = 128256\nllm_load_print_meta: n_merges         = 280147\nllm_load_print_meta: n_ctx_train      = 8192\nllm_load_print_meta: n_embd           = 4096\nllm_load_print_meta: n_head           = 32\nllm_load_print_meta: n_head_kv        = 8\nllm_load_print_meta: n_layer          = 32\nllm_load_print_meta: n_rot            = 128\nllm_load_print_meta: n_embd_head_k    = 128\nllm_load_print_meta: n_embd_head_v    = 128\nllm_load_print_meta: n_gqa            = 4\nllm_load_print_meta: n_embd_k_gqa     = 1024\nllm_load_print_meta: n_embd_v_gqa     = 1024\nllm_load_print_meta: f_norm_eps       = 0.0e+00\nllm_load_print_meta: f_norm_rms_eps   = 1.0e-05\nllm_load_print_meta: f_clamp_kqv      = 0.0e+00\nllm_load_print_meta: f_max_alibi_bias = 0.0e+00\nllm_load_print_meta: f_logit_scale    = 0.0e+00\nllm_load_print_meta: n_ff             = 14336\nllm_load_print_meta: n_expert         = 0\nllm_load_print_meta: n_expert_used    = 0\nllm_load_print_meta: causal attn      = 1\nllm_load_print_meta: pooling type     = 0\nllm_load_print_meta: rope type        = 0\nllm_load_print_meta: rope scaling     = linear\nllm_load_print_meta: freq_base_train  = 500000.0\nllm_load_print_meta: freq_scale_train = 1\nllm_load_print_meta: n_yarn_orig_ctx  = 8192\nllm_load_print_meta: rope_finetuned   = unknown\nllm_load_print_meta: ssm_d_conv       = 0\nllm_load_print_meta: ssm_d_inner      = 0\nllm_load_print_meta: ssm_d_state      = 0\nllm_load_print_meta: ssm_dt_rank      = 0\nllm_load_print_meta: model type       = 8B\nllm_load_print_meta: model ftype      = BF16\nllm_load_print_meta: model params     = 8.03 B\nllm_load_print_meta: model size       = 14.96 GiB (16.00 BPW) \nllm_load_print_meta: general.name     = Meta-Llama-3-8B-Instruct\nllm_load_print_meta: BOS token        = 128000 '<|begin_of_text|>'\nllm_load_print_meta: EOS token        = 128009 '<|eot_id|>'\nllm_load_print_meta: LF token         = 128 'Ä'\nllm_load_print_meta: EOT token        = 128009 '<|eot_id|>'\n```\n",
    "related_quantizations": []
  },
  "tags": [
    "gguf",
    "text-generation",
    "license:llama3",
    "endpoints_compatible",
    "region:us",
    "conversational"
  ],
  "likes": 43,
  "downloads": 94,
  "gated": false,
  "private": false,
  "last_modified": "2024-05-10T16:06:08.000Z",
  "created_at": "2024-05-10T15:47:56.000Z",
  "pipeline_tag": "text-generation",
  "library_name": ""
}

{
  "_id": "663e41aca17033aedb7ea13e",
  "id": "ddh0/Meta-Llama-3-8B-Instruct-bf16-GGUF",
  "modelId": "ddh0/Meta-Llama-3-8B-Instruct-bf16-GGUF",
  "sha": "cdd97fe0648691f001c2ed2edebe5a6372d8160d",
  "createdAt": "2024-05-10T15:47:56.000Z",
  "lastModified": "2024-05-10T16:06:08.000Z",
  "author": "ddh0",
  "downloads": 94,
  "likes": 43,
  "gated": false,
  "private": false,
  "pipeline_tag": "text-generation",
  "library_name": "",
  "siblings_count": 3
}