Seems to be working with PR and `--jinja`
Without --jinja it seemed to go off the rails and spit out chat template tokens when testing the chat completions endpoint.
This makes sense given the PR says:
not doing the chat template yet because not sure how to handle the interleaving thinking blocks.
Though with --jinja, it seems to work okay in my limited testing and seems to be in "thinking" mode. Getting over 20 tok/sec generation in short context on CPU-only (one big socket with AMD EPYC 9975 with 768GB DDR5-6400MT/s in NPS1):
model=/mnt/raid/models/DevQuasar/MiniMaxAI.MiniMax-M2-GGUF/MiniMaxAI.MiniMax-M2.Q8_0-00001-of-00019.gguf
SOCKET=0
numactl -N "$SOCKET" -m "$SOCKET" \
./build/bin/llama-server \
--model "$model"\
--alias DevQuasar/MiniMax-M2-GGUF \
--ctx-size 32768 \
-fa 1 \
-ub 4096 -b 4096 \
--parallel 1 \
--threads 96 \
--threads-batch 128 \
--numa numactl \
--host 127.0.0.1 \
--port 8080 \
--no-mmap \
--jinja \
--temp 1.0 \
--top-k 40 \
--top-p 0.95
print_info: file type = Q8_0
print_info: file size = 226.43 GiB (8.51 BPW)
Also note this model is a bit funky and wants "interleaved thinking" which is likely why the chat template issues:
IMPORTANT: MiniMax-M2 is an interleaved thinking model. Therefore, when using it, it is important to retain the thinking content from the assistant's turns within the historical messages. In the model's output content, we use the ... format to wrap the assistant's thinking content. When using the model, you must ensure that the historical content is passed back in its original format. Do not remove the ... part, otherwise, the model's performance will be negatively affected.
Thanks for making these available for testing!
I ran perplexity on the q8_0 with my usual 512 context size wiki.test.raw full ~1.3MB file:
model=/mnt/raid/models/DevQuasar/MiniMaxAI.MiniMax-M2-GGUF/MiniMaxAI.MiniMax-M2.Q8_0-00001-of-00019.gguf
SOCKET=0
numactl -N "$SOCKET" -m "$SOCKET" \
./build/bin/llama-perplexity \
-m "$model" \
-f wiki.test.raw \
--seed 1337 \
-fa 1 \
--ctx-size 512 \
-ub 4096 -b 4096 \
--numa numactl \
--threads 96 \
--threads-batch 128 \
--no-mmap
...
Final estimate: PPL = 6.9930 +/- 0.04889
What I've seen it's often skipping the but only at the first response. Later seemingly it uses correctly
I downloaded the latest source code for llama.cpp, compiled it, but it still doesn’t work. Could it be that the code hasn’t been integrated into the main project? The error message is: “llama_model_load: error loading model; error loading model architecture: unknown model architecture: ‘minimax-m2’”. Additionally, “llama_model_load_from_file_impl: failed to load model” is also displayed.
Build llama.cpp from the branch. The PR has not merged yet
https://github.com/ggml-org/llama.cpp/pull/16831