marin-community

/

marin-32b-base

like 23

Follow

The Marin Project 85

Hello! Congrats on the release!! Really excited to try the model.

Is there a recommended setup for vLLM? For example:

from vllm import LLM, SamplingParams

llm = LLM(model="marin-community/marin-32b-base")

prompts = [
    "We may have knowledge of the past but cannot control it; we may control the future but"
]

sampling_params = SamplingParams(
    temperature=0.7,
    top_p=0.9,
    max_tokens=128,
)

outputs = llm.generate(prompts, sampling_params)

for i, output in enumerate(outputs):
    print(prompts[i])
    print(output.outputs[0].text.strip())

I've tried the script, installing with:

# current vLLM
pip install vllm==0.11.0

# nightly vLLM
pip install -U vllm --pre --extra-index-url https://wheels.vllm.ai/nightly

In both cases, this script will fail with the error (note, I'm using the v1 engine):

...
(EngineCore_DP0 pid=11777) (Worker_TP0 pid=11783) ERROR 10-29 17:44:30 [multiproc_executor.py:631]   File "/oe-eval-default/davidh/marindebug/.venv/lib/python3.12/site-packages/vllm/model_executor/models/llama.py", line 503, in load_weights
(EngineCore_DP0 pid=11777) (Worker_TP0 pid=11783) ERROR 10-29 17:44:30 [multiproc_executor.py:631]     param = params_dict[name]
(EngineCore_DP0 pid=11777) (Worker_TP0 pid=11783) ERROR 10-29 17:44:30 [multiproc_executor.py:631]             ~~~~~~~~~~~^^^^^^
(EngineCore_DP0 pid=11777) (Worker_TP0 pid=11783) ERROR 10-29 17:44:30 [multiproc_executor.py:631] KeyError: 'layers.0.self_attn.k_norm.weight'

Finally, when trying the install with an earlier version vllm==0.9.0.1, I'm seeing this slightly different loading error:

2025-10-29T23:29:33.291Z (VllmWorker rank=0 pid=487) ERROR 10-29 23:29:33 [multiproc_executor.py:487]   File "/stage/src/vllm/vllm/model_executor/models/llama.py", line 601, in load_weights
2025-10-29T23:29:33.291Z (VllmWorker rank=0 pid=487) ERROR 10-29 23:29:33 [multiproc_executor.py:487]     return loader.load_weights(
2025-10-29T23:29:33.291Z (VllmWorker rank=0 pid=487) ERROR 10-29 23:29:33 [multiproc_executor.py:487]            ^^^^^^^^^^^^^^^^^^^^
2025-10-29T23:29:33.291Z (VllmWorker rank=0 pid=487) ERROR 10-29 23:29:33 [multiproc_executor.py:487]   File "/stage/src/vllm/vllm/model_executor/models/utils.py", line 291, in load_weights
2025-10-29T23:29:33.291Z (VllmWorker rank=0 pid=487) ERROR 10-29 23:29:33 [multiproc_executor.py:487]     autoloaded_weights = set(self._load_module("", self.module, weights))
2025-10-29T23:29:33.291Z (VllmWorker rank=0 pid=487) ERROR 10-29 23:29:33 [multiproc_executor.py:487]                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-10-29T23:29:33.291Z (VllmWorker rank=0 pid=487) ERROR 10-29 23:29:33 [multiproc_executor.py:487]   File "/stage/src/vllm/vllm/model_executor/models/utils.py", line 249, in _load_module
2025-10-29T23:29:33.291Z (VllmWorker rank=0 pid=487) ERROR 10-29 23:29:33 [multiproc_executor.py:487]     yield from self._load_module(prefix,
2025-10-29T23:29:33.291Z (VllmWorker rank=0 pid=487) ERROR 10-29 23:29:33 [multiproc_executor.py:487]   File "/stage/src/vllm/vllm/model_executor/models/utils.py", line 222, in _load_module
2025-10-29T23:29:33.291Z (VllmWorker rank=0 pid=487) ERROR 10-29 23:29:33 [multiproc_executor.py:487]     loaded_params = module_load_weights(weights)
2025-10-29T23:29:33.291Z (VllmWorker rank=0 pid=487) ERROR 10-29 23:29:33 [multiproc_executor.py:487]                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-10-29T23:29:33.291Z (VllmWorker rank=0 pid=487) ERROR 10-29 23:29:33 [multiproc_executor.py:487]   File "/stage/src/vllm/vllm/model_executor/models/llama.py", line 465, in load_weights
2025-10-29T23:29:33.291Z (VllmWorker rank=0 pid=487) ERROR 10-29 23:29:33 [multiproc_executor.py:487]     param = params_dict[name]
2025-10-29T23:29:33.291Z (VllmWorker rank=0 pid=487) ERROR 10-29 23:29:33 [multiproc_executor.py:487]             ~~~~~~~~~~~^^^^^^
2025-10-29T23:29:33.291Z (VllmWorker rank=0 pid=487) ERROR 10-29 23:29:33 [multiproc_executor.py:487] KeyError: 'layers.60.self_attn.k_norm.weight'

Edit: I also see the same problem with nightly transformers 5.0.0 (which I installed using this), transformers==4.57.1 (current main) and transformers==4.55.4 (listed in the config.json):

git clone https://github.com/huggingface/transformers.git
cd transformers

# install nightly transformers
pip install '.[torch]'

# install other versions
pip install transformers==4.55.4
pip install transformers==4.57.1