RuntimeError: No GPU or XPU found. A GPU or XPU is needed for FP8 quantization.
hi please help me with this error on my win11pro machine (a 4090 and some 64GB ddr5ram)
(2 errors: "You are using a model of type kimi...", and "...No GPU or XPU found...") below
from transformers import AutoConfig, AutoModel
model_name = "C:\mystuff\LLM\Models\Kimi-K2-Instruct-0905"
model = AutoModel.from_pretrained(model_name, trust_remote_code=True)
You are using a model of type kimi_k2 to instantiate a model of type deepseek_v3. This is not supported for all configurations of models and can yield errors.
Traceback (most recent call last):
File "", line 1, in
File "c:\mystuff\LLM\Models.venv_py312\Lib\site-packages\transformers\models\auto\auto_factory.py", line 597, in from_pretrained
return model_class.from_pretrained(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "c:\mystuff\LLM\Models.venv_py312\Lib\site-packages\transformers\modeling_utils.py", line 288, in _wrapper
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "c:\mystuff\LLM\Models.venv_py312\Lib\site-packages\transformers\modeling_utils.py", line 5008, in from_pretrained
hf_quantizer, config, dtype, device_map = get_hf_quantizer(
^^^^^^^^^^^^^^^^^
File "c:\mystuff\LLM\Models.venv_py312\Lib\site-packages\transformers\quantizers\auto.py", line 319, in get_hf_quantizer
hf_quantizer.validate_environment(
File "c:\mystuff\LLM\Models.venv_py312\Lib\site-packages\transformers\quantizers\quantizer_finegrained_fp8.py", line 48, in validate_environment
raise RuntimeError("No GPU or XPU found. A GPU or XPU is needed for FP8 quantization.")
RuntimeError: No GPU or XPU found. A GPU or XPU is needed for FP8 quantization.
solved:
pytorch needs to be installed with CUDA version..
also transformers is not directly supported in python.exe...