unsloth

/

Phi-4-mini-instruct

like 21

Follow

Unsloth AI 10.3k

seems like the modelling_phi3.py is not included

The Phi-4 mini release is very promising, but sadly it cannot load inside of the Unsloth framework: "RuntimeError: rope_scaling's short_factor field must have length 64, got 48".

Will unsloth release a fixed version possibly?

seems like the modelling_phi3.py is not included

unfortunately doesnt work atm in any framework. doesnt work in unsloth, ollama, llama.cpp etc because of new arch

will update u guys when it does

The architecture isn’t even particularly new, just none of these frameworks respect the “partial_rotary_factor” config line. (Only 3/4 of the embeddings are subject to RoPE, presumably to weight latest context more heavily than long context). I took a crack at adding it to Exllamav2, and while quantization now appears to work inference is wildly broken. Guess it’ll take a while before we see this in a usable state if all the upstream packages need to be updated to support it.

Hello @shimmyshimmer

We already added this 'partial_rotary_factor' support to the latest HF and VLLM before the release.
The new model feature is added to the latest HF(v4.49.0) and vllm (v0.7.3) already.

Can you take a look at the PRs?
They are relatively simple if the new config is utilized.

VLLM: https://github.com/vllm-project/vllm/pull/12718
HF: https://github.com/huggingface/transformers/pull/35947

Hello @shimmyshimmer

We already added this 'partial_rotary_factor' support to the latest HF and VLLM before the release.
The new model feature is added to the latest HF(v4.49.0) and vllm (v0.7.3) already.

Can you take a look at the PRs?
They are relatively simple if the new config is utilized.

VLLM: https://github.com/vllm-project/vllm/pull/12718
HF: https://github.com/huggingface/transformers/pull/35947

Can you guys prepare a complete fine-tuning solution for general users? I have tried a lot of methods and nothing works.
RuntimeError: rope_scaling's short_factor field must have length 64, got 48

I was getting this issue today on VLLM 0.8.3 serving Phi-4

Because my "vllm --version" reported 0.8.3 (which is more than 0.7.3, meaning my vLLM should be fine), I did "pip install transformers==0.49.0" as ykin362 suggested (although presented as a git pull) and now it works fine.

I had old transformers version 4.48.2 but that was not new enough. 0.49.0 (or higher) is required.

Coming back to this, we were able to fine tune phi-4-mini by doing the following:
(full fine tuning and Lora also works)
After installing unsloth, if you are on Google colab notebook or any other jupyter notebook, make your second cell after your pip installs:

import os
os.environ["TORCH_DYNAMO_DISABLE"] = "1"
os.environ["UNSLOTH_COMPILE_DISABLE"] = "1"

And to start fine tuning
import torch
import torch.nn as nn
torch._dynamo.config.disable = True
trainer_stats = trainer.train()

These steps did allow us to fine-tune phi-4-mini models. keep in mind it does use quite a bit of memory but the loss is good

Coming back to this, we were able to fine tune phi-4-mini by doing the following:
(full fine tuning and Lora also works)
After installing unsloth, if you are on Google colab notebook or any other jupyter notebook, make your second cell after your pip installs:

import os
os.environ["TORCH_DYNAMO_DISABLE"] = "1"
os.environ["UNSLOTH_COMPILE_DISABLE"] = "1"

And to start fine tuning
import torch
import torch.nn as nn
torch._dynamo.config.disable = True
trainer_stats = trainer.train()

These steps did allow us to fine-tune phi-4-mini models. keep in mind it does use quite a bit of memory but the loss is good

Thanks so much for the input, I'm sure people will find this useful! If there was only some way to pin this so others could know