Hugging Face's logo

deepseek-ai
/
DeepSeek-OCR

Image-Text-to-Text
Safetensors
multilingual
deepseek_vl_v2
deepseek
vision-language
ocr
custom_code
conversational
Model card Files
xet
Community
60

šŸ§  Why does DeepSeek-OCR not use Multi-Head Latent Attention (MLA)?

#53
by ZoneTwelve - opened
Discussion
ZoneTwelve

Hi DeepSeek team šŸ‘‹,

First of all, thank you for releasing DeepSeek-OCR ā€” itā€™s an impressive and elegant vision-to-text model.

While exploring the model architecture and configuration files (config.json), I noticed that Multi-Head Latent Attention (MLA) is default enabled in this OCR model.

Questions

Could you please share some insights into why MLA was not used in DeepSeek-OCR?

  • Was it due to compatibility issues between MLA and the vision encoderā€“decoder pipeline?
  • Or did MLA not provide practical benefits in the OCR setting (e.g., shorter sequence lengths or the main bottleneck lying elsewhere)?
  • Is there any plan to integrate MLA into future versions of DeepSeek-OCR to improve inference efficiency?

Iā€™m asking because MLA has demonstrated significant efficiency gains in your other models (e.g., DeepSeek-V2/V3), and Iā€™m curious about the reasoning behind excluding it here.

Thanks again for your excellent work and for open-sourcing this project! šŸ™

ZoneTwelve changed discussion title from Why does DeepSeek-OCR not use Multi-Head Latent Attention (MLA)? to šŸ§  Why does DeepSeek-OCR not use Multi-Head Latent Attention (MLA)?
HaoranWei
 DeepSeek org

Hello,
We actually have an internal MLA-enabled version of DeepSeek-OCR.
The only reason it hasnā€™t been open-sourced yet is simply that I havenā€™t had the bandwidth to implement the code needed to convert the internal weights into the Hugging Face format.
Best regards

Excel001

Hi

Sign up or log in to comment