granite-docling ONNX Conversion Guide

Technical Reproduction Instructions

This document provides complete instructions for reproducing the granite-docling ONNX conversion.

Prerequisites

Python 3.10+
~4GB available RAM
~2GB disk space for conversion environment

Step 1: Environment Setup

# Create isolated environment
python3 -m venv onnx_converter
source onnx_converter/bin/activate  # Linux/Mac
# or onnx_converter\Scripts\activate  # Windows

# Install dependencies
pip install torch torchvision transformers optimum[onnxruntime] safetensors

Step 2: Download Original Model

# Download granite-docling SafeTensors model
mkdir granite-docling-258m
cd granite-docling-258m

curl -L "https://huggingface.co/ibm-granite/granite-docling-258M/resolve/main/model.safetensors" -o model.safetensors
curl -L "https://huggingface.co/ibm-granite/granite-docling-258M/resolve/main/config.json" -o config.json
curl -L "https://huggingface.co/ibm-granite/granite-docling-258M/resolve/main/tokenizer.json" -o tokenizer.json
curl -L "https://huggingface.co/ibm-granite/granite-docling-258M/resolve/main/preprocessor_config.json" -o preprocessor_config.json

Step 3: Install IBM Experimental Fork

# Clone IBM experimental optimum-onnx fork
git clone https://github.com/gabe-l-hart/optimum-onnx.git
cd optimum-onnx
git checkout Idefics3Support

# Install experimental fork
pip install -e . --force-reinstall

Step 4: Convert to ONNX

import os
import torch
os.environ['CUDA_VISIBLE_DEVICES'] = ''  # Force CPU

from pathlib import Path
from transformers import Idefics3ForConditionalGeneration
from optimum.exporters.onnx import export
from optimum.exporters.onnx.model_configs import Idefics3OnnxConfig

# Load model
model = Idefics3ForConditionalGeneration.from_pretrained(
    './granite-docling-258m',
    trust_remote_code=True,
    torch_dtype=torch.float32
).to('cpu')

# Create ONNX config
onnx_config = Idefics3OnnxConfig(model.config, task='image-to-text')

# Export to ONNX
output_path = Path('./granite_docling.onnx')
export(model, onnx_config, output_path, 17)

print(f"ONNX conversion complete: {output_path}")

Expected Output

Initializing Idefics3ModelPatcher
Entering Idefics3ModelPatcher context
Patching Idefics3 model
Using patched position embedding forward
Exiting Idefics3ModelPatcher context
ONNX conversion complete: granite_docling.onnx (1.2GB)

Validation

import onnxruntime as ort

# Test ONNX model loading
session = ort.InferenceSession('granite_docling.onnx')
print("✅ ONNX model loads successfully")

# Check input/output specifications
for inp in session.get_inputs():
    print(f"Input: {inp.name} - {inp.shape}")
for out in session.get_outputs():
    print(f"Output: {out.name} - {out.shape}")

Troubleshooting

Common Issues

"Custom architecture" error: Ensure using IBM experimental fork
Memory errors: Use CPU-only conversion (CUDA_VISIBLE_DEVICES='')
Import errors: Verify experimental fork installed with -e .

Technical Notes

Conversion time: 5-10 minutes on typical CPU
Memory usage: ~4GB RAM during conversion
Warnings: TracerWarnings are expected for complex VLM
File size: ONNX (~~1.2GB) vs SafeTensors (~~492MB) due to graph inclusion

Attribution

Original model: IBM Research granite-docling-258M Conversion method: IBM experimental Idefics3Support optimum-onnx fork Documentation: lamco-development