Update README.md
Browse files
README.md
CHANGED
|
@@ -30,9 +30,9 @@ This is an INT4 quantized version of [SmolLM3-3B](https://huggingface.co/Hugging
|
|
| 30 |
|
| 31 |
## Model Overview
|
| 32 |
|
| 33 |
-
- **Base Model:** SmolLM3-3B
|
| 34 |
- **Quantization:** INT4 via OpenVINO
|
| 35 |
-
- **Size Reduction:**
|
| 36 |
- **Target Hardware:** CPUs, Intel GPUs, NPUs
|
| 37 |
- **Use Cases:** Local inference, edge deployment, resource-constrained environments
|
| 38 |
|
|
@@ -55,10 +55,13 @@ This is an INT4 quantized version of [SmolLM3-3B](https://huggingface.co/Hugging
|
|
| 55 |
|
| 56 |
> ⚠️ **Note:** This is an experimental quantization. Formal benchmarks pending.
|
| 57 |
|
| 58 |
-
Expected
|
| 59 |
-
-
|
| 60 |
-
-
|
| 61 |
-
-
|
|
|
|
|
|
|
|
|
|
| 62 |
|
| 63 |
## 🛠️ How to Use
|
| 64 |
|
|
@@ -108,8 +111,8 @@ text = tokenizer.apply_chat_template(
|
|
| 108 |
## ⚡ Optimization Tips
|
| 109 |
|
| 110 |
1. **CPU Inference:** Use OpenVINO runtime for best performance
|
| 111 |
-
2. **Batch Processing:**
|
| 112 |
-
3. **Memory:**
|
| 113 |
|
| 114 |
## 🧪 Experimental Status
|
| 115 |
|
|
|
|
| 30 |
|
| 31 |
## Model Overview
|
| 32 |
|
| 33 |
+
- **Base Model:** SmolLM3-3B
|
| 34 |
- **Quantization:** INT4 via OpenVINO
|
| 35 |
+
- **Size Reduction:** Significant compression achieved
|
| 36 |
- **Target Hardware:** CPUs, Intel GPUs, NPUs
|
| 37 |
- **Use Cases:** Local inference, edge deployment, resource-constrained environments
|
| 38 |
|
|
|
|
| 55 |
|
| 56 |
> ⚠️ **Note:** This is an experimental quantization. Formal benchmarks pending.
|
| 57 |
|
| 58 |
+
Expected benefits of INT4 quantization:
|
| 59 |
+
- Reduced model size
|
| 60 |
+
- Faster CPU inference
|
| 61 |
+
- Lower memory requirements
|
| 62 |
+
- Some quality trade-off
|
| 63 |
+
|
| 64 |
+
Actual metrics will be added after proper benchmarking.
|
| 65 |
|
| 66 |
## 🛠️ How to Use
|
| 67 |
|
|
|
|
| 111 |
## ⚡ Optimization Tips
|
| 112 |
|
| 113 |
1. **CPU Inference:** Use OpenVINO runtime for best performance
|
| 114 |
+
2. **Batch Processing:** Consider batching requests when possible
|
| 115 |
+
3. **Memory:** INT4 significantly reduces memory requirements
|
| 116 |
|
| 117 |
## 🧪 Experimental Status
|
| 118 |
|