|
|
--- |
|
|
tags: |
|
|
- vector-database |
|
|
- benchmarks |
|
|
- faiss |
|
|
- weaviate |
|
|
- chroma |
|
|
- multimodal |
|
|
- clip |
|
|
- retrieval |
|
|
license: apache-2.0 |
|
|
--- |
|
|
|
|
|
# Vector Database Benchmarks: FAISS vs Chroma vs Weaviate |
|
|
|
|
|
This repository contains experiments benchmarking popular vector databases on **multimodal embeddings** generated from the [Flickr8k dataset](https://huggingface.co/datasets/jxie/flickr8k). |
|
|
We focused on four key evaluation dimensions: |
|
|
|
|
|
1. **Latency per query** |
|
|
2. **Recall@5 vs Flat (accuracy tradeoffs)** |
|
|
3. **Queries per second (QPS throughput)** |
|
|
4. **Ingestion scaling performance** |
|
|
|
|
|
All experiments were run on **Google Colab** (T4 GPU for embedding generation, CPU backend for databases). |
|
|
|
|
|
--- |
|
|
|
|
|
## Methodology |
|
|
|
|
|
- Dataset: 6k images and 30k captions from Flickr8k. |
|
|
- Embeddings: CLIP (OpenAI ViT-B/32). |
|
|
- Workload: Caption-to-image retrieval (cross-modal). |
|
|
- Baseline: FAISS Flat index used as the ground-truth for recall calculations. |
|
|
|
|
|
Each vector database was tested under the same conditions for ingestion, search, and recall. |
|
|
|
|
|
--- |
|
|
|
|
|
## Results Summary |
|
|
|
|
|
| Metric | FAISS | Chroma | Weaviate | |
|
|
|--------------------------|------------------|------------------|------------------| |
|
|
| **Avg Latency per Query** | 0.19 ms | 0.76 ms | 1.82 ms | |
|
|
| **Recall@5 (Flat Baseline)** | 1.00 | 0.002 | 0.918 | |
|
|
| **QPS Throughput** | 1929.94 | 719.01 | 598.40 | |
|
|
| **Ingestion Scaling (20k)** | 0.024s | 2.806s | 4.000s | |
|
|
|
|
|
|
|
|
 |
|
|
|
|
|
--- |
|
|
|
|
|
## Key Takeaways |
|
|
|
|
|
- **FAISS** is fastest, leveraging in-memory array ingestion and customizable indexing strategies. |
|
|
- **Chroma** offers simplicity and ease of integration but struggles at scale due to batching and internal constraints. |
|
|
- **Weaviate** provides a more feature-rich ecosystem (schema, hybrid search, persistence) but at higher ingestion and query overhead. |
|
|
|
|
|
At the million-vector scale, speed alone will not decide your choice; **engineering tradeoffs, developer productivity, and system features** will. |
|
|
Benchmarks tell one part of the story, your use case tells the rest. |
|
|
|
|
|
--- |
|
|
|
|
|
## Usage |
|
|
|
|
|
You can reproduce these experiments using the provided notebook and Hugging Face dataset. |
|
|
See full code here: [rag-experiments/VectorDB-Benchmarks](https://huggingface.co/rag-experiments/VectorDB-Benchmarks). |
|
|
Dataset used: Flickr8k (train split — 6k images, 30k captions, multimodal — images and text), CLIP Embeddings. Dataset Author: Johnathan Xie |
|
|
|
|
|
--- |
|
|
|
|
|
## Citation |
|
|
|
|
|
If you find this useful, please cite this repository: |
|
|
|
|
|
|
