Skip to content

2_EMBEDDING_MODEL_SELECTION.md

Latest commit

 

History

History
479 lines (362 loc) · 16.6 KB

2_EMBEDDING_MODEL_SELECTION.md

File metadata and controls

479 lines (362 loc) · 16.6 KB
Raw

Embedding Model Selection and Research

Date: November 23, 2025 Project: KiloCode Codebase Indexing Hardware: AMD 5800X3d, 64GB RAM, RTX 4090 (24GB) + RTX 5070ti (16GB) Environment: Ubuntu Desktop, Docker, Ollama Network

Executive Summary

This document details the research and analysis conducted to select the optimal embedding model for local codebase indexing in KiloCode. After evaluating 9 embedding models across multiple criteria, Qwen3-Embedding-8B emerged as the top choice for code search, offering state-of-the-art performance while remaining cost-effective and privacy-preserving.

Research Findings:

  • Top Model: Qwen3-Embedding-8B-FP16
  • MTEB Ranking: #3 on Multilingual leaderboard (70.58), excellent Code performance (80.68); SOTA for consumer GPUs - top-ranked models require 44GB+ VRAM (as of November 2025)
  • Dimensions: 4096 (model's native output via Ollama)
  • Quantization: FP16 recommended for maximum quality
  • Hardware Requirements: ~15GB VRAM (fits RTX 4090/similar GPUs)

Note: Research showed 1024 dimensions would be sufficient (minimal quality loss), but Qwen3-Embedding-8B-FP16 through Ollama outputs 4096 dimensions natively. For hardware with 16GB+ VRAM, using 4096 provides maximum quality with no configuration needed. See Dimension Analysis for optimization options with hardware constraints.

Table of Contents

  1. Model Evaluation
  2. Final Model Selection
  3. Dimension Analysis
  4. Hardware Compatibility
  5. Cloud vs Local Comparison
  6. Future Enhancements
  7. References

Model Evaluation

Models Considered

Model Parameters Dimensions VRAM (Q8) MTEB Score Key Strengths
qwen3-embedding:8b 8B 1024-4096* ~9GB 70.58 (ML) / 80.68 (Code) SOTA for consumer GPUs, code-optimized
qwen3-embedding:4b 4B 1024-2560* ~4.5GB ~69 (ML) / ~78 (Code) Good balance
qwen3-embedding:0.6b 0.6B 1024* ~0.7GB ~65 (ML) / ~75 (Code) Very efficient
mxbai-embed-large 335M 1024 ~1.5GB 64.68 Beats OpenAI-3-large
nomic-embed-text v1.5 137M 768 ~0.5GB 62.39 Most popular OS model
nomic-embed-text v2-moe ~300M 768 ~1GB ~64 Multilingual MoE
snowflake-arctic-embed2 568M 1024 ~2GB ~63 Strong multilingual
bge-large 335M 1024 ~1.5GB 63.98 Good general purpose
all-minilm 22-33M 384 ~0.1GB ~56 Extremely lightweight

* Supports flexible dimensions via Matryoshka Representation Learning (MRL)

Evaluation Criteria

  1. Performance on Code Tasks (40% weight)

    • MTEB Code Retrieval benchmark
    • Programming language support
    • Code pattern recognition

  2. Model Quality (30% weight)

    • General MTEB ranking
    • Semantic understanding
    • Long context handling

  3. Resource Efficiency (20% weight)

    • VRAM requirements
    • Inference speed
    • Storage footprint

  4. Flexibility (10% weight)

    • Dimension options
    • Instruction-aware capabilities
    • Matryoshka support

Benchmark Comparison

MTEB Code Retrieval Scores (as of November 2025):

Qwen3-Embedding-8B:        80.68 ⭐ Excellent
Qwen3-Embedding-4B:        ~78
Voyage Code 3:             ~77
OpenAI text-embedding-3:   ~75
Qwen3-Embedding-0.6B:      ~75
mxbai-embed-large:         ~72 (general, not code-specific)
nomic-embed-text:          ~70 (general, not code-specific)

MTEB Multilingual Scores (as of November 2025):

KaLM-Embedding-Gemma3-12B: 72.32 (#1, but 44GB VRAM - impractical for consumer GPUs)
llama-embed-nemotron-8b:   69.46 (#2)
Qwen3-Embedding-8B:        70.58 ⭐ #3 (SOTA for consumer GPUs, ~15GB VRAM)
Qwen3-Embedding-4B:        ~69
Google Gemini Embedding:   ~68
BGE-M3:                    ~66
mxbai-embed-large:         64.68
nomic-embed-text v1.5:     62.39

Final Model Selection

Winner: Qwen3-Embedding-8B

Decision Rationale:

  1. SOTA Performance for Consumer GPUs

    • Ranks #3 on MTEB Multilingual leaderboard (70.58); top-ranked models require 44GB+ VRAM, impractical for consumer hardware
    • Excellent Code benchmark performance (80.68)
    • Best accessible model for consumer GPUs (16GB-24GB VRAM)
    • Released June 2025 - most recent training data

  2. Code-Optimized Training

    • Specifically trained on code retrieval tasks
    • Supports 100+ programming languages
    • Excellent cross-lingual code search
    • Strong on syntax and semantic patterns

  3. Hardware Compatibility

    • ~15GB FP16 model fits comfortably in 24GB VRAM
    • Leaves 9GB for simultaneous inference models
    • Can use secondary GPU if needed

  4. Advanced Features

    • Instruction-aware: Custom task prompts (1-5% boost)
    • Matryoshka support: Flexible dimensions (32-4096)
    • Long context: 32K tokens (vs 8K typical)
    • Multilingual: 100+ languages including programming

  5. Future-Proof

    • Latest model architecture
    • Active development (Alibaba)
    • Open-source (Apache 2.0)
    • Growing community adoption

Why Not Others?

mxbai-embed-large:

  • ✅ Good general performance
  • ❌ Not code-optimized
  • ❌ Older (March 2024)
  • ❌ Lower code retrieval scores

nomic-embed-text:

  • ✅ Very efficient (137M params)
  • ✅ Popular and well-tested
  • ❌ Not code-specialized
  • ❌ Smaller dimensions (768 vs 1024)
  • ❌ Lower overall benchmarks

Qwen3-4B/0.6B:

  • ✅ More efficient
  • ❌ Lower quality (~2-5% drop)
  • ❌ Your hardware can easily handle 8B

Dimension Analysis

Understanding Matryoshka Representation Learning (MRL)

Qwen3-Embedding-8B uses MRL, which means:

  • The model outputs 4096 dimensions by default
  • You can truncate to any size (32-4096) without retraining
  • Earlier dimensions contain more important information
  • Quality degrades gracefully as you reduce dimensions

Think of it like nested dolls - earlier dimensions contain the most important information:

First 256 dims:  Core semantic meaning      (noticeable degradation)
First 512 dims:  Rich understanding         (minor degradation)
First 1024 dims: Detailed semantics         (minimal degradation) ⭐
First 2048 dims: Very fine nuances          (near-full quality)
Full 4096 dims:  Maximum detail             (full quality)

Dimension Comparison

Dimension Quality Impact Storage/Vector Qdrant RAM (10K vectors) Use Case
256 Noticeable degradation 1KB ~15MB Mobile, 10M+ vectors, storage-critical
512 Minor degradation 2KB ~30MB Large scale (100K+ files), speed-critical
1024 Minimal degradation 4KB ~60MB Balanced quality/efficiency
2048 Near-full quality 8KB ~120MB Specialized/high-precision needs
4096 Full quality 16KB ~240MB Maximum precision (our choice)

Note: Quality retention varies by model and task. One study (Voyage-3-large) showed only 0.31% quality loss at 1024 vs 2048 dimensions. Matryoshka-trained models like Qwen3 are specifically designed for graceful degradation at lower dimensions.

Understanding Dimension Trade-offs

Performance vs Resources:

  • 1024 dimensions typically provide minimal quality degradation for most use cases
  • Example: Voyage-3-large showed only 0.31% quality loss from 2048→1024 dims
  • Quality gains from higher dimensions may not be noticeable in practice
  • However, maximum quality (4096) is preferred when hardware allows

Storage Requirements:

Typical codebase (10,000 code blocks):

1024 dims:  ~40MB vectors + ~60MB Qdrant = ~100MB total
2048 dims:  ~80MB vectors + ~120MB Qdrant = ~200MB total
4096 dims:  ~160MB vectors + ~240MB Qdrant = ~400MB total

Search Performance: Lower dimensions generally provide faster search, while higher dimensions provide more nuanced semantic understanding. With modern hardware (RTX 4090), the speed difference is negligible even at 4096 dimensions.

Production Considerations:

  • Many implementations use 1024 dims for balance
  • 1024 is sufficient for most code search use cases
  • Code semantics don't require extreme granularity
  • However, if hardware allows, 4096 provides maximum quality with no downsides

Our Implementation Choice: 4096 Dimensions

Why we use 4096 (not 1024):

  1. Model Output: When using Qwen3-Embedding-8B-FP16 through Ollama, it outputs 4096 dimensions (no configuration needed)

  2. Hardware Capacity: Our RTX 4090 (24GB VRAM) easily handles:

    • ~15GB for the model
    • ~240MB RAM for Qdrant (10K blocks at 4096 dims)
    • ~9GB VRAM remaining for other tasks

  3. Maximum Quality: 4096 provides 100% model performance with no quality trade-off

  4. Performance: Despite higher dimensions, search remains fast (milliseconds) with modern hardware

  5. Simplicity: Using the model's output as-is eliminates configuration complexity

Verification:

curl -s http://localhost:11434/api/embeddings -d '{
  "model": "qwen3-embedding:8b-fp16",
  "prompt": "test code snippet"
}' | jq '.embedding | length'
# Output: 4096

Recommendation: Use 4096 if your hardware allows (16GB+ VRAM). Only reduce dimensions if you have hardware constraints or need to optimize for massive scale (100K+ files).

When to Use Different Dimensions

Use 512 dimensions if:

  • Codebase >100K files
  • Storage/memory constrained
  • Speed is critical
  • Quality drop acceptable

Use 2048 dimensions if:

  • Specialized domains (legal/medical) where subtle nuances matter
  • Need between balanced and maximum quality

Use 4096 dimensions if (our choice):

  • Hardware allows (16GB+ VRAM)
  • Want maximum quality
  • Using the model's native output through Ollama (no configuration)
  • Storage/performance acceptable (~400MB for 10K blocks)

For most users: If you have adequate hardware (RTX 4090 or similar), use 4096 for simplicity and maximum quality. If hardware-constrained, 1024 is sufficient.

Hardware Compatibility

Your Hardware Profile

CPU:     AMD Ryzen 7 5800X3D (8C/16T @ 3.4-4.5GHz)
RAM:     64GB DDR4
GPU 0:   RTX 4090 (24GB VRAM) - AI/ML tasks
GPU 1:   RTX 5070ti (16GB VRAM, ~12GB free) - Display + overflow
Storage: Ample SSD space
Network: Docker network (ollama-network)

Resource Requirements

Qwen3-Embedding-8B-FP16:

Model Size:     ~15GB (on disk and in VRAM)
VRAM Usage:     ~15GB (during inference)
RAM Usage:      ~2-4GB (overhead)
Context:        32K tokens
Inference:      GPU-accelerated (RTX 4090)

Qdrant (typical codebase):

Docker Image:   ~200MB
Storage:        ~100-500MB (10K-50K code blocks)
RAM:            ~60-300MB (depends on collection size)
CPU:            1-2 cores (for indexing/search)

VRAM Allocation Strategy

RTX 4090 (24GB Total):
├── Qwen3-Embedding-8B:  ~15GB
├── System overhead:      ~1GB
└── Available:            ~8GB  (for inference models)

RTX 5070ti (16GB Total):
├── Display/OS:           ~4GB
└── Available:            ~12GB (overflow if needed)

Result: Comfortable fit with room for running inference models simultaneously.

Quantization Comparison

Version Model Size VRAM Quality Speed Recommendation
FP16 15GB 15GB 100% Medium ✅ Chosen for quality
Q8_0 8GB 8GB ~99.5% Fast ✅ Alternative if VRAM tight
Q4_K_M 4.7GB 5GB ~96% Fastest ❌ Noticeable quality loss

Decision: Use FP16 for maximum quality since your RTX 4090 has sufficient VRAM.

Cloud vs Local Comparison

Performance Comparison

Metric Local (Qwen3-8B) OpenAI-3-Large Voyage Code 3
Quality (MTEB Code) 80.68 ⭐ ~75 ~77
Quality (MTEB ML) 70.58 ⭐ ~64 ~66
Latency Local (milliseconds) Network-dependent Network-dependent
Data Privacy Complete Sent to OpenAI Sent to Voyage
Offline Work ✅ Yes ❌ No ❌ No
Cost Model Electricity only Per-token charges Per-token charges
Rate Limits None Provider-limited Provider-limited
VRAM Required ~15GB N/A N/A

Trade-offs

Local Advantages:

  • ✅ Excellent quality (Qwen3-8B outperforms common cloud APIs like OpenAI and Voyage on MTEB)
  • ✅ Complete privacy (code never leaves your machine)
  • ✅ No per-query costs (unlimited usage)
  • ✅ No rate limits
  • ✅ Consistent, predictable latency
  • ✅ Works offline

Local Requirements:

  • ⚠️ GPU hardware (16GB+ VRAM recommended)
  • ⚠️ Initial setup complexity
  • ⚠️ Self-managed infrastructure

When Cloud Makes Sense:

  • No GPU hardware available
  • Very sporadic usage (few times per month)
  • Zero maintenance preference
  • Team needs remote access
  • Compliance allows external API calls

Future Enhancements

Potential Improvements

Short-term (next 3-6 months):

  1. Add Reranker: Qwen3-Reranker-8B for improved precision

    • Two-stage: fast retrieval (1024 dims) → precise rerank
    • +2-5% accuracy improvement
    • Minimal latency increase (<50ms)

  2. Dimension Optimization: Test 512 dims for large codebases

    • 2× faster search
    • 2× less storage
    • ~2-3% quality drop (acceptable trade-off)

  3. Quantization: Enable INT8 quantization in Qdrant

    • 4× memory reduction
    • Faster search
    • <1% quality impact

Mid-term (6-12 months):

  1. Hybrid Search: Combine dense + sparse (BM25) retrieval

    • Better keyword + semantic matching
    • Useful for exact identifier search

  2. Multi-Vector: Different models for different file types

    • Qwen3 for code
    • Specialized model for documentation
    • Domain-specific models

  3. Instruction Tuning: Custom instructions per project type

    • "Represent this for web API search"
    • "Represent this for database query code"
    • 1-5% performance boost per domain

Long-term (1+ years):

  1. Distributed Qdrant: Multi-node setup for massive codebases
  2. Model Fine-tuning: Custom Qwen3 fine-tune on your company's code
  3. Active Learning: Feedback loop from search usage to improve results

Upcoming Model Releases

Keep an eye on:

  • Qwen4-Embedding (expected 2026): Next generation
  • Nomic Embed v2 updates: Continuous improvements
  • CodeBERT successors: Microsoft's code-specific models
  • Mistral Codestral Embed updates: Strong code retrieval

Strategy: Re-evaluate models every 6 months via MTEB leaderboard.

References

Official Documentation

Qwen3-Embedding:

Qdrant:

Ollama:

KiloCode:

Benchmarks

MTEB Leaderboard:

Research Papers:

Community Resources

GitHub Projects:

Tutorials:

Document Version: 2.0 Last Updated: November 23, 2025 Purpose: Research Report - Model Selection & Analysis Project: KiloCode Codebase Indexing with Qwen3-Embedding-8B