Skip to content

Amirhat/fast-embedding-api

BranchesTags

Repository files navigation

⚡ Fast Embedding API

Python Litestar FastEmbed Docker License

High-performance embedding API with intelligent caching, batch processing, and production-ready Docker deployment

Features Quick Start Documentation Docker API

🌟 Why Fast Embedding API?

  • 🚀 Blazing Fast: 3x faster startup, 5x faster batch processing
  • 🧠 Smart Caching: Intelligent LRU caching with auto-offloading
  • 📦 Production Ready: Docker support, monitoring, security built-in
  • 🔄 Batch Processing: Process multiple texts efficiently in one request
  • 📊 Observable: Built-in metrics, health checks, and structured logging
  • 🔒 Secure: Rate limiting, CORS, input validation, model whitelisting
  • 🐳 Deploy Anywhere: Docker, Kubernetes, AWS, GCP, Azure ready

✨ Features

Core Capabilities

  • ✅ Single & Batch Embeddings - Efficient processing of one or many texts
  • ✅ Intelligent Model Caching - LRU eviction with TTL-based cleanup
  • ✅ Auto-offloading - Automatically unload unused models to save memory
  • ✅ Startup Validation - Pre-validate and warm-up models before serving
  • ✅ Multiple Models - Support for various HuggingFace embedding models
  • ✅ Async Architecture - Non-blocking I/O with thread pool execution

Performance

  • ⚡ 3x Faster Startup - Models cached after validation
  • ⚡ 5x Faster Batch - Batch endpoint vs individual requests
  • ⚡ Optimized Threading - Configurable thread pool workers
  • ⚡ Request Timeouts - Prevent hanging requests
  • ⚡ Processing Metrics - Track response times

Production Features

  • 🔐 Security: Input validation, rate limiting, CORS, model whitelist
  • 📈 Monitoring: Metrics endpoint, health checks, structured logs
  • 🐳 Docker: Multi-stage builds, non-root user, health checks
  • 🔄 High Availability: Nginx load balancing ready
  • 🌐 Cloud Ready: Deploy to AWS, GCP, Azure with examples
  • 📊 Observability: Full metrics, logging, and tracing ready

🚀 Quick Start

Option 1: Docker (Recommended)

# Clone the repository
git clone https://github.com/Amirhat/fast-embedding-api.git
cd fast-embedding

# Deploy with one command
chmod +x scripts/deploy.sh
./scripts/deploy.sh deploy

# Or use docker-compose
docker-compose up -d

Option 2: Local Development

# Create virtual environment
python -m venv venv
source venv/bin/activate  # Windows: venv\Scripts\activate

# Install dependencies
pip install -r requirements.txt

# Start the server
python -m src.main

Test It

# Health check
curl http://localhost:8000/health

# Generate embedding
curl -X POST http://localhost:8000/embed \
  -H "Content-Type: application/json" \
  -d '{"model_name":"BAAI/bge-small-en-v1.5","text":"Hello, world!"}'

# Batch embedding (5x faster!)
curl -X POST http://localhost:8000/embed/batch \
  -H "Content-Type: application/json" \
  -d '{"model_name":"BAAI/bge-small-en-v1.5","texts":["Hello","World","AI"]}'

📚 Documentation

Document Description
📖 Quick Start Guide Get started in 5 minutes
🚀 Production Guide Deploy to production
🐳 Docker Setup Docker configuration
📝 Changelog Version history
📊 Project Status Current status

🔥 API Endpoints

POST /embed - Single Text Embedding

import requests

response = requests.post(
    "http://localhost:8000/embed",
    json={
        "model_name": "BAAI/bge-small-en-v1.5",
        "text": "Your text here"
    }
)
result = response.json()
# Returns: embedding, model_name, dimension, text_length, processing_time_ms

POST /embed/batch - Batch Embeddings (Recommended)

response = requests.post(
    "http://localhost:8000/embed/batch",
    json={
        "model_name": "BAAI/bge-small-en-v1.5",
        "texts": ["Text 1", "Text 2", "Text 3"]
    }
)
result = response.json()
# Returns: embeddings, model_name, dimension, count, processing_time_ms

GET /health - Health Check

curl http://localhost:8000/health
# Returns: status, cache_info, uptime_seconds

GET /metrics - API Metrics

curl http://localhost:8000/metrics
# Returns: total_requests, total_embeddings, cache_info, uptime

GET /models - List Models

curl http://localhost:8000/models
# Returns: required_models, cached_models, allowed_models

GET /models/{model_name} - Model Info

curl http://localhost:8000/models/BAAI/bge-small-en-v1.5
# Returns: model_name, is_cached, load_time, loaded_at, last_used

📊 Performance Benchmarks

Quick Summary

Metric Result Status
Single Embedding 12.62ms avg ✅ <15ms
Batch (10 texts) 4.16ms per text ✅ <5ms
Throughput 95.1 req/s ✅ >80 req/s
Concurrent (10) 140.2 req/s ✅ >100 req/s
Cache Speedup 3.2x faster ✅ >2x
Batch Speedup 3.0x faster ✅ >2.5x
Error Rate 0.00% ✅ <1%

Run Benchmarks

# Python benchmark
python benchmarks/benchmark.py

# K6 load testing
k6 run benchmarks/k6-load-test.js

📈 See BENCHMARKS.md for detailed results

🐳 Docker Deployment

Quick Deploy

# Automated deployment
./scripts/deploy.sh deploy

# View logs
./scripts/deploy.sh logs

# Check status
./scripts/deploy.sh status

Docker Compose

# Production
docker-compose up -d

# Development with hot reload
docker-compose --profile dev up fast-embedding-dev

# Scale to 3 instances
docker-compose up -d --scale fast-embedding=3

Manual Docker

# Build
docker build -t fast-embedding:latest --target production .

# Run
docker run -d \
  --name fast-embedding \
  -p 8000:8000 \
  -e ENABLE_RATE_LIMIT=true \
  -v model_cache:/app/.cache/fastembed \
  fast-embedding:latest

⚙️ Configuration

Environment Variables

# API Settings
HOST=0.0.0.0
PORT=8000
DEBUG=false
LOG_LEVEL=INFO

# Performance
THREAD_POOL_WORKERS=4        # CPU workers
MAX_TEXT_LENGTH=8192         # Max chars
MAX_BATCH_SIZE=32            # Max batch size
REQUEST_TIMEOUT=300          # Timeout (seconds)

# Cache
MODEL_CACHE_TTL=3600         # Cache TTL (seconds)
MAX_CACHED_MODELS=5          # Max cached models
CLEANUP_INTERVAL=60          # Cleanup interval

# Security
ALLOWED_MODELS=model1,model2 # Model whitelist
ENABLE_CORS=true
CORS_ORIGINS=*               # Set specific domains in production!
ENABLE_RATE_LIMIT=true
RATE_LIMIT_REQUESTS=100      # Requests per minute

# Monitoring
ENABLE_METRICS=true

Required Models

Default models validated on startup:

  • BAAI/bge-small-en-v1.5
  • BAAI/bge-base-en-v1.5
  • sentence-transformers/all-MiniLM-L6-v2

Configure in src/config.py or via REQUIRED_MODELS environment variable.

🏗️ Project Structure

fast-embedding/
├── src/                    # Source code
│   ├── __init__.py
│   ├── main.py            # Litestar API
│   ├── model_manager.py   # Model caching
│   └── config.py          # Configuration
│
├── tests/                 # Tests
│   ├── test_api.py        # API tests
│   └── example_client.py  # Usage examples
│
├── benchmarks/            # Performance benchmarks
│   └── benchmark.py       # Benchmark suite
│
├── docs/                  # Documentation
│   ├── QUICKSTART.md
│   ├── PRODUCTION.md
│   ├── DOCKER_SETUP.md
│   ├── CHANGELOG.md
│   └── PROJECT_STATUS.md
│
├── scripts/               # Utility scripts
│   ├── deploy.sh          # Deployment script
│   └── run.sh             # Local run script
│
├── .github/               # GitHub workflows
│   └── workflows/
│
├── Dockerfile             # Multi-stage Docker build
├── docker-compose.yml     # Docker orchestration
├── requirements.txt       # Python dependencies
└── README.md              # This file

🧪 Testing

Our comprehensive test suite includes unit tests, integration tests, and performance tests

Quick Test Commands

# Install test dependencies
make dev-install

# Run all tests
make test

# Run with coverage report
make test-cov

# Run specific test categories
make test-unit          # Unit tests only
make test-integration   # Integration tests only
make test-fast          # Fast mode (minimal output)

Test Structure

tests/
├── conftest.py              # Shared fixtures
├── test_config.py           # Configuration tests (16 tests)
├── test_model_manager.py    # ModelCache tests (25+ tests)
├── test_integration.py      # API endpoint tests (40+ tests)
├── test_api.py             # Legacy tests (requires server)
└── example_client.py        # Example API client

Using Pytest Directly

# All tests
pytest tests/ --ignore=tests/test_api.py -v

# Specific test file
pytest tests/test_config.py -v

# Specific test class
pytest tests/test_integration.py::TestEmbedEndpoint -v

# With coverage
pytest tests/ --ignore=tests/test_api.py --cov=src --cov-report=html

# Stop on first failure
pytest tests/ -x

# Run last failed tests
pytest tests/ --lf

Coverage Report

# Generate HTML coverage report
make test-cov

# Open the report
open htmlcov/index.html  # macOS

Run Examples

# Start server
docker-compose up -d

# Run example client
python tests/example_client.py

For detailed testing documentation, see tests/README.md.

Run Benchmarks Manually

# Run full benchmark suite
python benchmarks/benchmark.py

🔒 Security

Production Security Checklist

  • Input validation (text length, batch size)
  • Model whitelist (ALLOWED_MODELS)
  • Rate limiting (configurable)
  • CORS configuration
  • Request size limits (10MB)
  • Timeout protection
  • Non-root Docker user
  • Authentication (TODO: JWT/API keys)
  • HTTPS (configure reverse proxy)

Security Best Practices

# Set specific CORS origins
CORS_ORIGINS=https://yourdomain.com

# Enable rate limiting
ENABLE_RATE_LIMIT=true

# Use model whitelist
ALLOWED_MODELS=BAAI/bge-small-en-v1.5,BAAI/bge-base-en-v1.5

# Always use HTTPS in production (nginx/reverse proxy)

🌐 Cloud Deployment

AWS ECS/Fargate

# Push to ECR
docker tag fast-embedding:latest <account>.dkr.ecr.<region>.amazonaws.com/fast-embedding
docker push <account>.dkr.ecr.<region>.amazonaws.com/fast-embedding

Google Cloud Run

gcloud builds submit --tag gcr.io/PROJECT-ID/fast-embedding
gcloud run deploy --image gcr.io/PROJECT-ID/fast-embedding --memory 4Gi

Azure Container Instances

az acr build --registry myregistry --image fast-embedding:latest .
az container create --resource-group rg --name fast-embedding \
  --image myregistry.azurecr.io/fast-embedding --cpu 4 --memory 4

See PRODUCTION.md for detailed deployment guides.

📈 Monitoring

Built-in Metrics

# Health check
curl http://localhost:8000/health

# Metrics
curl http://localhost:8000/metrics

Metrics Include

  • Total requests processed
  • Total embeddings generated
  • Cache statistics (hits, misses, size)
  • API uptime
  • Model load times
  • Processing times

Integration Ready

  • Prometheus metrics export (coming soon)
  • ELK Stack logging
  • CloudWatch integration
  • Custom dashboards

🤝 Contributing

Contributions are welcome! Please feel free to submit a Pull Request.

  1. Fork the repository
  2. Create your feature branch (git checkout -b feature/AmazingFeature)
  3. Commit your changes (git commit -m 'Add some AmazingFeature')
  4. Push to the branch (git push origin feature/AmazingFeature)
  5. Open a Pull Request

📄 License

This project is licensed under the MIT License - see the LICENSE file for details.

🙏 Acknowledgments

  • Litestar - Modern, performant web framework
  • FastEmbed - Lightweight embedding library
  • Qdrant - Vector database and FastEmbed creators

📞 Support

⭐ Star this repo if you find it useful!

Made with ❤️ by the Fast Embedding Team

About

High-performance embedding API with intelligent caching, batch processing, and production-ready Docker deployment

Resources

Readme
MIT license

Contributing

Contributing
Activity

Stars

1 star

Watchers

0 watching

Forks

0 forks
Report repository

Releases

Packages

Contributors

Languages