Vector Database

• Scale: billions of vectors on a single node with SSD disk-based indexes
• Index types: HNSW (best recall) vs IVF (cheaper memory) vs DiskANN (huge scale)
• Hybrid search: dense vector + sparse BM25 + metadata filter
• Cloud managed: Pinecone, Qdrant Cloud, Weaviate Cloud, Zilliz (Milvus)
• Self-host: Qdrant / Weaviate / Milvus Docker image, pgvector extension in Postgres 14+

# Qdrant: create collection + insert + search
curl -X PUT http://localhost:6333/collections/docs \
  -H 'Content-Type: application/json' \
  -d '{"vectors": {"size": 1536, "distance": "Cosine"}}'

# Insert
curl -X PUT http://localhost:6333/collections/docs/points \
  -d '{"points": [{"id": 1, "vector": [0.1, 0.2, ...], "payload": {"text": "..."}}]}'

# Search top-5
curl -X POST http://localhost:6333/collections/docs/points/search \
  -d '{"vector": [0.1, 0.2, ...], "limit": 5}'

• Embeddings
• HNSW
• Semantic Search

A vector database is a specialized database designed to store, index, and manage high-dimensional vectors, often used in machine learning and AI applications for similarity search and retrieval tasks. Unlike traditional databases that rely on structured queries, vector databases utilize vector embeddings to enable efficient searches through large datasets, making them essential for applications like recommendation systems, image retrieval, and natural language processing.

A vector database is a system optimized for handling high-dimensional vector data, which represents information in a format suitable for machine learning algorithms. Each vector is a point in a multi-dimensional space, and the distance between these points can signify similarity. Vector databases leverage this spatial representation to facilitate operations like nearest neighbor search, clustering, and classification.

Typically, vectors are generated via techniques such as:

• Word Embeddings: Techniques like Word2Vec or GloVe transform words into vector representations based on their context and usage.
• Image Feature Extraction: Convolutional Neural Networks (CNNs) can convert images into vector representations, enabling image similarity searches.
• Custom Models: Organizations can create vectors tailored to specific applications, using techniques such as autoencoders or deep learning models.

For instance, a vector representing the word 'apple' in a word embedding model may be close to vectors of similar fruits like 'banana' and 'orange,' while being distant from unrelated terms like 'car' or 'house.'

Implementing a vector database can enhance the performance of applications that require rapid similarity searches. Below is a step-by-step guide to deploying a vector database using Milvus, an open-source vector database solution.

1. Install Milvus: You can run Milvus using Docker. Execute the following command:

docker run -d --name milvus_db -p 19530:19530 milvusdb/milvus:v2.0.0-20211116-3d8f9f

1. Create a Collection: After installation, connect to the Milvus service and create a collection to store vectors. Use the following Python code:

from pymilvus import connections, CollectionSchema, FieldSchema, DataType, Collection

connections.connect(host='localhost', port='19530')

fields = [
    FieldSchema(name='embedding', dtype=DataType.FLOAT_VECTOR, dim=128),
    FieldSchema(name='id', dtype=DataType.INT64, is_primary=True)
]

schema = CollectionSchema(fields=fields, description='Example collection')
collection = Collection(name='example_collection', schema=schema)

1. Insert Vectors: You can now insert vectors into the collection. Here's an example of adding random vectors:

import numpy as np

num_vectors = 1000
vectors = np.random.rand(num_vectors, 128).tolist()
ids = [i for i in range(num_vectors)]

collection.insert([ids, vectors])

1. Search for Similar Vectors: To find vectors similar to a given input, you can execute a search query:

search_vector = np.random.rand(1, 128).tolist()
search_results = collection.search(search_vector, 'embedding', param={'nprobe': 10}, limit=5)

This query will return the top 5 vectors most similar to the generated search vector. The nprobe parameter controls the number of partitions to search, balancing search speed and accuracy.

In summary, vector databases like Milvus provide essential tools for developers looking to implement AI-driven applications that require efficient handling of high-dimensional data.