Vector Database

A vector database is a collection of data stored as mathematical representations known as vectors. It indexes and stores vector embeddings for fast retrieval and similarity search, with capabilities like CRUD operations, metadata filtering, horizontal scaling, and serverless.

A vector is an array of numerical values that expresses the location of a floating point along several dimensions, like: {12, 13, 19, 8, 9}

Embeddings are vectors created by machine learning models for the purpose of capturing meaningful data about each object

When one visits a shoe store, a salesperson may suggest shoes that are similar to the pair one prefers. Likewise, when shopping in an ecommerce store, the store may suggest similar items under a header like “Customers also bought…” Vector databases enable machine learning models to identify similar objects, just as the salesperson can find comparable shoes and the ecommerce store can suggest related products.

The diagram below gives us a better understanding of the role of vector databases in this type of application:

Let’s break this down:

1. First, we use the embedding model to create vector embeddings for the content we want to index.
2. The vector embedding is inserted into the vector database, with some reference to the original content the embedding was created from.
3. When the application issues a query, we use the same embedding model to create embeddings for the query and use those embeddings to query the database for similar vector embeddings.

How does a vector database work?

A vector database uses a combination of different algorithms that all participate in Approximate Nearest Neighbour (ANN) search. These algorithms optimize the search through hashing, quantization, or graph-based search.

These algorithms are assembled into a pipeline that provides fast and accurate retrieval of the neighbours of a queried vector. Since the vector database provides approximate results, the main trade-offs we consider are between accuracy and speed. The more accurate the result, the slower the query will be. However, a good system can provide ultra-fast search with near-perfect accuracy.

1. Indexing: The vector database indexes vectors using an algorithm such as PQ, LSH, or HNSW . This step maps the vectors to a data structure that will enable faster searching.
2. Querying: The vector database compares the indexed query vector to the indexed vectors in the dataset to find the nearest neighbours.
3. Post Processing: In some cases, the vector database retrieves the final nearest neighbours from the dataset and post-processes them to return the final results. This step can include re-ranking the nearest neighbours using a different similarity measure.

Conclusion:

Vector databases are purpose-built databases that are specialized to tackle the problems that arise when managing vector embeddings in production scenarios. For that reason, they offer significant advantages over traditional scalar-based databases and standalone vector indexes.