Vendor demonstrations of vector search tend to feature the same reassuring ingredients: fast responses, flexible indexing, sophisticated filtering, and deployment options for every scale. A procurement decision needs more demanding evidence. Choosing a vector database should begin with a representative dataset and a collection of questions that the eventual users actually ask.
Some teams can store embeddings in an existing database or retrieve the right documents with a conventional search engine. Others need specialized similarity indexes, hybrid retrieval, multitenant filtering, or unusually high throughput. The workload should determine how much new infrastructure is warranted.
Start with a test collection and real questions
Gather a representative set of documents or records and examples of what users are likely to ask. Include straightforward searches, awkward phrasing, exact identifiers, uncommon vocabulary, stale documents, and questions with no valid result. Mark the answers people should reasonably find. That creates a basis for evaluating relevance, rather than judging a system by a demonstration assembled by its seller.
For nearest-neighbor search, teams commonly track measures such as recall at a chosen result count: how often the returned candidates include the items a trusted reference method considers nearest. That is not identical to human relevance. A mathematically close passage can still be misleading, inaccessible to a particular user, or too old to answer the question.
Measure the whole retrieval path
A vector engine has to store embeddings and return candidates, but a useful application may also require metadata filters, keyword matching, ranking, document updates, and secure access. A product can excel at unfiltered similarity search and struggle when the question must be restricted to one customer’s documents. Compare end-to-end response times, not just a vendor’s index benchmark measured in isolation.
Run tests with your expected dataset size, concurrent traffic, and update rate. Record latency distributions rather than a single best-case time. Ask how soon a newly changed document appears in results and what happens during an index rebuild or service failure. Estimate storage and query costs with the workload you anticipate, including the cost of generating embeddings and moving data between services.
Questions to ask every vendor
Relevance: Does the system support the similarity measure your embedding model requires? Can it combine keyword and vector retrieval, and can you tune the ranking? Filters: Are metadata and permissions applied before results are returned? Operations: How are backups, restore tests, updates, monitoring, and outages handled? Portability: Can you export your vectors and metadata in usable formats? Commercials: What drives recurring charges, and what happens when usage rises sharply?
The answers should be demonstrated or documented. A feature marked “supported” can still have important limitations, especially around filters and concurrency. Weaviate’s documentation, for example, discusses differences between pre-filtering and post-filtering because the order of filtering can affect the usefulness and number of results. Such details matter more than a vague claim of intelligent search.
What a useful vendor comparison looks like
Operating a small document assistant is a different task from serving billions of recommendations. A team with strict data-residency requirements may reject an otherwise excellent managed service, while another team values the freedom from server maintenance. A shortlist is useful only if its evaluation criteria reflect those differences.
Use the SI Database directory to find candidates such as Pinecone, Weaviate, and Milvus. Then demand evidence from each one using the same dataset, comparable settings, and an evaluation protocol agreed upon before the tests begin. If a trial cannot reproduce the result shown in a sales demonstration, the discrepancy deserves investigation.