Put your vectors in the Postgres you already run
A retrieval prototype usually keeps its vectors in memory. The question is what it costs to move them somewhere that survives a restart, and whether you need a vector index once you do. I measured both.
Adapted from Chapter 6, “Vector Storage: An Index That Outlives the Process”, of AI Engineering, Measured.
The price of moving out of memory
Across five chunk configurations, an in-memory numpy dot product took a median of 0.0043 to 0.0089 ms per query. The same questions against the same vectors in Postgres with pgvector and no index took 1.3037 to 1.5437 ms. That's 151x to 303x slower.
It looks worse than it is. The Postgres figure barely moves between a 10-row and a 44-row table, while the numpy figure roughly doubles across the same range. What's being measured is round-trip cost, not search cost. The query isn't the expensive part; the socket is.
So the trade is about a millisecond and a half per retrieval, against an index that survives a restart, that a second process can read, and that gets backed up on the schedule you already have. Next to a model call of several seconds, that's an easy trade. And it changed no ranking, on any configuration or any question.
The index nobody used
The planner declined to use a vector index on every corpus in the book: 10, 20, 30, 30 and 44 rows, with both ivfflat and hnsw. Forced, the query takes an index scan, so the index was valid. Postgres had simply decided that reading 44 rows was cheaper than consulting a structure about them. It started choosing an index somewhere between 1,000 and 10,000 rows.
The index wasn't free while it declined to help. On one configuration it was 32,768 bytes over a 16,384-byte table: twice the size of the data it indexed.
Approximate means approximate
At small scale, an approximate index wasn't stable across builds. Ten rebuilds with identical settings gave ten orderings, and recall@3 from 2/7 to 5/7 against an exact 4/7. Two settings returned the exact ordering every time, and finding out which took a gold set.
What I'd do with this
- Put the vectors in the database you already run. The reason is that the index outlives the process, not that it searches faster.
- Don't add a vector index until the planner will use one. Below that point it costs build time and disk and returns nothing.
- Keep a gold set. It's the only way to tell an approximate index's reordering from a real regression.
About these numbers. Small corpora, 10 to 44 rows, from the book's own test set. Your crossover point depends on your data and hardware; measure it.
The full chapter, with the code, the diagram and a hands-on exercise, is in AI Engineering, Measured. Building something like this and want a second pair of eyes? Book a 1:1 session.