Vector databases for RAG: choosing and running a store in Australia

Most retrieval-augmented generation projects don’t fail on the model. They fail on the plumbing underneath it — and the vector database is the piece teams choose fastest and regret longest. Pick the wrong store, or run a good one badly, and your “AI answers” quietly drift into confidently wrong ones.

The summary you can act on: a vector database is a swappable component, not a strategic bet. Choose it on four boring criteria — data residency, hybrid search quality, metadata filtering, and total operating cost — and budget as much for keeping it fresh as for standing it up. The re-indexing and monitoring work is where the money and the risk actually live.

This is the layer we spend a lot of time on in vector database consulting engagements, precisely because it’s where sensible-looking early decisions turn into expensive rework.

What a vector database actually does in RAG

In a RAG system, your documents are chopped into chunks, each chunk is passed through an embedding model, and the model returns a vector — a long list of numbers that encodes the chunk’s meaning. The vector database stores those vectors and, at query time, finds the ones closest to the vector of the user’s question. Those nearest chunks become the context you hand to the language model.

The core job is approximate nearest-neighbour search: finding the most semantically similar vectors across millions of records in milliseconds. “Approximate” matters. Exact nearest-neighbour search over large collections is too slow for interactive use, so these systems use index structures (HNSW graphs and similar) that trade a small, tunable amount of accuracy for a large speed gain.

If you want the wider picture of where retrieval fits against other techniques, our RAG vs fine-tuning vs prompting piece frames the trade-offs. This article assumes you’ve decided retrieval is right and now need the store to support it.

Selection criteria that actually separate the options

The market is crowded and most tools demo identically. The differences show up under load and under audit, not in the quickstart. Here’s what to weigh.

Scale and latency

Be honest about your numbers before you shortlist anything:

• Vector count. A hundred thousand chunks is a different problem from a hundred million. Below a few million vectors, almost any option works and you’re choosing on other factors. Above that, index build time, memory footprint and sharding behaviour start to dominate.
• Query latency budget. Interactive chat needs retrieval in tens of milliseconds so the end-to-end response stays snappy. Batch or back-office use cases can tolerate far more, which widens your options and cuts cost.
• Query concurrency. Peak simultaneous queries drives how much you provision. A tool that’s fast for one user can fall over at fifty.

Index type is the lever here. Graph-based indexes give excellent recall and speed but hold vectors in memory, which sets your cost floor. Disk-backed indexes trade some latency for dramatically cheaper storage at scale.

Hybrid search: semantic plus keyword

This is the criterion teams most often overlook and most often come back to fix. Pure semantic search is excellent at meaning but weak at exact terms — product codes, policy numbers, section references, surnames, acronyms. Ask for “clause 14.2” and a purely semantic system may hand you clauses that are about the same topic while missing the exact one.

Hybrid search runs a traditional keyword search (the same lexical matching that powers search engines) alongside the vector search and blends the results. For most enterprise corpora — contracts, policies, technical manuals, case files — hybrid materially beats either method alone. Treat first-class hybrid support as a hard requirement, not a nice-to-have, and test it on your own documents with real queries your users would type.

Metadata filtering

Your chunks carry attributes: source system, document date, security classification, business unit, jurisdiction. You will constantly need to retrieve “only from documents this user is allowed to see” or “only from the current policy version”. That’s metadata filtering applied at query time.

The subtlety is how the store combines filtering with vector search. Naive implementations filter after finding nearest neighbours, so a restrictive filter can leave you with too few results — or force the engine to scan far more than expected. Good implementations filter efficiently during the search. If access control or recency matters to you (it almost always does, and for regulated clients it’s non-negotiable), stress-test filtering with genuinely selective filters, not toy examples.

Cost, honestly totalled

Managed vector databases usually price on some mix of stored vectors, dimensions, query volume and provisioned compute. The headline number is rarely the real one. Total cost includes:

• Storage that grows with every re-indexing and every new data source.
• Query compute that scales with traffic.
• The embedding-model calls to create and refresh vectors, which are easy to forget and can exceed the database bill.
• Engineering time to operate the thing — the largest line item for self-hosted setups.

We walk through this kind of full-picture costing in our broader RAG consulting work, because the sticker price and the run-rate are often a factor of several apart.

Hosting and Australian data residency

For AU organisations this is frequently the deciding factor, so decide it early rather than retrofitting it.

The question is not only “where does the vendor say it runs?” but “where do my vectors and my source text physically live, and who can access them?” Two points teams miss:

• Vectors are derived from your content, not a safe abstraction of it. They are not human-readable, but with the embedding model they can leak information about the source. Treat the vector store with the same data-classification discipline as the documents it was built from.
• The database and the embedding model are separate residency decisions. You can host the store in an Australian region and still be shipping every chunk offshore to an embedding API. Map the whole path.

Your realistic options are a managed service that offers Australian hosting regions, a managed service inside your own cloud tenancy in an Australian region, or fully self-hosted on infrastructure you control. For organisations under APRA CPS 230 and CPS 234, or handling personal information under the Privacy Act and the Australian Privacy Principles, or bound by government data-handling rules, residency and access control usually push toward self-hosted or in-tenancy managed options. Whatever you choose, document the data flow so it survives an audit and a vendor review.

Managed versus self-hosted

The trade-off is the familiar one, sharpened by how operationally demanding these systems are.

Managed gets you running in an afternoon, handles scaling and upgrades, and is the right default for pilots and for teams without dedicated platform engineers. You pay a premium and you accept the vendor’s residency and access model.

Self-hosted gives you full control over where data lives and who touches it, and can be cheaper at large, steady scale. You take on the operational burden — and with vector databases that burden is real: index tuning, memory management, backups, version upgrades, and capacity planning as your corpus grows.

A pattern that works well: prove the use case on a managed store, then reassess hosting once you know your true scale and residency constraints. Our Generative AI Pilot is built to answer exactly these questions with real data before you commit to a platform.

What running one in production actually takes

Standing up a vector database is a day. Running it is the job. Three realities catch teams out.

Indexing is a pipeline, not a one-off. Chunking strategy, embedding model choice and index parameters all affect retrieval quality, and you’ll tune them iteratively. Building an index over a large corpus takes real time and compute — plan for it.

Re-indexing on data change is the hidden ongoing cost. Your source content changes: contracts get amended, policies get superseded, records get added daily. Every change means re-chunking and re-embedding the affected content and updating the store. You need a repeatable pipeline that keeps the index in sync — and a firm answer to stale data, because a RAG system confidently citing a superseded policy is worse than no system at all. If you ever change embedding models, you must re-embed everything, which is effectively a full rebuild.

Monitoring is different from monitoring a normal database. Uptime and latency are necessary but not sufficient. You also need to watch retrieval quality — are the returned chunks actually relevant? Quality degrades silently as content drifts away from what the index holds, and no infrastructure alert will tell you. Build an evaluation set of representative questions with known-good answers and run it regularly. That evaluation harness is the single most valuable asset for keeping a RAG system trustworthy over time.

Where to start

Choose the store on residency, hybrid search, filtering and total cost — in that order for most AU organisations — and treat the re-indexing pipeline and quality monitoring as first-class parts of the build, not afterthoughts. Get those right and the database itself becomes the boring, swappable component it should be.

If you’re weighing options, our generative AI services team does this work vendor-neutrally, with your residency and audit obligations front of mind. Get in touch and we’ll help you choose a store you won’t have to rip out in a year.