Vector Search in MySQL vs MariaDB: Look Beyond QPS

A few days ago, I came across an interesting post by Hussain Patel on LinkedIn about HNSW index implementation differences in every vector databases. I was very surprised not to see MariaDB in the list.

One day later, AskDBA (Alkin Tezuysal) published a benchmark article comparing vector search using MyVector on MySQL with the native vector capabilities available in MariaDB. Thank you for that.

I always like independent benchmarks. They usually teach us much more than benchmarks produced by the vendors themselves, and this one is no exception, at least for the MariaDB side.

At first sight, the result seems quite simple: MyVector reaches around 548 queries per second while MariaDB reaches around 257.

So, case closed? MySQL is twice as fast? Not really.

When benchmarking approximate nearest-neighbour search, queries per second alone don’t tell us very much. You also need to look at what the search actually returns — in other words, the recall.

What is recall in vector search?
Recall measures how many of the true nearest matches were actually returned by the approximate vector search.
For example, with recall@10, we compare the 10 results returned by the vector index with the real 10 nearest vectors obtained by an exact search.
A recall of 0.90 means that, on average, about 90% of the correct nearest neighbours were found.

And this is where the benchmark becomes much more interesting for MariaDB.

MyVector reaches roughly 0.82 recall@10, while MariaDB reaches around 0.95. Comparing those two QPS figures directly isn’t an apples-to-apples comparison. To make it apples-to-apples, MariaDB should be allowed to drop the recall@10 to roughly 0.82, and then it delivers 1196 QPS. Very different story.

Configuration changes the picture

The first MariaDB result in the benchmark was not very flattering: around 29 QPS.

But this turned out not to be representative of what MariaDB Vector could actually do.

The HNSW index used for the test was around 27 MB, while MariaDB’s mhnsw_max_cache_size was still set to its default value of 16 MB. The complete vector index therefore did not fit in the configured HNSW cache.

Once the cache was increased, the result changed dramatically: throughput jumped to around 257 QPS, while recall reached roughly 0.95. The index creation time also dropped from about 583 seconds to 92 seconds.

I really appreciate that Alkin investigated the unexpected result instead of stopping at the first number. Publishing only the initial 29 QPS result would have given a completely different impression. Thank you.

But another interesting aspect comes up here.

MyVector currently loads its HNSW index into memory. The benchmark article itself points this out, explaining that MyVector “sidesteps this by loading its entire index into memory.” MariaDB, on the other hand, exposes an explicit HNSW cache size through mhnsw_max_cache_size and can still operate when the complete vector index does not fit in that configured cache, although with an obvious performance impact.

This distinction probably does not matter much with 50,000 vectors, but it becomes much more interesting when we start talking about millions, tens of millions, or hundreds of millions of vectors.

With MyVector, the available memory effectively becomes an important constraint on the size of the active HNSW index. MariaDB gives the DBA control over how much memory is dedicated to the vector index.

So there are actually two lessons here.

First, the current 16 MB default in MariaDB is probably worth reconsidering, or at least deserves to be made very visible to users experimenting with Vector.

Second, when comparing vector implementations, it is important to understand not only the QPS and recall figures, but also what needs to be kept in memory to achieve them.

Native is more than packaging

There is also one wording choice in the article that I would challenge: calling this “native MySQL vector search.”

MyVector is a very interesting project, but it remains an additional component installed on top of MySQL. MariaDB’s VECTOR type, vector distance functions and VECTOR INDEX are part of MariaDB Server itself.

And “native” here is not only about installation.

In MariaDB, vectors live in the same transactional database as the rest of the application data. Vector inserts and updates participate in normal ACID transactions, with the usual MariaDB isolation levels, and vector searches can run while the underlying data is being updated concurrently.

This means an application doesn’t need to maintain a separate vector index outside the database and keep it synchronized with its relational data. The embedding, the metadata and the business data can all be updated together in the same transaction.

It also means vector search remains regular SQL. You can combine semantic similarity with WHERE conditions, joins and the rest of the relational model in the same statement, while staying transactionally consistent with the data being searched.

For me, this is an important part of the comparison. Performance obviously matters, but operational simplicity and consistency matter too. A fast vector index is much more useful when it behaves like a normal database index rather than like another subsystem that the DBA has to deploy, monitor and keep synchronized.

Specific advantages of MariaDB Vectors:

• ACID transactions and all isolation levels are explicitly supported by MariaDB Vector.
• Concurrent reads and writes are supported.
• Embeddings can live next to relational data, instead of requiring a separate vector service or synchronization layer.
• You can combine vector similarity with normal SQL filtering and joins in the same query.
• MariaDB Vector is not a separate extension to install; it is part of MariaDB Server from 11.8 LTS onward.

Conclusion

So is MyVector fast? Absolutely.
Is the project interesting? Definitely.

But does this benchmark show that MySQL vector search is simply twice as fast as MariaDB? I don’t think so.

What it actually shows is that MariaDB’s integrated vector implementation can deliver very high recall with competitive performance — and that looking only at the biggest QPS number misses the most interesting part of the benchmark.

The main catchy graphic would then look like this (when mhnsw_max_cache_size is tuned) 😉

Again, thank you to Alkin for measuring and digging into MariaDB Vector search capabilities.

Enjoy MariaDB Vector!