Back in July, I wrote about a native ADBC driver for MariaDB. That was something I’d been working on to use MariaDB Server with DuckDB via Apache Arrow Database Connectivity (ADBC).
The driver was working and I was already able to do what I originally wanted. It was still very much a personal project, living in my GitHub repo and anyone who wanted to give it a go would have had to build it themselves.
Then something interesting happened.
A bit later, after publishing the article, I got in touch with Columnar. They run the ADBC Driver Foundry, the infrastructure that’s used to build and distribute the ADBC drivers. They saw my post and they asked if I’d be interested in having the MariaDB driver included in their offering.
Of course I was interested as that was also something I was planning on if there was any interest at all. To be honest, I was expecting interest from the user community, not necessarily from Driver Foundry themselves!
That kick started a collaboration which turned out to be far more than just moving a repo. We went through the code, cleaned up a fair few things, improved the tests, added some missing functionality and got the driver into a state where it could be built and distributed via their infrastructure.
And I’m happy to announce that the MariaDB ADBC driver is now part of the ADBC Driver Foundry and is available on the Columnar CDN!
You can find it here:
- Source code: https://github.com/adbc-drivers/mariadb
- Driver documentation: https://adbc-drivers.org/drivers/mariadb/
- Driver CDN: https://dbc-cdn.columnar.tech/
And instead of having to compile the driver, you can now install it with:
dbc install mariadbNice, isn’t it?
More Than Just Moving a Repository
Getting the driver into the ADBC Driver Foundry wasn’t just a matter of moving source code to another GitHub org.
The original implementation did indeed support SQL execution, Arrow record batches, prepared statements, transactions, bulk ingestion and metadata discovery.
But making a driver available to a wider audience also means improving its quality, compatibility, testing, and packaging.
During this work, I went back and looked at several parts of the implementation.
Better Handling of MariaDB Data Types
One of the main reasons for building a dedicated MariaDB driver was to get right the MariaDB specific features and data types.
MariaDB and MySQL have a shared history and a compatible client protocol, but they are not the same database.
Using a generic MySQL driver will work for a lot of queries but it doesn’t necessarily preserve all the information that’s available in MariaDB.
The driver now includes more comprehensive handling of data types such as:
- UUID: exposed via the standard Arrow UUID extension type when the logical column type is known.
- VECTOR(n): represented as a fixed size list of float32 values when the dimension is known, which makes it particularly pertinent for vector based applications.
- JSON is recognized as a logical JSON type via the schema metadata even though MariaDB actually uses LONGTEXT as its underlying type.
- INET4 and INET6: represented as strings with additional logical-type metadata.
- Spatial types: exposed as Well-Known Binary (WKB), the MariaDB specific SRID prefix is dealt with by the driver.
- ENUM and SET: exported as strings with their allowed values retained in the Arrow field metadata.
- Unsigned integers and DECIMAL: mapped with care to respect for precision and range.
- Date and time values: including fractional seconds and MariaDB’s extended TIME range.
This is important because moving data from one database to another (or from a database into an analytical tool) is not simply about moving the values. We also want to retain as much of the semantic information as possible.
For example a MariaDB VECTOR column shouldn’t become an anonymous collection of bytes when the application on the receiving end can understand the structure.
There are still constraints of course. In particular, the metadata returned for an arbitrary SQL expression doesn’t always contain enough information to allow the original logical type to be reconstructed. The driver therefore avoids guessing and goes with the metadata it has.
Improved Metadata and Schema Discovery
Another area that got a lot of attention was metadata discovery.
ADBC provides standardized interfaces to discover database objects, table schemas and statistics.
The MariaDB driver can expose information including:
- Catalogs, tables, and columns.
- Data types, precision, scale and nullability.
- Primary keys, unique constraints, foreign keys and check constraints.
- Default values, comments and generated columns.
- Table schemas and parameter counts for prepared statements.
- Table and index statistics.
There’s also special handling for a number of MariaDB specificities, such as invisible columns.
This might seem a bit dull in comparison to actually running a query, but metadata is absolutely vital for any application that wants to be able to browse a database automatically, build queries or simply get an understanding of the data it’s working with.
Transactions and Isolation Levels
The driver now supports the standard ADBC transaction isolation option, with the isolation levels provided by MariaDB:
Apps can explicitly turn autocommit on/off, issue a commit or a rollback, and set their isolation level.
This makes the driver useful for things beyond just read-only analytical queries.
Better Testing and Project Structure
A large part of the work was invisible to the end user, but no less important.
I spent some time tidying up the implementation, improving the tests and bringing the project into line with the conventions that the ADBC Driver Foundry use.
The repo now contains the supporting project structure for collaboration, code quality, build and distribution.
This is very valuable for the project going forward in that contributors have a much clearer starting point and the driver can benefit from the infrastructure and the experience that’s already been built up around the other ADBC drivers.
What does being on the CDN actually change?
Quite a lot!
Previously you’d have had to get the source code and build it for your platform if you wanted to give the driver a go.
Now ready-to-install binaries are available from the Columnar ADBC CDN.
At the time of writing, version 0.1.6 is available for:
- Linux x86-64 and ARM64.
- macOS ARM64.
- Windows x86-64 and ARM64.
The CDN isn’t simply a place to go and download a shared library. It’s a registry of packaged ADBC drivers, for use with tools that can understand ADBC driver manifests.
One of the tools is dbc (the command-line tool) developed by Columnar.
After installing dbc, you can discover available drivers and install MariaDB:
dbc search mariadbdbc install mariadbThe driver can then be located by compatible ADBC driver managers via its manifest.
This makes ADBC adoption much easier. Apps don’t need to have a separately compiled MariaDB driver as part of their own distribution and the end user doesn’t need to know a thing about the implementation language.
The fact that the driver is in Go is pretty much an implementation detail. Applications interact with the standard ADBC interface.
Where can you use the MariaDB ADBC driver?
This is probably the most interesting bit.
ADBC was designed to make it easier to exchange data using Apache Arrow (a columnar in-memory data format).
That makes the driver extremely useful for analytical work, data engineering and any tool that already understands Arrow.
1. DuckDB: Query MariaDB Data From an Analytical Engine
This was my original reason for writing the driver!
With DuckDB’s ADBC extension we can attach a MariaDB database and query its tables directly from DuckDB.
First, install the MariaDB driver:
dbc install mariadbThen create an ADBC connection profile, for example in ~/.config/adbc/profiles/mydb.toml:
profile_version = 1
driver = "mariadb"
[Options]
uri = "mariadb://myuser:mypassword@127.0.0.1:3306/test"Start DuckDB:
INSTALL adbc FROM community;
LOAD adbc;
ATTACH 'profile://mydb' AS mariadb (
TYPE adbc,
DELIMITER '``'
);The DELIMITER option is important as MariaDB defaults to using backticks for quoted identifiers.
Now we can query our MariaDB tables:
SHOW ALL TABLES;
SELECT *
FROM mariadb.main.my_table
LIMIT 10;We can also use DuckDB’s SQL capabilities to analyze the data.
This opens up interesting possibilities for combining the transactional data stored in MariaDB with analytical processing in DuckDB.
One important point of clarification: ADBC doesn’t in any way turn MariaDB into a columnar storage engine. MariaDB still uses its standard client protocol and the driver is responsible for converting the results into Arrow record batches.
It’s possible to turn MariaDB into a columnar storage engine using the DuckDB Engine, see https://lefred.be/content/mariadb-duckdb-a-new-playground-for-analytics-a-first-look-at-the-new-storage-engine/
The benefit is that, once the data reaches the Arrow ecosystem, compatible applications can work with a common columnar representation.
2. Python, PyArrow, and Data Science
Another interesting use case is Python.
Install the ADBC driver manager:
pip install adbc-driver-manager pyarrowWith the MariaDB driver already installed using dbc, accessing a database is straightforward:
from adbc_driver_manager import dbapi
with dbapi.connect(
driver="mariadb",
db_kwargs={
"uri": "mariadb://myuser:mypassword@localhost:3306/test"
},
) as conn:
with conn.cursor() as cursor:
cursor.execute("""
SELECT *
FROM my_table
LIMIT 100
""")
table = cursor.fetch_arrow_table()
print(table)The result is an Apache Arrow table.
From there, it can be used with PyArrow and other Arrow-compatible Python libraries, such as Polars.
For example:
import polars as pl
df = pl.from_arrow(table)
print(df.head())This makes the driver interesting for data exploration, notebooks, data pipelines, and analytical applications.
Instead of building a separate MariaDB integration for every tool, applications that support ADBC can reuse the same database interface.
3. Data Engineering and Other ADBC Applications
The potential use cases are not limited to DuckDB or Python.
Because the driver implements the ADBC interface, it can be used by applications and libraries that support loading ADBC drivers through a compatible driver manager.
This includes opportunities for integration with:
- Analytical and data science tools.
- ETL and data processing pipelines.
- Arrow-based applications.
- Database exploration tools.
- Multi-database applications that want a common connectivity interface.
Of course, the exact integration depends on the application’s ADBC support and the features it exposes.
ADBC also supports more than reading query results. The MariaDB driver implements prepared statements, parameter binding, and Arrow-based bulk ingestion, making it possible to send data to MariaDB and retrieve it.
Why a Dedicated MariaDB Driver?
Some people may wonder why we need a dedicated MariaDB driver when an ADBC driver for MySQL already exists.
That’s a reasonable question.
The two databases have a lot in common in terms of their history and they both use the same general client/server protocol. For a lot of fairly simple SQL queries, the MySQL driver can also talk to MariaDB.
But MariaDB has developed its own features, its own types and its own metadata.
Explicitly supporting those differences gives us greater control over how MariaDB data is exposed to Arrow applications.
The native driver is therefore not just another way of establishing a connection. It’s an opportunity to provide better MariaDB-specific integration into the ADBC ecosystem.
And it gives us a place to add support for new MariaDB capabilities as they become available.
Open Source Collaboration in Action
There’s also a story behind that announcement which I really like.
I originally created the driver because I wanted to solve a practical problem. I published the source code, wrote about it, and shared what I had learned.
The Columnar team discovered the work and reached out.
Together, we improved the project and made it available to a much larger audience through an existing open-source ecosystem.
This is exactly the kind of collaboration I like to see in open source.
Instead of maintaining another isolated project, we can build on shared infrastructure, benefit from other contributors’ experience, and make MariaDB available to users of tools that follow an open standard.
A big thank you to the Columnar team for reaching out, helping with the integration, reviewing the work, and welcoming the MariaDB driver into the ADBC Driver Foundry!
What’s Next?
The driver is now much easier to install and use, but that certainly doesn’t mean development is finished.
There are still opportunities to improve compatibility, extend test coverage, refine data-type mappings, and support additional ADBC features.
I’m especially interested in how the driver can help bridge MariaDB with analytical engines and Arrow-based applications, and how it can evolve alongside new MariaDB Server functionality.
If you are using DuckDB, Python, or another ADBC-compatible application, give it a try!
The code is open source and we’d be very happy to have contributions, bug reports, tests and feedback.
Useful links:
- My original article: A Native ADBC Driver for MariaDB
- MariaDB ADBC driver on GitHub
- MariaDB ADBC driver documentation
- Columnar ADBC CDN
- dbc command-line tool
One driver, one standard, and many more possibilities for MariaDB in the Apache Arrow ecosystem!