By ByteBot
DuckDB’s engineering team published a detailed async I/O implementation post on July 31, revealing what’s coming in v2.0 — and the numbers are hard to dismiss. DuckDB async I/O cuts CSV reads from S3 from 877 seconds to 45 seconds: a 19.4x speedup. Moreover, Parquet queries against S3 get roughly 3x faster. Preview builds are available now; v2.0 ships in September 2026 with async I/O on by default. If your DuckDB workloads touch remote object storage, this is the most significant performance change since DuckDB went columnar.
Why DuckDB Was Slow on S3
The old synchronous model had a fundamental problem: worker threads blocked waiting for network bytes. Every time a thread issued a read against S3, it stalled until the data arrived — doing nothing useful in between. Consequently, on a 4-concurrent-query test using a 64-core EC2 r7i.16xlarge, DuckDB v1.5.5 averaged just 5.9 active cores and 10.7 Gbit/s of network throughput. That’s 90% of a powerful machine sitting idle, waiting on I/O.
This isn’t a DuckDB-specific mistake — it’s the natural outcome of a design built for local data that later expanded to cloud storage. The synchronous assumption made sense when DuckDB was an embedded analytics engine for local Parquet files. However, it became a bottleneck as developers started pointing DuckDB at S3 data lakes containing hundreds of gigabytes. In fact, this single architectural choice was responsible for leaving most of a 64-core machine idle.
How the Dual Thread Pool Fixes It
The async I/O implementation separates computation from I/O using two distinct thread pools. The REGULAR pool runs one thread per CPU core and handles decoding, joins, and aggregations — the actual work. Meanwhile, the ASYNC pool runs up to 4x system threads (capped at 256) and does nothing but block on network calls. Since ASYNC threads spend most of their time waiting, running many of them is cheap and doesn’t starve compute threads.
A read-ahead strategy completes the picture. Instead of issuing reads on demand, DuckDB schedules fetch tasks ahead of what workers currently need. When a REGULAR thread finishes its current task and its next data chunk isn’t ready, it parks that scan and picks up other pipeline work rather than stalling. As a result, concurrent queries averaged 48.1 cores and 24.9 Gbit/s throughput on the same 64-core machine — nearly 10x the core utilization of the synchronous model.
Furthermore, developers can tune the behavior with a few SET commands:
-- Async I/O is enabled by default in v2.0 preview builds
SET read_ahead_depth = 5; -- Jobs prefetched ahead; -1 = unlimited (default); 0 = sync
SET async_threads = 48; -- Override ASYNC pool size (default: 4x CPU cores)
SET http_retries = 8; -- Increase for flaky connections
SET http_retry_wait_ms = 50; -- Initial retry wait before backoffDuckDB Async I/O Benchmark Results
The 19.4x headline comes from CSV reads — a format that downloads far more raw data than Parquet because it lacks column pruning. Parquet already filters at the storage level, so the baseline is lower and the relative gain is smaller but still substantial: 8.2 seconds drops to 2.8 seconds on a 22GB file. Additionally, concurrent queries show the most dramatic real-world improvement: four simultaneous TPC-H queries went from 35.8 to 15.6 seconds, and those requests drove network utilization from 10.7 to 24.9 Gbit/s — nearly saturating the available bandwidth.
Local SSD users get a 1.5x improvement on cold reads, nothing meaningful on hot cached data. Therefore, async I/O is primarily a remote storage optimization. If you’re querying DuckDB databases on local NVMe, this is a nice-to-have, not a must-have. For S3-backed data lakes, however, it’s a different conversation entirely.
One caveat worth knowing: Parquet gains depend heavily on row group count. A single-row-group file took 25 seconds versus 2.8 seconds for a properly partitioned file with 300+ row groups on the same 64-core machine. In other words, async I/O enables parallelism — your data layout has to expose it. If your Parquet writer uses default row group sizes, check them before expecting benchmark-level results.
Try DuckDB Async I/O Today
You don’t have to wait for the September release. Preview builds are available now and async I/O is enabled by default in them. Point it at an S3 workload you already run and compare the runtime. For example, setting read_ahead_depth = 0 disables async for a clean A/B comparison against your current numbers.
The team’s next targets are JSON and native DuckDB file formats, followed by io_uring investigation for Linux — which would reduce system-call overhead even further. Combined with the Quack protocol hitting stable in v2.0, DuckDB is executing a clear “year of DuckDB as a server” strategy. Consequently, async I/O is the performance foundation that makes multi-user server workloads practical.
Key Takeaways
- DuckDB async I/O ships on by default in v2.0 (September 2026); preview builds are available today
- S3 and remote object storage see the biggest gains: CSV queries up to 19.4x faster, Parquet up to 3x faster
- The fix is architectural: a dedicated ASYNC thread pool stops compute threads from stalling on network waits
- Concurrent queries benefit most — a 64-core machine went from 5.9 to 48.1 active cores on the same workload
- Partition your Parquet data with 300+ row groups to fully exploit parallelism; local SSD gains are real but modest at 1.5x
