Data Accelerators

Data sourced by Data Connectors can be locally materialized and accelerated using a Data Accelerator.

Acceleration is enabled on a dataset by setting the acceleration configuration. E.g.

datasets:
  - name: accelerated_dataset
    acceleration:
      enabled: true

For the complete reference specification see datasets.

By default, datasets will be locally materialized using in-memory Arrow records.

Data Accelerators using DuckDB, SQLite, or PostgreSQL engines can be used to materialize data in files or attached databases.

Currently supported Data Accelerators include:

Engine Name Description Status Engine Modes
arrow In-Memory Arrow Records Alpha memory
duckdb Embedded DuckDB Alpha memory, file
sqlite Embedded SQLite Alpha memory, file
postgres Attached PostgreSQL Alpha
PostgreSQL Data Accelerator

PostgreSQL Data Accelerator Documentation

SQLite Data Accelerator

SQLite Data Accelerator Documentation