Skip to main content
Version: Next

Data Accelerators

Data sourced by Data Connectors can be locally materialized and accelerated using a Data Accelerator.

A Data Accelerator queries/fetches data from a connected data source and stores/updates it locally in an embedded acceleration engine, such as Spice Cayenne, DuckDB, or SQLite. To set data refresh behavior, such as refreshing data on an interval, see Data Refresh.

Dataset acceleration is enabled by setting the acceleration configuration:

datasets:
  - name: accelerated_dataset
    acceleration:
      enabled: true

For the complete reference specification, see datasets.

By default, datasets are locally materialized using in-memory Arrow records.

Supported Data Accelerators​

NameDescriptionStatusEngine Modes
cayenneSpice CayenneStablememory, file, file_create, file_update
arrowIn-Memory Arrow RecordsStablememory
duckdbEmbedded DuckDBStablememory, file, file_create, file_update
postgresAttached PostgreSQL (Spice.ai Enterprise)Release CandidateN/A
sqliteEmbedded SQLiteRelease Candidatememory, file, file_create, file_update
tursoEmbedded TursoBetamemory, file, file_create, file_update

Choosing an Accelerator​

Select the appropriate accelerator based on dataset size, query patterns, and resource constraints:

Use CaseRecommended AcceleratorRationale
Small datasets (under 1 GB), maximum speedarrowIn-memory storage provides lowest latency
Small datasets (1-10 GB), complex SQLduckdbMature SQL support with memory management
Datasets 10 GB and above (up to 1+ TB)cayenneNeeds 1/3 to 1/2 the memory of duckdb; Vortex columnar format scales beyond single-file limits
Point lookups on large datasetscayenneVortex provides 100x faster random access vs Parquet
Simple queries, low resource usagesqliteLightweight, minimal overhead
Async operations, concurrent workloadstursoNative async support, modern connection pooling
External database integrationpostgresUse existing PostgreSQL infrastructure

Spice Cayenne vs DuckDB​

Both Spice Cayenne and DuckDB support file-based acceleration, but differ in architecture and performance characteristics. Spice Cayenne is recommended for any dataset of 10 GB or larger, because of DuckDB's memory requirements: Cayenne typically needs one-third to one-half the memory of the DuckDB accelerator for the same dataset.

Choose Spice Cayenne when:

  • Datasets are 10 GB or larger
  • Memory headroom is constrained, or the deployment must run in a smaller container
  • Multi-file data ingestion is required (e.g., partitioned S3 data)
  • Workloads benefit from Vortex's 10-20x faster scans
  • Point lookups and random access patterns are common (100x faster than Parquet)

Choose DuckDB when:

  • Datasets are under 10 GB
  • Complex SQL features are required (window functions, CTEs)
  • Existing DuckDB tooling integration is beneficial
  • Explicit index control is required

Data Types​

Data Accelerators may not support all possible Apache Arrow data types. For complete compatibility, see specifications.

Memory Considerations

When accelerating a dataset using mode: memory (the default), some or all of the dataset is loaded into memory. Ensure sufficient memory is available, including overhead for queries and the runtime, especially with concurrent queries.

In-memory limitations can be mitigated by storing acceleration data on disk, which is supported by duckdb, sqlite, and turso accelerators by specifying mode: file.

Schema Handling​

Data accelerators store the schema that Spice infers from the data source at startup. This schema is fixed for the lifetime of the runtime process and defines the column names, data types, and nullability of the accelerated table.

If the source schema changes while the runtime is running (for example, new columns are added or data types change), subsequent data refreshes into the accelerator will fail because the incoming data no longer matches the schema of the accelerated table. Restart the runtime to re-infer the schema and re-initialize the accelerated table.

For details on how schema inference works per connector and recommendations for managing schema drift, see Schema Inference.

Data Accelerator Docs​