In the world of data engineering, moving data from one place to another in real time has always been a complex challenge. Whether you're building analytics pipelines, syncing data across systems, or creating data lakes, the underlying infrastructure can quickly become unwieldy.
Enter Supabase ETL, an elegant Rust framework that transforms PostgreSQL logical replication into a powerful, developer-friendly streaming platform. Built by the team behind Supabase, this framework represents a fresh approach to change data capture that combines performance with simplicity.
supabase
Organization
etl
Stream your Postgres data anywhere in real-time. Simple Rust building blocks for change data capture (CDC) pipelines.The Challenge of Real-Time Data Movement
Modern applications generate data at incredible rates, and businesses need that data flowing to analytics platforms, data warehouses, and other systems in real time. Traditional ETL processes often rely on batch jobs that run periodically, creating delays between when data changes and when those changes become available downstream.
This lag can be critical for applications that require fresh data for machine learning models, real-time dashboards, or synchronized multi-database architectures. Furthermore, building reliable change data capture systems from scratch requires deep expertise in database internals, replication protocols, and distributed systems engineering. Many existing solutions are either too complex, too expensive, or lock you into proprietary ecosystems.
Key Takeaways
- Real-time streaming: Captures PostgreSQL changes as they happen through logical replication, enabling nearly instant data propagation to downstream systems.
- High performance: Configurable batching and parallel processing maximize throughput while minimizing latency for high-volume workloads.
- Fault tolerance: Built-in retry logic with exponential backoff, state management, and graceful error handling ensure pipeline reliability.
- Extensible architecture: Plugin system allows custom destinations, state stores, and schema stores to be implemented for any use case.
- Rust native: Type-safe API with compile-time guarantees, memory safety, and zero-cost abstractions for predictable performance.
- Production ready: Comprehensive testing across PostgreSQL 14-17, automated security audits, and observability features including structured logging and metrics.
Supabase ETL: A Developer-Friendly Answer
Supabase ETL tackles the inheriant challenges by providing a Rust-native framework that sits on top of PostgreSQL's robust logical replication protocol. Instead of building yet another black box, the Supabase team created a set of composable building blocks that developers can use to construct custom replication pipelines.
The framework handles the complex low-level details of connecting to PostgreSQL, decoding binary replication messages, managing state, and ensuring fault tolerance, while exposing a clean API that lets you focus on your business logic.
Whether you're streaming changes to BigQuery, building custom data transformations, or synchronizing data across multiple databases, Supabase ETL provides the foundation without imposing rigid constraints on how you use it.
Why This Framework Stands Out
What immediately caught my attention about Supabase ETL is its pragmatic design philosophy. The framework doesn't try to be everything to everyone. Instead, it focuses on doing one thing exceptionally well: providing a solid foundation for building real-time data pipelines from PostgreSQL.
I'm particularly fond of the Rust implementation which ensures memory safety and predictable performance, both of which are critical factors when you're processing potentially millions of database changes.
The modular architecture is particularly impressive - rather than forcing you into a specific workflow, it provides traits and abstractions that you can implement for your specific use cases.
The inclusion of configurable batching, parallelism controls, and built-in retry logic demonstrates that this was designed by people who understand production systems. Plus, the fact that it's open source under the Apache 2.0 license means you can inspect, modify, and deploy it without vendor lock-in concerns.
Core Capabilities That Matter
Supabase ETL delivers several standout features that make it production-ready. The framework provides real-time streaming of PostgreSQL changes through logical replication slots, capturing inserts, updates, and deletes as they happen.
Performance tuning is built in through configurable batching that lets you balance latency against throughput, alongside parallel processing support for handling high-volume workloads.
The fault-tolerance mechanisms include automatic retry logic with exponential backoff, state management to track replication progress, and graceful error handling that prevents data loss.
The extensibility model is particularly well-thought-out - you can implement custom destination traits to send data anywhere, create your own state stores for tracking pipeline progress, and build schema stores for managing table metadata.
The framework ships with a memory-based destination for testing and development, with BigQuery support available as an optional add-on. All of this is wrapped in a type-safe Rust API that catches many potential bugs at compile time rather than runtime.
The Technical Foundation
Supabase ETL is implemented as a Rust workspace comprising multiple focused crates. The core etl crate provides the fundamental abstractions and pipeline orchestration logic.
Supporting crates include etl-postgres for PostgreSQL-specific connectivity and protocol handling, etl-destinations for pre-built destination implementations, etl-config for configuration management, and etl-telemetry for observability.
The framework requires Rust 1.88.0 or newer and supports PostgreSQL versions 14, 15, 16, and 17, with PostgreSQL 15 and above recommended for accessing advanced publication features like column-level and row-level filtering.
At its heart, the framework uses PostgreSQL's logical replication protocol to receive a stream of change events. The Pipeline type orchestrates the entire process, coordinating between different worker threads that handle table synchronization and CDC streaming.
The architecture employs a worker pool pattern visible in the workers module, allowing multiple tables to be processed in parallel while maintaining consistency guarantees. State management happens through pluggable store interfaces, with implementations available in the store directory including both in-memory and persistent options.
The framework handles type conversions from PostgreSQL's binary format through a comprehensive conversions module that maps database types to Rust equivalents. Error handling is robust, with a detailed error taxonomy defined in error.rs covering everything from connection failures to data transformation issues.
The project uses Cargo workspaces for build management, with a comprehensive Cargo.toml that specifies dependencies and build profiles. The release profile enables thin link-time optimization for faster compilation while maintaining most performance benefits.
Dependencies include tokio for async runtime, sqlx for database connectivity, serde for serialization, and various AWS and GCP libraries for cloud integrations.
The team has made thoughtful choices in dependency management, using specific git revisions for some libraries like postgres-replication to ensure stability. Documentation generation and continuous integration workflows are defined in the .github directory, demonstrating professional development practices.
Building Your First Pipeline
Getting started with Supabase ETL involves adding the dependency to your Cargo.toml and writing a small amount of Rust code. Since the framework is not yet published to crates.io, you install it from the GitHub repository. Here's a minimal example that sets up a pipeline streaming to an in-memory destination:
use etl::{
config::{BatchConfig, PgConnectionConfig, PipelineConfig, TlsConfig},
destination::memory::MemoryDestination,
pipeline::Pipeline,
store::both::memory::MemoryStore,
};
#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
let pg = PgConnectionConfig {
host: "localhost".into(),
port: 5432,
name: "mydb".into(),
username: "postgres".into(),
password: Some("password".into()),
tls: TlsConfig { enabled: false, trusted_root_certs: String::new() },
};
let store = MemoryStore::new();
let destination = MemoryDestination::new();
let config = PipelineConfig {
id: 1,
publication_name: "my_publication".into(),
pg_connection: pg,
batch: BatchConfig { max_size: 1000, max_fill_ms: 5000 },
table_error_retry_delay_ms: 10_000,
table_error_retry_max_attempts: 5,
max_table_sync_workers: 4,
};
let mut pipeline = Pipeline::new(config, store, destination);
pipeline.start().await?;
Ok(())
}This example demonstrates the key components. You configure your PostgreSQL connection including authentication and TLS settings. The PipelineConfig specifies batching behavior, retry policies, and parallelism limits. The framework provides MemoryStore and MemoryDestination implementations for development, while production deployments would use persistent stores and real destinations like BigQuery.
Real-World Applications
Supabase ETL excels in scenarios requiring real-time data movement from PostgreSQL. The framework's flexibility and performance make it suitable for a wide range of applications across different industries and technical domains. Below are comprehensive use cases with real-world examples and references.
Analytics and Business Intelligence
Organizations need real-time insights from their operational databases to make data-driven decisions. Supabase ETL enables streaming database changes directly into data warehouses like BigQuery for near-instantaneous reporting and analysis. This approach eliminates the traditional batch ETL lag, where business intelligence dashboards show stale data hours or days old.
According to Debezium's analysis of CDC benefits, log-based change data capture provides significant advantages for analytics workloads by capturing every change without impacting source database performance. Teams can build real-time materialized views, aggregate metrics as data arrives, and power live dashboards that reflect current business state. The configurable batching in Supabase ETL allows tuning the latency-throughput tradeoff based on whether you need second-level freshness or can accept slightly higher latency for better efficiency.
Machine Learning and AI Pipelines
Machine learning systems require continuous access to fresh training data to maintain accuracy and adapt to changing patterns. Supabase ETL enables online machine learning workflows by streaming new data points as they're created in the operational database.
This pattern supports several ML use cases including: continuous model training where models update incrementally with new data, feature store population where computed features are kept current for inference, and data drift detection where statistical properties of incoming data are monitored.
The framework's fault tolerance ensures no training examples are lost even during temporary failures, critical for maintaining model quality. The extensible destination trait allows direct integration with ML platforms and vector databases for embedding-based retrieval systems.
Multi-Region Data Synchronization
Applications serving global users often require data replicated across multiple geographic regions for low-latency access. Supabase ETL provides the building blocks for sophisticated multi-region architectures where changes in one region propagate to others in real time.
PostgreSQL's logical replication foundation ensures consistency while allowing flexibility in topology - you can implement hub-and-spoke patterns, bidirectional sync, or complex multi-master setups.
The parallel processing capabilities mean high-volume tables can replicate efficiently even with geographic distance. This approach is more flexible than PostgreSQL's built-in physical replication since you can selectively replicate tables, transform data during replication, and target non-PostgreSQL destinations alongside PostgreSQL replicas.
Event-Driven Architectures and Microservices
Modern microservices architectures often need to react to database changes without tight coupling between services. Supabase ETL enables the outbox pattern where transactional writes and event publishing are atomically coupled through database replication. Services write to an outbox table in their local database, and the CDC pipeline reliably publishes those events to downstream consumers.
This guarantees at-least-once delivery without distributed transaction coordinators. The pattern is particularly powerful for domain events in event sourcing architectures, ensuring all state changes are captured and can trigger side effects in other services. Supabase ETL's extensible destinations mean you can publish to message queues, event buses, or directly invoke other services.
Data Lakes and Lakehouses
Organizations building data lakes benefit from preserving complete change history for compliance, auditing, and time-travel queries. Supabase ETL captures every insert, update, and delete operation, providing a complete audit trail. The planned Delta Lake support will enable ACID transactions in data lakes with efficient upserts and schema evolution.
This pattern supports slowly changing dimensions in data warehousing, regulatory compliance requirements for financial services, and debugging production issues by replaying historical state. The extensible destination architecture means you can write to cloud storage in formats like Parquet or Avro, optimized for analytical queries, while maintaining full change context.
Cache Invalidation and Materialized Views
High-performance applications rely on caching to reduce database load, but cache invalidation is notoriously difficult. Supabase ETL enables automated cache invalidation with CDC by detecting when cached data becomes stale and triggering updates.
Similarly, materialized aggregate views can be maintained in real time by incrementally updating aggregates as base data changes. This approach provides the query performance of denormalized data with the consistency of normalized sources. The parallel processing in Supabase ETL ensures cache updates don't fall behind source changes even under heavy load.
Full-Text Search and Indexing
Applications requiring full-text search often maintain separate search indexes in systems like Elasticsearch or Algolia. Supabase ETL enables streaming data to search engines by capturing database changes and synchronizing them to search indexes in real time. This ensures search results reflect current data without manual index rebuilds.
The extensible destination trait makes it straightforward to implement custom transformations for search documents, handling denormalization and enrichment required for optimal search relevance. The Algolia Connector mentioned by Supabase demonstrates this pattern for their broader platform.
Compliance and Audit Trails
Regulated industries require complete audit trails showing who changed what data and when. Supabase ETL captures all changes with transaction metadata, providing an immutable history suitable for compliance requirements.
Building audit logs with CDC avoids the brittleness of application-level audit logging where bugs or malicious actors might bypass logging. The CDC stream becomes the authoritative record of all data changes, which can be archived to immutable storage for long-term retention. This pattern is essential for GDPR, HIPAA, SOC 2, and other compliance frameworks requiring data lineage and change tracking.
Database Migration and Modernization
Organizations migrating to new database platforms or modernizing legacy systems face the challenge of moving data without downtime. Supabase ETL enables zero-downtime migrations by continuously replicating from the source PostgreSQL database to new systems while both remain operational.
Teams can gradually cut over traffic once the target system catches up, with the ability to roll back if issues arise. The extensible destination architecture means you can replicate to any target system - another PostgreSQL instance, a cloud-native database, or even NoSQL stores for polyglot persistence patterns.
Open Source Development
The project is actively developed by the Supabase team with contributions from the broader community. The GitHub issues show ongoing work on features like schema change handling, generated column support, and Delta Lake integration. Recent commits demonstrate continuous improvements to Kubernetes deployment, performance optimization, and bug fixes. The team maintains comprehensive documentation at supabase.github.io/etl including tutorials, API references, and how-to guides.
An AGENTS.md file provides development guidelines for AI assistants and contributors. The repository includes extensive examples in the etl-examples directory, helping new users understand common patterns. CI workflows ensure code quality through automated testing across multiple PostgreSQL versions, with coverage tracking and security audits built into the development process.
Apache 2.0 License Terms
Supabase ETL is released under the Apache License 2.0, one of the most permissive open source licenses available. This license grants you extensive rights to use, modify, and distribute the software, both in open source and proprietary applications. You can incorporate the framework into commercial products without royalty obligations, make modifications to suit your needs, and distribute your modified versions.
The license includes a patent grant from contributors, providing additional legal protection. Your main obligations are to include the license text and copyright notices with any distribution, and to note any modifications you make to the original code. The Apache 2.0 license also includes indemnification clauses protecting contributors from liability related to the use of the software.
This licensing model makes Supabase ETL an excellent choice for both startups needing flexibility and enterprises requiring clear legal terms. The permissive nature means you can build proprietary extensions without releasing your source code, while still benefiting from community improvements to the core framework.
The Company Behind the Framework
Supabase is building what they call the open source Firebase alternative, providing developers with a complete backend-as-a-service platform built on PostgreSQL. Founded with a mission to make powerful developer tools accessible to everyone, Supabase offers a full suite of services including database hosting, authentication, real-time subscriptions, file storage, edge functions, and vector embeddings.
The platform is trusted by innovative companies ranging from startups to enterprises like Mozilla, GitHub, and PwC.
What sets Supabase apart is their commitment to open source - nearly every component of their stack, including this ETL framework, is available under permissive licenses. Their tagline "Build in a weekend, scale to millions" captures their focus on developer experience and scalability. The company provides both a hosted platform at supabase.com and self-hosting options, giving teams flexibility in how they deploy.
The ETL framework fits naturally into Supabase's ecosystem, enabling customers to move data from their Supabase-hosted PostgreSQL databases to analytics platforms and other destinations. Beyond the technical platform, Supabase has cultivated a vibrant community through Discord, regular launch weeks showcasing new features, and extensive educational content.
Enterprise-Grade Reliability
The framework demonstrates several characteristics of production-ready software. Security audits are automated through GitHub Actions, with dependencies monitored for vulnerabilities. The test suite covers both unit and integration scenarios, running against multiple PostgreSQL versions to ensure compatibility. Performance benchmarks in the etl-benchmarks crate help track regression and optimization opportunities.
The codebase follows Rust best practices with comprehensive error handling, proper resource cleanup, and attention to edge cases. Observability is built in through structured logging and metrics export, making it straightforward to monitor pipeline health in production.
The framework includes Docker support and Kubernetes deployment configurations in the etl-api crate, demonstrating readiness for container orchestration. Recent commits show active maintenance with regular bug fixes and performance improvements, suggesting this is not abandonware but a living project backed by a well-funded company.
The Future of Real-Time Data
Supabase ETL arrives at an interesting moment in the data engineering landscape. As organizations move from batch processing to streaming architectures, the demand for reliable, performant change data capture tools will only grow.
The framework's focus on PostgreSQL is strategic - Postgres has become the default choice for many modern applications, and its logical replication capabilities provide a robust foundation for CDC. The open source nature of the project means it can evolve with community needs rather than being constrained by vendor priorities.
Looking at the roadmap visible in GitHub issues, features like schema change handling and support for additional destinations will expand the framework's applicability. The integration of this framework into Supabase's broader platform could create interesting synergies, particularly around managed pipeline services.
As Rust continues gaining adoption in systems programming, having high-quality data infrastructure tools like this helps justify choosing Rust for new projects. The real impact of Supabase ETL may be in democratizing real-time data pipelines, making what was once the domain of specialized platforms accessible to any developer comfortable with Rust.
A Solid Foundation for Modern Data Pipelines
Supabase ETL represents thoughtful engineering applied to a real problem in modern data architecture. Rather than creating yet another proprietary CDC platform, the Supabase team has contributed a flexible, performant framework that developers can adapt to their specific needs.
The combination of Rust's safety guarantees, PostgreSQL's robust replication protocol, and a well-designed API makes this a compelling option for teams building real-time data infrastructure. Whether you're streaming changes to a data warehouse, building event-driven architectures, or synchronizing distributed databases, Supabase ETL provides the building blocks without imposing unnecessary constraints.
The active development, comprehensive documentation, and permissive licensing reduce the risks typically associated with adopting open source infrastructure. If you're working with PostgreSQL and need to move data in real time, this framework deserves serious consideration. Explore the GitHub repository, review the examples, and join the Supabase community to learn more about building real-time data pipelines that scale.
Supabase ETL: Building Real-Time PostgreSQL Replication Pipelines in Rust