arrow-schema

Spacedrive is an open source cross-platform file explorer, powered by a virtual distributed filesystem written in Rust.

Scalable datastore for metrics, events, and real-time analytics

A high-performance observability data pipeline.

dbt enables data analysts and engineers to transform their data using the same practices that software engineers use to build applications.

Incremental engine for long horizon agents 🌟 Star if you like it!

Developer-friendly OSS embedded retrieval library for multimodal AI. Search More; Manage Less.

A data visualization and analytics component, especially well-suited for large and/or streaming datasets.

Data Agent Ready Warehouse : One for Analytics, Search, AI, Python Sandbox. — rebuilt from scratch. Unified architecture on your S3.

Event streaming platform for agentic AI. Continuously ingest, transform, and serve event streams in real time, at scale.

Apache DataFusion SQL Query Engine

Simple, Elastic-quality search for Postgres

Open Lakehouse Format for Multimodal AI. Convert from Parquet in 2 lines of code for 100x faster random access, vector index, and data versioning. Compatible with Pandas, DuckDB, Polars, Pyarrow, and PyTorch with more integrations coming..

The open-source Observability 2.0 database. One engine for metrics, logs, and traces — replacing Prometheus, Loki & ES.

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Distributed stream processing engine in Rust

Language model tokenization at GB/s

DORA (Dataflow-Oriented Robotic Architecture) is middleware designed to streamline and simplify the creation of AI-based robotic applications. It offers low latency, composable, and distributed dataflow capabilities. Applications are modeled as directed graphs, also referred to as pipelines.

Official Rust implementation of Apache Arrow

Drop-in Apache Spark replacement written in Rust, unifying batch processing, stream processing, and compute-intensive AI workloads.

A native Rust library for Delta Lake, with bindings into Python

An extensible, state-of-the-art framework for columnar compression, and the fastest FOSS columnar file format. Formerly at @spiraldb, now an Incubation Stage project at LFAI&Data, part of the Linux Foundation.

A portable accelerated SQL query, search, and LLM-inference engine, written in Rust, for data-grounded AI apps and agents.

Parseable is an observability datalake built from first principles.

An AI agent for teams, communities, and multi-user environments.

The Feldera Incremental Computation Engine

Communication infrastructure for the AI era — one binary, one broker, one storage layer, any protocol

The Auron accelerator for distributed computing framework (e.g., Spark) leverages native vectorized execution to accelerate query processing

A cloud-native open source distributed time series database with high performance, high compression ratio and high availability.

Tonbo is an embedded database for serverless and edge runtimes.

Apache Iceberg

Local-first ETL/ELT studio: a drag-and-drop visual pipeline designer that compiles to SQL and runs on DuckDB. Tiny desktop app, no servers, git-friendly workspaces.

Local-first ETL/ELT studio: a drag-and-drop visual pipeline designer that compiles to SQL and runs on DuckDB. Tiny desktop app, no servers, git-friendly workspaces.

Lakehouse native graph engine with git-style workflows

AI-Native & Cloud-Native FS: A high-performance file semantic layer for cloud object storage, integrated with high-speed cache

Flow-Like: Strongly Typed Enterprise Scale Workflows. Built for scalability, speed, seamless AI integration and rich customization.

Postgres Foreign Data Wrapper development framework in Rust.

Intuitive Data Workflows

Rust Agent Development Kit (ADK-Rust): Build AI agents in Rust with modular components for models, tools, memory, realtime voice, and more. ADK-Rust is a flexible framework for developing AI agents with simplicity and power. Model-agnostic, deployment-agnostic, optimized for frontier AI models. Includes support for real-time voice agents.

Rust-based WebAssembly bindings to read and write Apache Parquet data

Scalable graph analytics database powered by a multithreaded, vectorized temporal engine, written in Rust

Elodin simulation and flight software monorepo.

Analytical database for data-driven Web applications 🪶

A single-node analytical database engine with geospatial as a first-class citizen

The cost efficiency of S3 with the speed of local RAM. A multi-tenant vector and full-text search engine featuring a tiered RAM -> NVMe -> S3 architecture for microsecond latency on top of object storage

GeoArrow in Rust, Python, and JavaScript (WebAssembly) with vectorized geometry operations

Protocol and libraries for sending and receiving OpenTelemetry data using Apache Arrow

Fully Managed, Streaming Ingestion (CDC) into your Lakehouse

Zorai is a persistent, multi-agent, auditable, learning execution platform where the daemon owns work, memory, approvals, tools, and long-running goals.

SciRS2 - Scientific Computing and AI in Rust., providing SciPy-compatible APIs while leveraging Rust's performance, safety, and concurrency features. Unlike traditional scientific libraries

The native Rust implementation for Apache Hudi, with C++ & Python API bindings.