Apache Gluten

Apache Gluten is an incubating project at the Apache Software Foundation designed to accelerate JVM-based SQL engines, particularly Apache Spark, by offloading compute-intensive tasks to native execution engines. It acts as a middle layer that bridges Spark’s distributed control flow with high-performance native libraries, enabling faster query execution without changing the user-facing APIs.

Features

• Offloads Spark SQL execution to native engines like Velox and ClickHouse
• Transforms Spark’s physical plan into Substrait plans for native execution
• Uses Apache Arrow for columnar data exchange between JVM and native layers
• Provides clear JNI interfaces for native integration
• Supports multiple native backends with easy switching
• Reuses Spark’s control flow and JVM components
• Extensible architecture for future native accelerators

Capabilities

• Accelerates Spark SQL performance by leveraging native vectorized processing
• Maintains compatibility with Spark’s DataFrame and SQL APIs
• Enables out-of-the-box integration with native libraries via configuration
• Supports distributed execution with native compute optimization
• Facilitates efficient data sharing between JVM and native environments

Benefits

• Significant performance improvements for Spark SQL workloads
• No need to modify existing Spark queries or applications
• Combines Spark’s scalability with native engine speed
• Reduces resource consumption and execution time
• Open-source and community-driven innovation
• Future-proof design with support for emerging native technologies