Build Apache Spark applications that read and write data from Amazon Redshift

Why Amazon Redshift Integration for Apache Spark?

Amazon Redshift Integration for Apache Spark simplifies and accelerates Apache Spark applications accessing Amazon Redshift data from AWS analytics services such as Amazon EMR, AWS Glue, and Amazon SageMaker. Using Amazon EMR, AWS Glue, and SageMaker, you can quickly build Apache Spark applications that read from and write to your Amazon Redshift data warehouse, without compromising performance or transactional consistency. Amazon Redshift Integration for Apache Spark also uses AWS Identity and Access Management (IAM)–based credentials to enhance security. With Amazon Redshift Integration for Apache Spark, there is no manual setup and maintenance of uncertified versions of third-party connectors. You can start with Apache Spark jobs using data in Amazon Redshift in seconds. This new integration improves the performance of Apache Spark applications using Amazon Redshift data.

Benefits of Amazon Redshift

Enhance Apache Spark analytics with Amazon Redshift data

Expand the breadth of data sources that you can use in your rich analytics and machine learning (ML) applications running in Amazon EMR, AWS Glue, or SageMaker by reading from and writing data to your data warehouse.

Access Amazon Redshift data with minimal setup

Streamline the cumbersome and often manual process of setting up uncertified connectors and JDBC drivers, reducing the preparation time for analytics and ML tasks.

Enhance performance and security with an Amazon certified connector

Use several pushdown capabilities such as sort, aggregate, limit, join, and scalar functions so that only relevant data is moved from the Amazon Redshift data warehouse.

How it works

Use AWS services to build Apache Spark applications that read and write to your Amazon Redshift data warehouse.

Use cases

Build ETL, ML, and interactive applications

Create Apache Spark applications in Java, Scala, and Python with Apache Spark–based AWS analytics services.

Connect to your Amazon Redshift data warehouse

Read and write data to and from Amazon Redshift with Amazon EMR, AWS Glue, SageMaker, and AWS analytics and ML services.

Run queries in seconds

Use Amazon EMR or AWS Glue to take data frame code from your Apache Spark job or notebook and connect to Amazon Redshift.

AWS certified connector for immediate use

Streamline your process with no installation or testing, enhanced security (IAM-based credentials) and operational pushdowns, and Parquet file format for performance.