Aerospike Event Stream Processing

Change Notification (CN) in Aerospike is similar to CDC (Change Data Capture), but differs in how it reports changes. Unlike CDC, Aerospike’s CN may include several changes in a single notification. It uses Aerospike’s Cross-Datacenter Replication (XDR) to publish changes as they occur. Each message that the system sends contains either an updated or inserted record, or a record deletion notification, as well as useful metadata.

Aerospike Connect for Streaming Event Processing (ESP) converts change notifications over XDR into HTTP/1 or HTTP/2 requests and streams them to downstream consumers. This enables several use cases, such as server-less event processing, by using either AWS Lambda or Google Cloud Functions.

Connect for ESP is not a replacement for Kafka or Pulsar. For certain use cases, such as server-less event processing, exporting Aerospike data to Elasticsearch or Splunk, etc., it can obviate the need for Kafka or Pulsar in your streaming pipeline. However, there are a few tradeoffs depending on your use case. For instance, unlike Kafka or Pulsar, there is no message retention, topics and partitions, or schema registry.

Features

• Connect for ESP converts CN over XDR into HTTP/1 or HTTP/2 requests, which potentially opens up connections to several HTTP-based systems such as ElasticSearch, AWS Lambda, Splunk, etc.
• Serializes the change notification payload into a text format such as JSON, or a binary format such as Avro or MessagePack, for efficient data exchange.
• Ships the LUT (last-update-time) of the record to enable downstream applications to build their own custom logic for ordering messages.
• Extends XDR’s at-least-once delivery guarantee to ensure zero message loss.
• Can be used with XDR filter expressions to filter out records before transmission. For example, ensure that only records with a compliance bin set to true end as searchable data in Elasticsearch.
• Offers flexible deployment options, either in the cloud or on-prem, with support for Docker and Kubernetes.
• Can be configured to route CNs to multiple destinations using the same ESP connector cluster.
• Includes an option to add custom headers specific to the target system to each HTTP request.
• Supports HTTPS, which can be configured with PEM files.