As data becomes an increasingly important part of modern businesses, organizations often find themselves needing to process large amounts of data. Two common approaches to processing data are batch processing and event streams.

Batch processing

• Batch processing involves processing data at once usually during a scheduled time interval such as daily or weekly.

• Commonly used for tasks that do not require real-time processing and tasks that can tolerate some delay.

• Great if you have a fixed set of input data you want to process.

• Example of batching would be AWS Batch or Apache Spark

Event streaming

• Event streams involve processing data as it happens. Event streams are a continuous flow of data that can be collected and processed in (near) real time.

• Commonly used for applications that require real-time insights. Example of this could be fraud detection, or real-time recommendations.

• Processing data in real time gives organisations information they can act on and analyse straight away, done right this could provide a competitive edge to your business.

• Events can be unbounded data (continuous data, or never ending), so if you need to process this kind of information in real-time then streaming can help.

• Example of streaming would be Amazon Kinesis or Apache Kafka

When to use batching or event streaming?

• Like always, depends on your use case. Can your data processing tolerate some delay? Then maybe batching might be a better fit. Do you need to process the data right now? Will you have an advantage to processing information straight away? Then streaming events might make more sense.

• Remember to consider the costs between batching options and streaming options.

• Use patterns to help your downstream consumers. Remember with streaming and batching you can apply integrations patterns during that particular phase. Example would be enriching events/messages as they go through your stream for downstream consumers.

Extra Resources

• Designing data-intensive applications - Book has some great insights into batching and streaming. Recommend reading.
• Understanding event streaming - Dive deeper into event streaming with this visual.