Common issues when scaling event-driven architectures By David Boyne

Common issues when scaling event-driven architectures By David Boyne

Common issues when scaling event-driven architectures

Talking to many folks in the community there seems to be shared problems that occur when you scale event-driven architecture solutions within your organisation.

Starting with a few producers/consumers can be great, but without considering a few fundamentals you may end up in the same place many others do, looks explore some of of these common issues.

Discoverability

  • Events are added over time, 10s, 100s or 1000s. All flying through your architecture

  • Many events can have many consumers, knowing who is consuming events can help

  • Many people scale events, but don’t keep track on who is consuming what. This can lead into event management issues (changing events, schemas etc).

  • Many folks want to consume events to enhance, their functionality. But where can they find events in your system? Are they documented somewhere? Probably not?

  • Event based architectures are super powerful and can help us scale and be decoupled, but without governance, standards and discoverability you will end up lost, who is coming what!? Can we change this field?

  • Ways to help, documenting your events, explore tools and solutions that can help. (visual to help)

  • Think about this up front, 2 years from now how will people discover your events?

Accidental coupling

  • So, you are building event-driven architectures, using events to communicate between systems, you are decoupled? Maybe not.

  • It’s important to remember we are still coupled by the event contract itself; consumers rely on the contract of the event and schema (often overlooked).

  • You have many ways to create events (e.g. notification, stateful events), depending on your choice depends on how coupled downstream consumers will be to the contracts.

  • Keeping track of who is consuming what can help you make schema/payload changes.

  • What is your schema management path? E.g no breaking changes? Forward compliable? Consider, in 2 years how are you going to make changes to these events?

  • Making changes to events, do you want to produce two different versions and give time for consumers to migrate, this is a valid pattern that some follow.

Leaky events, exposing too much info

Favour implementation over behaviour

  • It’s easy to get carried away with coding implementation details, and overlook the behaviour of the system, knowing what events should be fired when, and what should go inside these events.

  • Events can be technical events, but also business level events, everyone having an understanding of your business domain helps (sounds simple right? So many assume they know business behaviours).

  • Limit mental translations between teams, be clear on business intent and focus on behaviour then focus on implementation details.

  • EventStorming can help identify your domains and bounded context.

Extra Resources

  • Types of events - There are many types of events, understanding them can help.
  • Learning Domain Driven Design - Some notes taken from the book “Learning Domain Driven Design” by Vladik Khononov.
  • EventStorming - EventStorming is a great way to find events and domains, and also go that next level, start to map and shape your schema contracts.
  • CloudEvents - A specification for describing event data in a common way. Want to standarise your events, why not use industry standard?
  • AsyncAPI - Specification to help you define async APIS and event-driven architectures. Recommend looking into this.
  • Amazon EventBridge: Event Payload Standards - Blog post I wrote awhile ago, inspired by community, this is an example of event standards and payloads.

Download EDA Visuals

Join over 8,000 others learning EDA and download all the EDA visuals directly to your computer. This document updates everytime a new visual is added so make sure you come back to get the latest.

Download now →
Diagrams and thoughts by @boyney123 to help you learn.