Common issues when scaling event-driven architectures By David Boyne
Common issues when scaling event-driven architectures By David Boyne
Talking to many folks in the community there seems to be shared problems that occur when you scale event-driven architecture solutions within your organisation.
Starting with a few producers/consumers can be great, but without considering a few fundamentals you may end up in the same place many others do, looks explore some of of these common issues.
Discoverability
-
Events are added over time, 10s, 100s or 1000s. All flying through your architecture
-
Many events can have many consumers, knowing who is consuming events can help
-
Many people scale events, but don’t keep track on who is consuming what. This can lead into event management issues (changing events, schemas etc).
-
Many folks want to consume events to enhance, their functionality. But where can they find events in your system? Are they documented somewhere? Probably not?
-
Event based architectures are super powerful and can help us scale and be decoupled, but without governance, standards and discoverability you will end up lost, who is coming what!? Can we change this field?
-
Ways to help, documenting your events, explore tools and solutions that can help. (visual to help)
-
Think about this up front, 2 years from now how will people discover your events?
Accidental coupling
-
So, you are building event-driven architectures, using events to communicate between systems, you are decoupled? Maybe not.
-
It’s important to remember we are still coupled by the event contract itself; consumers rely on the contract of the event and schema (often overlooked).
-
You have many ways to create events (e.g. notification, stateful events), depending on your choice depends on how coupled downstream consumers will be to the contracts.
-
Keeping track of who is consuming what can help you make schema/payload changes.
-
What is your schema management path? E.g no breaking changes? Forward compliable? Consider, in 2 years how are you going to make changes to these events?
-
Making changes to events, do you want to produce two different versions and give time for consumers to migrate, this is a valid pattern that some follow.
Leaky events, exposing too much info
-
What goes into your event? Do you just throw anything in? Probably not.
-
When starting out it’s often overlooked and payloads in events can be anything.
-
Take time, think and consider standards in your events.
-
Consider your domains and bounded context, consider what information should stay within a domain and what information can be exposed.
-
Use a public interface/language between your bounded context and others.
-
If you are consuming events, you might want to transform these, before external domain models leak into your domain.
-
Spend time here, think about bounded context and the language used within each domain. Be mindful in what you expose.
Favour implementation over behaviour
-
It’s easy to get carried away with coding implementation details, and overlook the behaviour of the system, knowing what events should be fired when, and what should go inside these events.
-
Events can be technical events, but also business level events, everyone having an understanding of your business domain helps (sounds simple right? So many assume they know business behaviours).
-
Limit mental translations between teams, be clear on business intent and focus on behaviour then focus on implementation details.
-
EventStorming can help identify your domains and bounded context.
Extra Resources
- Types of events - There are many types of events, understanding them can help.
- Learning Domain Driven Design - Some notes taken from the book “Learning Domain Driven Design” by Vladik Khononov.
- EventStorming - EventStorming is a great way to find events and domains, and also go that next level, start to map and shape your schema contracts.
- CloudEvents - A specification for describing event data in a common way. Want to standarise your events, why not use industry standard?
- AsyncAPI - Specification to help you define async APIS and event-driven architectures. Recommend looking into this.
- Amazon EventBridge: Event Payload Standards - Blog post I wrote awhile ago, inspired by community, this is an example of event standards and payloads.
Explore other visuals
Want to work together?
If you're interested in collaborating, I offer consulting, training, and workshops. I can support you throughout your event-driven architecture journey, from design to implementation. Feel free to reach out to discuss how we can work together, or explore my services on EventCatalog.
Download EDA Visuals
Join over 10,000 others learning EDA and download all the EDA visuals directly to your computer. This document updates everytime a new visual is added so make sure you come back to get the latest.