Ben Stopford
From Confluent
Ben is an engineer and architect working on the Apache Kafka Core Team at Confluent Inc (the company behind Apache Kafka). He's worked with distributed data infrastructure for over a decade, switching between engineering products and helping companies use them. Before Confluent he designed and built a central streaming database for large investment bank. His earlier career spanned a variety of projects at Thoughtworks and UK-based enterprise companies. Find out more at http://benstopford.com.
Blog: http://benstopford.com
Architecture & Security
Rethinking Microservices with Stateful Streams
Microservices are one of those polarising concepts that technologists either love or hate. Splitting applications into autonomous units clearly has advantages, but larger service-based systems tend to struggle as the interactions between the services grow.
At the core of this sits a dichotomy: Data systems are designed to make data as accessible as possible. Services, on the other hand, actively encapsulate. These two forces inevitably compete in the architectures we build. By understanding this dichotomy we can better reason about how services should be sewn together. We strike a balance between the ability to adapt quickly and the loose coupling we need to retain autonomy, long term.
In this talk we'll examine how Stateful Stream Processing can be used to build Event Driven Services, using a distributed log like Apache Kafka. In doing so this Data-Dichotomy is balanced with an architecture that exhibits demonstrably better scaling properties, be it increased complexity, team size, data volume or velocity.