Information Technology

<p style="text-align: justify;">Event-driven architecture is a mechanism for services in a computer software system to communicate with each other in a seamless manner using messages, also known as events or triggers.&nbsp;</p><p style="text-align: justify;">Events are fired as a way for services to execute a task and notify another service to enable a complex system to achieve the sequence of tasks it has in an established way. For this kind of architecture, usually, there has to be much planning around capacity and load for sizing, followed by determining the sequence and implementing it.&nbsp;</p><p style="text-align: justify;">Messaging has always been present as part of the infrastructure for more asynchronous tasks, such as moving data from point A to point B, indexing for search, and passing messages in microservices.&nbsp;</p><p style="text-align: justify;">However, in the current time and the era of data and scalability, we see a shift, and new tools are developed for achieving moving data at scale in a reliable and fast manner.&nbsp;</p><p style="text-align: justify;">In this article, we discuss Kafka as an example of an open-source platform that enables people to implement an event-driven architecture. The idea here is continuous execution and implementation of a sequence of tasks reliably as quickly as possible. We will start by explaining some examples, terminologies, and mechanisms that Kafka uses to move data to explore the landscape.<br />&nbsp;<br />Studying the examples and use cases helps you understand the concept better to expand on it for your systems and leverage existing tools.&nbsp;</p><p style="text-align: justify;">&nbsp;</p><h2 style="text-align: justify;"><span style="font-size: 14pt;">Use Cases of an Event-Driven Architecture&nbsp;</span></h2><p style="text-align: justify;"><br />When you think of an event-driven architecture, you want to visualize how a set of messages are being passed around between different parts of the system and how they are tied to the execution of tasks.&nbsp;</p><p style="text-align: justify;">One example would be a notification system built for monitoring where we want to notify users of specific changes in the system. These changes are continuously happening, so the system must always be up and running, listening to them, and transporting them.&nbsp;<br />Here we can assume there is a service capturing the changes and sending them over to the user via another service. In an ideal scenario, we want these notifications to be immediate. In a banking system, for example, that deals with people's personal and critical information, if there is any change to the user record, we want to be able to notify the user to prevent fraud.&nbsp;</p><p style="text-align: justify;">Hence, an important quality of an event-driven architecture is the speed, reliability, and consistency of the action to be taken upon the event instantaneously. The sequence of code that pushes and pulls the change should all be seamlessly running with minimal delays. Here is where optimizations are very significant at the code, network, and infrastructure layer and when there are security protocols as well. Where data is being stored and accessed and where services are pushing and pulling it and executing logic over it, is where parallel processing comes into play to help achieve the speed.&nbsp;</p><p style="text-align: justify;">When using Kafka, data is stored in a notion called topic, and you can think of it as a specific stream of data. We can have multiple instances known as clients running to read and write this data and keep track of things, so it can manage large throughput when writing the data and fast access when consuming it.&nbsp;</p><p style="text-align: justify;">Another very common instance of event-driven architecture is when we have a set of connected microservices. An example would be an e-commerce flow or IOT devices where these tools and services must cross-communicate and sync with each other reliably, and we expect this communication to be very responsive.&nbsp;<br />&nbsp;<br />The figure below describes this flow where we have a simple purchase flow in an e-commerce system that is implemented using various microservices. A user service sends a message to Kafka, the order service picks it up, processes it, and sends another message to the fulfillment, and once that's processed, the service executes the shipment flow and closes the cycle. This basic sequence is very popular, and KStreams or Consumer/Producer can be used to achieve this. We explain them below in the next section of Kafka terminologies.&nbsp;</p><p style="text-align: justify;">&nbsp;</p><p style="text-align: justify;"><img style="display: block; margin-left: auto; margin-right: auto;" src="https://kradminasset.s3.ap-south-1.amazonaws.com/ExpertViews/Avapic1.PNG" width="392" height="231" /></p><p style="text-align: justify;">&nbsp;</p><p style="text-align: justify;">The third use is integrating data, and when we want to copy part of one database, change to another database. In the figure below, DB1 is modified with a change that is recorded in the change table, and we copy this change to the intermediary system, Kafka, and copy that over to the destination DB. All of this can be accomplished using Kafka connectors that are more like fast pipes that have their own distributed architecture as a connected component in Kafka.</p><p style="text-align: justify;">&nbsp;</p><p style="text-align: justify;"><img style="display: block; margin-left: auto; margin-right: auto;" src="https://kradminasset.s3.ap-south-1.amazonaws.com/ExpertViews/Avapic2.PNG" width="464" height="230" /></p><p style="text-align: justify;">&nbsp;</p><h2 style="text-align: justify;"><span style="font-size: 14pt;">Kafka at Uber&nbsp;</span></h2><p style="text-align: justify;">A real-life example is Kafka at Uber.&nbsp;</p><p style="text-align: justify;">As you can see below, various client applications such as a rider and driver app, as well as other APIs and services, all push this data to Kafka as the central communication system and what happens here is that on the other side, there are other client applications such as surge that determines where are higher traffic areas based on rider app to compensate drivers more and determine those hours or areas.&nbsp;</p><p style="text-align: justify;">This is where real-time processing takes into account all the riders requesting during a certain period of time and in a certain area to determine surge windows.&nbsp;</p><p style="text-align: justify;">This is all possible using the parallel computation abilities that client applications of Kafka provide where we do large computations. Elk, which is another client application, is in charge of debugging and searching the information for troubleshooting purposes and real-time querying.</p><p style="text-align: justify;">Apache Storm and Apache Samza are other real-time processing APIs to create real-time analytics and alerting abilities by processing the data from Kafka on the consumer side using consumer libraries and connectors.</p><p style="text-align: justify;"><img style="display: block; margin-left: auto; margin-right: auto;" src="https://kradminasset.s3.ap-south-1.amazonaws.com/ExpertViews/Avapi3.PNG" width="460" height="233" /></p><p style="text-align: justify;">&nbsp;</p><h2 style="text-align: justify;"><span style="font-size: 14pt;">Kafka Ecosystem</span>&nbsp;</h2><p style="text-align: justify;">So far, we have touched on the use cases and ideas of real-time processing. Next, we focus on some of the terminologies used when dealing with the implementation and design of Kafka as part of the infrastructure.&nbsp;</p><p style="text-align: justify;">The easiest way to break this infrastructure is to divide it into three pieces: core, component, and data.&nbsp;<br />The core is where the data is stored, components are connected to the core, such as services and connectors in the ecosystem, and data is how we structure information.&nbsp;</p><p style="text-align: justify;">We define these concepts below.&nbsp;</p><p style="text-align: justify;">The system's core is also referred to as Kafka Cluster as it has multiple units that communicate with each other for coordination, data transmission, protocol implementation, and storage.<br />&nbsp;<br /><span style="font-size: 14pt;"><strong>Core&nbsp;</strong></span></p><p style="text-align: justify;"><strong>Brokers&nbsp;</strong><br />Processing units and where data is stored. Where the magic happens, a typical cluster has three brokers, and one is always the controller.&nbsp;</p><p style="text-align: justify;"><strong>Zookeeper&nbsp;</strong><br />The management unit is in charge of managing brokers and leadership of brokers. We usually have three zookeepers in the basic setup.&nbsp;</p><p style="text-align: justify;">&nbsp;</p><p style="text-align: justify;"><span style="font-size: 14pt;"><strong>Data&nbsp;</strong></span></p><p style="text-align: justify;"><strong>Topics&nbsp;</strong><br />Categories that refer to the same type of information, similar to a database table.&nbsp;</p><p style="text-align: justify;"><strong>Partitions&nbsp;</strong><br />Units of parallelism within a topic and how data is being separated across them to enable parallel processing, and faster read/write. There is a direct correlation between a load of a topic and the number of partitions to be able to achieve success.&nbsp;</p><p style="text-align: justify;"><strong>Messages&nbsp;</strong><br />Units of data assigned to a specific topic. Think of it as a database row representing a single information piece.&nbsp;</p><p style="text-align: justify;"><br /><strong>Replication factor&nbsp;</strong></p><p style="text-align: justify;">The degree to which a message is replicated across brokers for the redundancy to provide reliability and resilience in case of broker failures.&nbsp;</p><p style="text-align: justify;"><strong>Consumer offset&nbsp;</strong><br />A pointer is being kept for each consumer within each topic partition to track where we are with the data processing.&nbsp;</p><p style="text-align: justify;">&nbsp;</p><p style="text-align: justify;"><span style="font-size: 14pt;"><strong>Components/Clients&nbsp;</strong></span></p><p style="text-align: justify;"><strong>Producers&nbsp;</strong><br />Clients that read the data&nbsp;</p><p style="text-align: justify;"><strong>Consumers</strong>&nbsp;<br />Clients that write the data&nbsp;</p><p style="text-align: justify;"><strong>Consumer group</strong>&nbsp;<br />A group of consumers that cooperate to consume the same or different topics&nbsp;</p><p style="text-align: justify;"><strong>Connectors(Source/Sink)</strong>&nbsp;<br />Clients that read and write the data suited for large-volume transmission. Installation is configuration-based connectors run on a connected server, and you can have multiple connectors running on a single server working seamlessly interacting with the cluster. Very popular component indeed.&nbsp;<br />They also provide simple transformation functions to modify data on the fly as data passes through.&nbsp;</p><p style="text-align: justify;"><strong>KStreams/KSQL </strong></p><p style="text-align: justify;">Stream libraries for consuming and producing messages using code or querying language Schema&nbsp;</p><p style="text-align: justify;"><strong>Registry</strong>&nbsp;<br />A place to introduce and maintain a schema of topics similar to a database schema. This is for governance, versioning, and having a clean structure when data is structured.&nbsp;</p><p style="text-align: justify;"><strong>Rest Proxy</strong>&nbsp;<br />A client component with limited capabilities to read and write topic data for easier access. As far as language goes for Kafka, Java is the most common language that many choose; however, there are librdkafka and derived clients in C/C++, .NET, and Python available as well.&nbsp;</p><p style="text-align: justify;">&nbsp;</p><h2 style="text-align: justify;"><span style="font-size: 14pt;">Example Code&nbsp;</span></h2><p style="text-align: justify;">In the example code below, we use a simple pipe stream where we read from streams-input and write it to the streams-output topic. The configurations on top are broker address for connectivity and application id for grouping of streams apps. If you are writing an application and want to have multiple instances of the same stream, you need to use the same.</p><p style="text-align: justify;">&nbsp;</p><h2 style="text-align: justify;"><span style="font-size: 14pt;">Future and Data in Evolution</span>&nbsp;</h2><p style="text-align: justify;">We discussed a wide range of use cases of an event-driven architecture, a paradigm shift introduced into the infrastructure as systems became more distributed and more complex and data needed to be moved in massive amounts and more frequently.&nbsp;<br />This concept comes into play in any scalable solution today when we look at the enterprise space across various industries. Ultimately everything comes down to the interactions of software applications with data in terms of management, maintenance, and processing.&nbsp;</p><p style="text-align: justify;">Given the volume of data today besides Kafka, we are seeing various solutions that are dominating the space. Some of these cloud solutions are Databricks and Snowflake, which are highly popular, as well as Pulsar, which mimics some behaviors of Kafka architecture.&nbsp;</p><p style="text-align: justify;">Redpanda is another player in the field that is becoming popular in streaming data and interacting with multiple data points. It works seamlessly with Kafka as well.&nbsp;</p><p style="text-align: justify;">The newer solutions often are focused on addressing performance flaws and optimization challenges and introducing more simplicity and out-of-the-box solutions to the landscape of distributed architecture and event streaming. New design patterns and containerized environments such as Kubernetes(K8) add a new layer of complexity and abstraction to build a complex architecture compared to the past, where infrastructure was more monolithic, flat, and not so externally service oriented.&nbsp;</p><p style="text-align: justify;">These new paradigm shifts bring new innovations and thoughts to the leadership of infrastructure when it comes to managing data in evolution. Thinking about cost vs. long-term and short-term goals, security and safety of data at rest or in transit, managing and monitoring the system, capacity planning, optimized communication between teams, and cross-team leadership all play tactical roles in being able to succeed in managing a roadmap of an event-driven architecture as complexity grows exponentially and new challenges arise.&nbsp;</p><p style="text-align: justify;">Predicting what the future holds in this arena is like looking into a crystal ball, not knowing what questions to ask. The monopoly of tech and data makes moving forward and innovation challenging. On top of that cloud is a beautiful place to innovate and centralize data to operate on it using a service-oriented model; however, this is quite challenging because we have highly regulated spaces such as healthcare, finance, and government that need a privacy-conscious style of handling data.&nbsp;</p><p style="text-align: justify;">These sectors have the most challenging times with efficiency and innovation since they deal with the most critical and time-sensitive private data.&nbsp;</p><p style="text-align: justify;">Public cloud solutions are not even an option on the discussion table often. Security layer of these systems and cybersecurity as an external umbrella or when data is the transmission is challenging, and it can create delays as data is packaged in motion and complicate the architecture.&nbsp;</p><p style="text-align: justify;">Overall, staying a practical optimist and sharing what we learn is the best path forward as we explore solving challenging problems of the modern world and engineering the best solutions of tomorrow for scalability and speed.</p><p style="text-align: justify;">&nbsp;</p><p style="text-align: justify;">&nbsp;</p><p style="text-align: justify;">&nbsp;</p>