-
Reducing Kafka Lag: Optimizing Kafka Performance
Apache Kafka is a powerful distributed messaging system, but like any system, it can face performance bottlenecks. One of the most common challenges is Kafka lag—the delay between producing and consuming messages. Addressing Kafka lag is crucial for maintaining real-time data pipelines and ensuring optimal performance. In this article, we explore practical strategies to reduce…
-
KRaft: Apache Kafka Without ZooKeeper
Apache Kafka has been a cornerstone of modern event streaming architectures, enabling reliable and scalable data pipelines for businesses worldwide. Traditionally, Kafka has relied on ZooKeeper for managing metadata, configurations, and cluster coordination. However, the introduction of KRaft (Kafka Raft) marks a significant shift in Kafka’s architecture, eliminating the need for ZooKeeper and simplifying cluster management. What…
-
Fluentd: How to Use a Parser With Regular Expression (regexp)
This guide explains configuring Fluentd to extract structured data from unstructured log messages using the parser plugin with a regular expression (regexp). If you need to extract specific fields, such as log_source and index, from a log message, you can do this as follows. Input Log: { “message”: “Log source ‘WinCollect DSM – SRV-AD-001’ has…
-
Using Kafka as a Fast Correlation Engine
In this article, we explore how Kafka Streams can be utilized for filtering and correlating events in real time, effectively transforming Kafka into a high-speed correlation engine. By leveraging the capabilities of ksqlDB, you can deploy content rules and filter alerts directly within Kafka. This approach enables real-time filtration and aggregation of log event flows using…