uLinga for Kafka – High Performance, Fault-Tolerant Kafka Streaming

Introduction

HPE NonStop users are becoming more familiar with Kafka, as Kafka in turn becomes more and more prevalent inside large enterprise environments. Kafka is now used by thousands of companies, including 60% of the Fortune 100, and many NonStop shops. Well-known users in the finance space include Goldman Sachs, Rabobank, Barclays, Jack Henry and PayPal, and household names that use Kafka include Netflix, Oracle, LinkedIn, and AirBnB. A full list of Kafka users is available here.

These organizations use Kafka to manage “streams” of data, which have become prevalent as internet usage massively boosts the amount of data being generated and needing to be processed. Kafka allows these huge volumes of data to be processed in real-time, via a combination of “producers” and “consumers”, which work with a Kafka “cluster” – the main data repository.

NonStop Servers have valuable data on them that users may want to stream to Kafka, and NonStop applications may also have a need to send their data directly to Kafka. In these situations, NonStop users need a reliable, high-performing way to get that data streamed to Kafka. This is where uLinga for Kafka comes in.

uLinga for Kafka – Overview

uLinga for Kafka is the latest addition to Infrasoft’s uLinga product range, a solution suite that has been used by large banks, telcos, and manufacturers to provide reliable mission-critical communications infrastructure for many years. uLinga for Kafka brings the same performance, scalability, security, and manageability to the Kafka space as users have successfully utilized with the other uLinga products.

uLinga for Kafka takes a unique approach to Kafka integration: it runs as a natively compiled Guardian process pair and supports the Kafka communications protocols directly over TCP/IP. This removes the need for Java libraries or intermediate databases, providing the best possible performance on HPE NonStop. It also allows uLinga for Kafka to directly communicate with the Kafka cluster, getting streamed data across as quickly and reliably as possible.

This approach also allows uLinga to provide some unique fault-tolerance features, as we’ll explore later in the article.

uLinga for Kafka – Application Integration

uLinga for Kafka (uLinga) supports a range of options to communicate with HPE NonStop applications and read/process NonStop data. Applications can use standard Guardian IPC messages, or Pathsend requests, to send data to uLinga, and HTTP clients can send data via uLinga’s inbuilt HTTP interface. uLinga can process disk records directly from Enscribe files, and as of the most recent product release, also supports TMF Audit Trails, meaning updates to all TMF-protected files can be streamed by uLinga to Kafka.

Figure 1. uLinga for Kafka Accessing TMF Audit Trail Data

Fault Tolerance and Reliability

There are a number of features within the uLinga for Kafka implementation that ensures it is one of the most reliable Kafka solutions available.

FILEREADER and AUDITREADER Guaranteed Delivery

When processing Enscribe files or TMF audit trails, uLinga has a unique method of guaranteeing delivery of data to Kafka. Whenever an Enscribe or TMF audit trail record on NonStop is read and streamed to Kafka, the record sent by uLinga includes a Kafka header that contains a record key. That record key allows uLinga to locate that record in the original NonStop file (either Enscribe or TMF audit trail). If for some reason streaming to Kafka is interrupted – because the Kafka cluster is down, or the network connection is unavailable, or even if the uLinga process stops – uLinga will recommence streaming records whenever it is able to restart. When that restart begins, uLinga will query the Kafka cluster to determine the most recent record that was streamed, and will automatically start reading from that point in the Enscribe file or TMF audit trail. This ensures that all records are sent to Kafka, even following an outage.

Kafka EOS Support

Along with this, uLinga supports Kafka’s exactly-once semantics (EOS). EOS is a framework that allows applications to stream data to Kafka without loss or duplication, ensuring that messages are always sent to Kafka, even if a network connection to the cluster fails.

Checkpointing

Finally, like all uLinga products, uLinga for Kafka optionally runs as a NonStop process pair. When run in this manner, it can be configured to checkpoint data at various points in uLinga’s processing. If we consider an application that is writing data to uLinga via uLinga’s IPCSERVER interface, we would see a configuration similar to this:

Figure 2. uLinga for Kafka’s IPCSERVER interface

The IPCSERVER resource presents an application interface for a NonStop process to communicate with via $RECEIVE. The KAFKAPRODUCER resource is configured with Kafka-specific information, including broker IP address and port, and Kafka Topic name. Checkpointing can be configured on both the IPCSERVER and the KAFKAPRODUCER resources. IPCSERVER checkpointing will ensure that all data received and sent between the Guardian (or Pathsend process) is checkpointed to the uLinga backup process. KAFKAPRODUCER checkpointing will do the same thing for all data sent to/received from the Kafka cluster. Depending on the application environment, checkpointing may be required at one or both points.

In the example depicted below, uLinga for Kafka is running as a NonStop process pair, with checkpointing configured on the IPCSERVER resource. All data flowing through the IPCSERVER resource is checkpointed to the backup process, as indicated by the arrow.

Figure 3. uLinga Running as a NonStop Process Pair with Checkpointing

In the event of the main process dying, due to a CPU failure, or similar, the backup process will take over, and the operating system will redirect the Guardian process or Pathsend Client to the backup process. This happens seamlessly, and transparently to the Guardian/Pathsend process.

Figure 4. uLinga Backup Process Takeover

No data will be lost, and the Guardian/Pathsend process is able to continue processing data without any outage. uLinga will also re-start the process that died when it is able to, and that process then becomes the backup.

High Performance

Because uLinga for Kafka is written in C, and does not require any additional libraries or interim servers, it is able to achieve impressive performance figures, with extremely low latency.

Infrasoft testing has shown that the product is able to exceed 25,000 TPS with sub-millisecond latency. Perhaps more importantly given most NonStops are used in OLTP environments, when running at a steady state of 1000TPS, uLinga for Kafka has a very low CPU requirement. This will allow uLinga for Kafka to process data from the busiest of NonStop OLTP applications with minimal overhead.

uLinga for Kafka is now available. Please contact [email protected] for more information or to arrange a trial.