uLinga for Kafka – Support for Custom Code via USEREXITS

Introduction

HPE NonStop users are becoming more familiar with Kafka, as Kafka in turn becomes more and more prevalent inside those large environments where the NonStop platform is often present. Kafka is being used by thousands of companies, including 60% of the Fortune 100.

These organizations use Kafka to manage “streams” of data, which have become more common as internet usage massively boosts the amount of data being generated and needing to be processed. Kafka allows these huge volumes of data to be processed in real-time, via a combination of “producers” and “consumers”, which work with a Kafka “cluster” – the main data repository.

NonStop Servers have valuable data on them that users may want to stream to Kafka, and NonStop applications may also have a need to send their data directly to Kafka. In these situations, NonStop users need a reliable, high-performing way to get that data streamed to Kafka. This is where uLinga for Kafka comes in.

 

uLinga for Kafka – Overview

uLinga for Kafka is the latest addition to Infrasoft’s uLinga product range, a solution suite that has been used by large banks, telcos, and manufacturers to provide reliable mission-critical communications infrastructure for many years. uLinga for Kafka brings the same performance, scalability, security, and manageability to the Kafka space as users have successfully utilized the other uLinga products.

uLinga for Kafka takes a unique approach to Kafka integration: it runs as a natively compiled Guardian process pair and supports the Kafka communications protocols directly over TCP/IP. This removes the need for Java libraries or intermediate databases, providing the best possible performance on HPE NonStop. It also allows uLinga for Kafka to directly communicate with the Kafka cluster, getting streamed data across as quickly and reliably as possible.

With the latest release of uLinga for Kafka now available, a number of new key features have been implemented. The product has once again been benchmarked and proven to perform at massive volumes – over 25,000 TPS running on a single core of a NonStop X3 processor. This article provides a refresher on uLinga for Kafka and summarises those new features and performance figures.

 

uLinga for Kafka – Application Integration

uLinga for Kafka (uLinga) supports a range of options to communicate with HPE NonStop applications and read/process NonStop data. Applications can use standard Guardian IPC messages, or Pathsend requests, to send data to uLinga, and HTTP clients can send data via uLinga’s inbuilt HTTP interface. uLinga can process disk records directly from Enscribe files, and as of the most recent product release, also supports TMF Audit Trails, meaning updates to all TMF-protected Enscribe files can be streamed by uLinga to Kafka.

Figure 1. uLinga for Kafka Accessing TMF Audit Trail Data

User Exit Support

The most recent release of uLinga for Kafka includes support for customer-written user exit processing. Custom user exit code can manipulate/transform data before it is sent to Kafka, or can conditionally determine whether a particular record should be sent to Kafka or omitted. The functionality implemented by the custom code is entirely up to the customer – virtually anything is possible.

Custom user exit functionality can be invoked from a number of configuration points within uLinga. Consider the following implementation, where uLinga is reading/streaming data from an Enscribe file using its FILEREADER capability:

Figure 2. NonStop Application Sending Data to uLinga over $RECIEVE

In this configuration, a uLinga USEREXIT could be invoked from the IPCSERVER, from the KAFKAPRODUCER, or both. This gives significant flexibility in terms of where customer-written code is executed.

If uLinga was configured to monitor two different sets of Enscribe files, it would have two different FILEREADER resources. In this case, you could configure a different USEREXIT for each FILEREADER, providing different functionality for each file being processed.

Figure 3. Different Custom Code for Different Enscribe Files

If you wanted all data being streamed to a single topic to have the same custom code logic applied, you could configure a USEREXIT at the KAFKAPRODUCER. This would ensure that all data, regardless of the origin, would have the same custom code invoked.

Figure 4. Common Custom Code for Single Kafka Topic

Customer-written user exit code can be managed by uLinga for Kafka, using uLinga’s Worker Farm framework. Customer-written code can also be run as a Pathway server class, and can even be run on a remote HTTP server, accessed by uLinga’s built-in HTTP support.

Figure 5. Custom User Exit Code running on an Off-board Web Server

High Performance

Because uLinga for Kafka is written in C, and does not require any additional libraries or interim servers, it is able to achieve impressive performance figures, with extremely low latency.

Infrasoft testing has shown that the product is able to exceed 25,000 TPS with sub-millisecond latency, running in a single core of a NonStop X3 processor. Perhaps more importantly given most NonStops are used in OLTP environments when running at a steady state of 1000TPS, uLinga for Kafka has a very low CPU requirement. This will allow uLinga for Kafka to process data from the busiest of NonStop OLTP applications with minimal overhead.

uLinga for Kafka is now available. Please contact productinfo@infrasoft.com.au for more information or to arrange a trial.

 

Author

  • Andrew Price

    Andrew Price has worked on the NonStop for his entire career. He spent most of the 90s as a BASE24 developer at different banks in different countries, then many years at Insession and ACI. He’s also worked at XYPRO and most recently served as NuWave Technologies’ Chief Operating Officer. He has extensive experience in most aspects of NonStop software development and sales, having been a coder, product manager, sales support technician, and engineering manager. He has been with Infrasoft since January 2020 where he is Director of Business Operations. You can connect with him at https://www.linkedin.com/in/andrew-g-price/ or on Twitter @andrewgprice