Measuring and Monitoring Java Applications on HPE NonStop

“Data is like a faint light when you’re lost in a dark room. Follow it, try to make sense of it, and you might actually know where you are and what’s around you.” — David Sides

I like this quote because a single piece of data can sometimes be the missing link that connects the problem to the solution.  To truly harness the power of data, we need to

1.  collect concrete data quantifying a specific attribute, typically numerical data or metrics that can be easily analyzed and compared against predefined standards.
2. collect data over time to transform the data into information, for example, identifying patterns and trends which may not be directly quantifiable
3. collect and collate diverse types of data, analyze patterns and anomalies, thus connecting pieces of information to get the right insight

The verbs for the above are Measure, Monitor and Observe respectively. These three crucial activities are used to gain insights into complex systems and aid in proactive decision making. While observability is a broader term that requires additional capabilities, measuring and monitoring are foundational pieces of observability.

In this article, we will discuss Measuring and Monitoring Java Applications on HPE NonStop.

Measurement in NonStop

NonStop provides “performance management bundle”, a set of software tools to collect, manage, analyze, and report NonStop server performance data.  A major part of the NonStop performance management bundle is NonStop Measure.

With NonStop Measure, we can gather information from applications as well as servers and network components. Typically, the data collected by NonStop Measure, is used to tune the server, detect bottlenecks , unbalanced workloads as well as for  sizing the resources needed for new applications and capacity planning.

NonStop Measure is used for continuous measurements as the overhead is low. Most system components/subsystems and applications in the NonStop Guardian environment have well-defined integration with measure. Most problems can be identified, and proactive measures can be taken using the NonStop Measure. However, open-source runtimes like Java and Python on NonStop OSS have their own measurement and monitoring libraries, tools, and frameworks as part of the language ecosystem. While most of these can be used on the NonStop platform, owing to the platform independence, the data emanating from these cannot be visualized in the NonStop Measure.

Typically, Java applications on NonStop exercise the entire NonStop OS stack along with using transactions subsystem, databases and network.  For an engineer, collating the data from Java applications and underlying system metrics from  NonStop Measure is time-consuming. They must deal with multiple data collection and visualization tools, each with its own data format. Not only is precision lost, but this exercise also leads to delays in linking various pieces of information to uncover patterns and relationships. Due to the disconnect, the visualization is not holistic, and so the right insights are not available.

So, the first bridge would be the ability to quantify data from Java applications, that is, Java Metrics in NonStop Measure.

Metrics and Types

Before we see how to quantify metrics from Java Applications, let us understand some terminology in HPE NonStop Measure and Java

Entity – In NonStop Measure, the resource to be measured is called an entity. Example PROCESS is an entity.

Counter- Each entity has a set of pre-defined counters that hold the measurement data. Counters are programmatic structure that collects a specific type of performance data. Continuing the same example CPU-BUSY-TIME is a counter that gives the aggregate of the BUSY-TIMEs of the individual IPUs that comprise the logical CPU.

While tightly integrated subsystems have predefined entities and counters, an entity called USER-DEF is a provision in NonStop Measure to measure activity in a user process. Typically designed to be used for applications that are not integrated with NonStop Measure, the USER_DEF entity is flexible and extensible design feature and can be used for Java Applications.

The USER-DEF Entity allows for three types of Counters, namely

1. Incrementing counters count events. The counter is advanced each time the event occurs. For example, to get the number of times a method is executed.
2. Queue counters measure the total amount of time that all queued elements spend on the queue. For example, to get how long a method executed.
3. Snapshot counters return a measurement value taken at a specific moment

Metrics in Java applications are broadly classified as

1. Increment-only metrics are used to track the occurrence of specific events over time.
2. Timers measure the duration of specific operations or processes within an application, offering insights into performance and latency.
3. Gauges measure the current value of a metric at a particular point in time, such as the current number of active threads.

The obvious correlation of Java metrics to the NonStop Measure’s USER-DEF counter, makes it an obvious choice for defining application specific Java metrics to NonStop Measure.

Java Wrappers around Measure USER-DEF

As a part of Middleware Tools (T1150), Java Wrappers around Measure USER-DEF are made available through library MeasureUtility-1.0.jar . This library has an Object representation of Measure USER-DEF Entity. Each USER-DEF consists of one or more counters. FACCUM, QUEUE and FSNAPSHOT representing the Incrementing, Queue and Snapshot counters.  This library provides CRUD operations on the counter. That is, it allows for adding, listing, updating, and removing counters.

Usage

A simple usage is demonstrated below

The API docs are available under “docs” folder, the sample programs demonstrating the library usage is available in the “demo” folder of the product installation (typically /usr/tandem/middlewaretools). The README file in the “demo’ folder provides the details of how to use the library and sample.

Thus, the MeasureUtility library in MiddlewareTools allows for defining Java Metrics in Measure, publishing, and visualization of the metrics.

Monitoring Java Applications

The standard definition of monitoring states that it is the act of collecting data and observing the performance of a system over time and involves tracking specific metrics and thresholds, alerting when predefined conditions are met. There are multiple tools, such as profilers, which profile JVM, memory and runtime metrics, transaction profilers, and APM tools. The CPU, PROCESS, PROCESSH, FILE, TMF, SQLPROC & SQLSTMT entities in Measure provide a good amount of metrics to monitor Java Applications on NonStop. However, it is important to see these metrics in correlation to the application-level metrics to provide visibility into the external behaviour.

While the USER-DEF entity of the NonStop Measure and MeasureUtility library provides a mechanism to define and publish (that is, how of ) application-level metrics, what, where, and when (metrics data generation) are crucial.

If we take a step back and see the bigger picture, any application for that matter provides a set of services (executable element of the code)  that act on sets of data. In a programming language like Java, the “method” is the executable element of the code and data can be explicit in nature contained in class global variables, local variables, return values from methods and input parameters to a method. The implicit data is the number of times a service, i.e., the method is called and the average time spent executing the method. Mapping them to the USER-DEF entity, the explicit data are Snapshot counters, and the implicit data are Incrementing counters and Queue counters. The triggers to collect data is obviously at the start and end of the service that is at the start and end of the method.

NonStop Application Monitoring Facility

Using the above concept, NonStop Application Monitoring framework is conceived as a Java annotation-based framework for application-level metrics generation.  The NSAMF (NSAMFUtility-1.0.jar) is part of Middleware Tools (T1150) and has the following components.

1. NSProductMonitor– an application-agnostic interface to define (what, where & when) the annotations in applications for metrics
2. NSAnnotationProcessor – An annotation process to process any code annotated with NSProductMonitor to generate and publish application-level metrics
3. NSJavaAppMonitoringPublisher – A generic interface that any data collector/publisher tool or any time series database can extend to get metrics from Java applications. A Measure based publisher is available as a part of NSAMFUtility-1.0.jar.

The finer details of each of  the above is available in the Middleware Tools Reference Manual under the Application Monitoring Facility section.

Using NSAMF

The entire process of using NSAMF is captured in this diagram. The key steps are

Adding annotations for statistics collection

Use the NSProductMonitor to add annotations to the Java Application above the method definition.

The ID is used to generate a unique USER-DEF counter name. MVAR, LVAR, IPARAM, and CALL are keywords for specifying a member variable, local variable, input parameters, and return value. To specify the actual variable in the application code, Expression Language (EL) syntax is used. This EL is processed by the annotation processor.

For example, to watch a member variable of a class in a method getHits, the annotations will be given below.

Compiling with the Annotation Processor

Once the sites of application metric generation are marked up in the code, the code must be compiled with NSAnnotationProcessor either using the processorpath  and processor options of the javac compiler or using <annotationProcessors> tag of the maven-compiler-plugin.

Publishing metrics to  data collector or time series database

By default, NSMAF does not publish data. Using dependency injection, either updating the jar or using maven at build time, the publisher can be mentioned.

For more details, refer to Using the NonStop Application Monitoring Facility

Usage

A famous quote, “Nothing has more strength than dire necessity,” is true in the case of Measuring and Monitoring Java Applications on NonStop.  What started out as a necessity for NonStop Middleware is extended to user applications and workloads as well through the Middleware Tools products. The MeasureUtility and NSAMF are used by middleware products.  To quote specific examples,

1. 1. API Gateway – Users of API Gateway  can also include their custom processors into the route.

Challenge –  The amount of time taken by each processor contributes to the response time perceived by the clients of API Gateway.

Solution – The MeasureUtility library was extended to a pluggable MeasureProcessor. MeasureProcessor is used to  measure the time consumed by each custom processor and how it affects the overall response time.

MeasureProcessor  by itself doesn’t add any overhead during the performance measurement and does not impact the functionality. Further it publishes statistical data only when measurement is enabled. Multiple instances can co-exist. These are  the reasons why API Gateway chose the MeasureUtility library.

1. 2. Shared Statement Caching Datasource – SSCD is a library that implements the statement-sharing feature. It shares statements across SSCD connections and manages transactions on behalf of the JDBC library

Challenge – Some of the challenges include tracking cache information like number of statements in the cache, number of copies of each statement, number of times statement is retrieved from cache and time taken for the same, uniquely identify method execution per statement and most importantly storing non-numerical data such connection objects for monitoring.

Solution – Using NonStop Application Monitoring Facility (NSAMF) and annotating SSCD helped solve the challenges. The annotations with the unique ID helped in tracking cache statistics. The pluggable architecture to configure publishers helped in plugging-in the right publisher for non-numerical data. The Expression language syntax and managing EL expressions helped to generate complex data statistics which were hither to not possible.

To conclude, for Measuring and Monitoring Java Applications on NonStop, infrastructure is available in Middleware Tools in the form of MeasureUtility and NSAMF. These are open to be used by customer applications.