Collecting Java Runtime, System- and Process-Metrics using OpenTelemetry Java Auto Instrumentation
With the release of version 1.0.0 the OpenTelemetry Java Auto Instrumentation now supports collecting system- and process metrics. In this blog post we will show you how to enable the collection of those metrics and how to expose them via a Prometheus endpoint.
Introduction
OpenTelemetry is an open-source observability framework that aims to define standards for collecting distributed traces, metrics and logs as well as provide instrumentations with extensive support for common technologies [https://opentelemetry.io/docs/java/automatic_instrumentation/] to automatically do so. Resulting from the merge of OpenTracing and OpenCensus and being part of the Cloud Native Computing Foundation, various Cloud- and APM-providers already offer support and integration of OpenTelemetry [https://www.dynatrace.com/integrations/opentelemetry/, https://www.dynatrace.com/monitoring/integrations/opentelemetry/]. In our virtual 23. Meetup last November we demoed how a Java application can be instrumented using the OpenTelemetry Java Auto Instrumentation with very little effort to collect distributed traces. Based on Willie Wheeler’s article and corresponding demo “Auto-Instrumentation with OpenTelemetry” [https://medium.com/wwblog/auto-instrumentation-with-opentelemetry-3b096fdd068f], we demonstrated how to collect traces, but did not show how to collect system- and process metrics at the time. With recent releases this capability was introduced in the OpenTelemetry Java Auto Instrumentation. Here we will show you how to enable the collection of process- and system-metrics and how to expose them via a Prometheus endpoint. A disclaimer: OpenTelemetry is very much in relatively early active development and there are issues to be resolved – in case of Java process metrics the consumed memory metric and consumed CPU time metric are broken: [https://github.com/open-telemetry/opentelemetry-java-instrumentation/issues/2231]. This article refers to v1.0.0 of the OpenTelemetry Java Auto Instrumentation.
Auto Instrumentation Metrics
As of release 1.0.0, OpenTelemetry’s Java Auto Instrumentation supports collecting three types of metrics: Java runtime metrics, system-metrics and process-metrics.
Java Runtime Metrics
These metrics collect information on Java garbage collection and memory pools. For garbage collection, the overall runtime is observed while the memory metrics observe the used, committed, and maximum storage of the memory pools of a Java runtime.
System and Process Metrics
System metrics encompass statistics on memory and network io. The process metrics collect memory and CPU usage of the Java process.
Setting up OpenTelemetry instrumentation
OpenTelemetry’s Java Auto Instrumentation is realized through a Java agent, which is available on the Github repository [https://github.com/open-telemetry/opentelemetry-java-instrumentation/releases]. The agent is embedded into a Java process using the -javaagent flag:
Copy to Clipboard
java -javaagent:path/to/opentelemetry-javaagent-all.jar \
-jar myapp.jar
This agent automatically starts recording traces in your Java process and can be configured to export them to e.g. Jaeger via Java system properties (-Dkey=value parameters). In the below example, traces would be exported to a Jaeger collector listening at port 14250 of host “jaeger”:
Copy to Clipboard
java -javaagent:path/to/opentelemetry-javaagent-all.jar \
-Dotel.traces.exporter=jaeger \
-Dotel.exporter.jager.endppoint=http://jaeger:14250 \
-Dotel.resource.attributes=service.name=myapp \
-jar myapp.jar
This activates the exporter to export traces to a Jaeger collector listening on http://jaeger:14250 under the application name “myapp”.
In order to „export“ metrics to Prometheus, a Prometheus exporter must be configured:
Copy to Clipboard
java -javaagent:path/to/opentelemetry-javaagent-all.jar \
-Dotel.metrics.exporter=prometheus \
-Dotel.exporter.prometheus.host=0.0.0.0\
-Dotel.exporter.prometheus.port=9464 \
-jar myapp.jar
While an exporter in OpenTelemetry sends data to the configured endpoint, the Prometheus exporter is an exception, inverting the flow of communication. It exposes a Prometheus endpoint under the configured port and (locally available) host, which can then be scraped by a Prometheus server.
Traces are collected by the OpenTelmetry Java Auto Instrumentation by merely configuring it as shown above and some debugging metrics are already computed and provided via the configured Prometheus endpoint without any further configuration. To start collecting and exporting Java runtime, system- and process-metrics however, a few more steps are necessary.
Configuring metrics collection
Activating Java Runtime Metrics
All that is necessary to start collecting basic Java runtime metrics is to add the property “otel.instrumentation.runtime-metrics.enabled=true”:
Copy to Clipboard
java -javaagent:path/to/opentelemetry-javaagent-all.jar \
-Dotel.metrics.exporter=prometheus \
-Dotel.exporter.prometheus.host=0.0.0.0\
-Dotel.exporter.prometheus.port=9464
-Dotel.instrumentation.runtime-metrics.enabled=true \
-jar myapp.jar
Passing the otel.instrumentation.runtime-metrics.enabled=true system property will activate collection of the java runtime metrics. On the host where the application is executed, navigating to the Prometheus endpoint at http://localhost:9464/metrics will return these garbage collection and memory pool metrics:
Copy to Clipboard
# HELP processedSpans The number of spans processed by the BatchSpanProcessor. [dropped=true if they were dropped due to high throughput]
# TYPE processedSpans counter
processedSpans_total{dropped="false",spanProcessorType="BatchSpanProcessor",} 83.0
# HELP queueSize The number of spans queued
# TYPE queueSize gauge
queueSize{spanProcessorType="BatchSpanProcessor",} 0.0
# HELP runtime_jvm_memory_pool Bytes of a given JVM memory pool.
# TYPE runtime_jvm_memory_pool gauge
runtime_jvm_memory_pool{pool="Compressed Class Space",type="committed",} 8519680.0
runtime_jvm_memory_pool{pool="CodeHeap 'non-profiled nmethods'",type="used",} 5179904.0
runtime_jvm_memory_pool{pool="CodeHeap 'profiled nmethods'",type="committed",} 1.5663104E7
runtime_jvm_memory_pool{pool="CodeHeap 'non-nmethods'",type="max",} 5836800.0
runtime_jvm_memory_pool{pool="Metaspace",type="committed",} 6.5404928E7
runtime_jvm_memory_pool{pool="G1 Eden Space",type="used",} 5.24288E7
runtime_jvm_memory_pool{pool="G1 Old Gen",type="max",} 2.147483648E9
runtime_jvm_memory_pool{pool="Compressed Class Space",type="used",} 7933832.0
runtime_jvm_memory_pool{pool="CodeHeap 'non-nmethods'",type="used",} 1251712.0
runtime_jvm_memory_pool{pool="G1 Survivor Space",type="used",} 1.3631488E7
runtime_jvm_memory_pool{pool="CodeHeap 'profiled nmethods'",type="max",} 1.22908672E8
runtime_jvm_memory_pool{pool="CodeHeap 'profiled nmethods'",type="used",} 1.5660288E7
runtime_jvm_memory_pool{pool="Compressed Class Space",type="max",} 1.073741824E9
runtime_jvm_memory_pool{pool="G1 Old Gen",type="used",} 2.414644E7
runtime_jvm_memory_pool{pool="G1 Survivor Space",type="committed",} 1.3631488E7
runtime_jvm_memory_pool{pool="CodeHeap 'non-profiled nmethods'",type="max",} 1.22912768E8
runtime_jvm_memory_pool{pool="CodeHeap 'non-nmethods'",type="committed",} 2555904.0
runtime_jvm_memory_pool{pool="G1 Old Gen",type="committed",} 1.894776832E9
runtime_jvm_memory_pool{pool="G1 Eden Space",type="committed",} 2.39075328E8
runtime_jvm_memory_pool{pool="CodeHeap 'non-profiled nmethods'",type="committed",} 5242880.0
runtime_jvm_memory_pool{pool="Metaspace",type="used",} 6.3689456E7
# HELP runtime_jvm_memory_area Bytes of a given JVM memory area.
# TYPE runtime_jvm_memory_area gauge
runtime_jvm_memory_area{area="non_heap",type="committed",} 9.7386496E7
runtime_jvm_memory_area{area="heap",type="committed",} 2.147483648E9
runtime_jvm_memory_area{area="heap",type="max",} 2.147483648E9
runtime_jvm_memory_area{area="non_heap",type="used",} 9.3715192E7
runtime_jvm_memory_area{area="heap",type="used",} 9.0206728E7
# HELP runtime_jvm_gc_collection Time spent in a given JVM garbage collector in milliseconds.
# TYPE runtime_jvm_gc_collection counter
runtime_jvm_gc_collection_total{gc="G1 Old Generation",} 0.0
runtime_jvm_gc_collection_total{gc="G1 Young Generation",} 673.0
Activating System and Process Metrics
The collection of system- and process-metrics in OpenTelemetry’s Java Auto Instrumentation relies on the OSHI library [Operating System and Hardware Information, https://github.com/oshi/oshi]. In theory, the metrics collection ought to be activated if Oshi is found on the classpath [https://github.com/open-telemetry/opentelemetry-java-instrumentation/blob/f8dd8c8f561240392ce3f4b17dd9caeecaa0499b/javaagent-bootstrap/src/main/java/io/opentelemetry/javaagent/OpenTelemetryAgent.java#L71]
Adding the Oshi library to the classpath alone however did not work in our tests and documentation on the feature is pending [https://github.com/open-telemetry/opentelemetry-java-instrumentation/issues/1566]. But it is still possible to start collecting these metrics through a workaround with additional configuration, although it requires modification to the application. Two dependencies are required:
Copy to Clipboard
io.opentelemetry.instrumentation
opentelemetry-oshi
0.16.1
compile
com.github.oshi
oshi-core
5.3.1
The opentelemetry-oshi is the part of the OpenTelemetry Java Agent that contains the implementation of the process- and systems-metrics collection, while oshi-core adds OSHI itself. To register the metrics the observers, call SystemMetrics.registerObservers() and ProcessMetrics.registerObservers() once. Ideally at startup, e.g. as in this dummy Spring Boot application:
Copy to Clipboard
package com.your.spring.boot.app;
import io.opentelemetry.instrumentation.oshi.ProcessMetrics;
import io.opentelemetry.instrumentation.oshi.SystemMetrics;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
@SpringBootApplication
public class YourApplication {
static {
SystemMetrics.registerObservers();
ProcessMetrics.registerObservers();
}
public static void main(String[] args) {
SpringApplication.run(YourApplication.class, args);
}
As with the Java runtime metrics, the System and Process metrics can be scraped from the configured Prometheus endpoint, e.g. https://localhost:9464/metrics:
Copy to Clipboard
# HELP processedSpans The number of spans processed by the BatchSpanProcessor. [dropped=true if they were dropped due to high throughput]
# TYPE processedSpans counter
processedSpans{dropped="false",spanProcessorType="BatchSpanProcessor",} 33.0
# HELP queueSize The number of spans queued
# TYPE queueSize gauge
queueSize{spanProcessorType="BatchSpanProcessor",} 0.0
# HELP system_memory_utilization System memory utilization
# TYPE system_memory_utilization gauge
system_memory_utilization{state="used",} 0.24315197243730013
system_memory_utilization{state="free",} 0.7568480275626999
# HELP system_network_packets System network packets
# TYPE system_network_packets counter
system_network_packets{device="eth0",direction="receive",} 195.0
system_network_packets{device="eth0",direction="transmit",} 163.0
# HELP system_disk_operations System disk operations
# TYPE system_disk_operations counter
system_disk_operations{device="/dev/sdc",direction="read",} 14362.0
system_disk_operations{device="/dev/sdb",direction="read",} 436.0
system_disk_operations{device="/dev/sda",direction="read",} 42.0
system_disk_operations{device="/dev/sda",direction="write",} 8316.0
system_disk_operations{device="/dev/sdb",direction="write",} 79.0
system_disk_operations{device="/dev/sdc",direction="write",} 8263.0
# HELP system_disk_io System disk IO
# TYPE system_disk_io counter
system_disk_io{device="/dev/sdc",direction="read",} 4.0380928E8
system_disk_io{device="/dev/sdb",direction="read",} 2.5220096E7
system_disk_io{device="/dev/sda",direction="read",} 168960.0
system_disk_io{device="/dev/sda",direction="write",} 4.296175616E9
system_disk_io{device="/dev/sdb",direction="write",} 561152.0
system_disk_io{device="/dev/sdc",direction="write",} 1.326931968E9
# HELP system_network_io System network IO
# TYPE system_network_io counter
system_network_io{device="eth0",direction="receive",} 30172.0
system_network_io{device="eth0",direction="transmit",} 56757.0
# HELP system_memory_usage System memory usage
# TYPE system_memory_usage gauge
system_memory_usage{state="used",} 4.869128192E9
system_memory_usage{state="free",} 1.5155912704E10
# HELP system_network_errors System network errors
# TYPE system_network_errors counter
system_network_errors{device="eth0",direction="receive",} 0.0
system_network_errors{device="eth0",direction="transmit",} 0.0
# HELP runtime_java_cpu_time Runtime Java CPU time
# TYPE runtime_java_cpu_time gauge
runtime_java_cpu_time{type="user",} 9.934E7
runtime_java_cpu_time{type="system",} 9090000.0
# HELP runtime_java_memory Runtime Java memory
# TYPE runtime_java_memory gauge
runtime_java_memory{type="vms",} 7.97949952E9
runtime_java_memory{type="rss",} 6.92097024E8
# HELP system_memory_utilization System memory utilization
# TYPE system_memory_utilization gauge
system_memory_utilization{state="used",} 0.24315197243730013
system_memory_utilization{state="free",} 0.7568480275626999
# HELP system_network_packets System network packets
# TYPE system_network_packets counter
system_network_packets{device="eth0",direction="receive",} 195.0
system_network_packets{device="eth0",direction="transmit",} 163.0
# HELP system_disk_operations System disk operations
# TYPE system_disk_operations counter
system_disk_operations{device="/dev/sdc",direction="read",} 14362.0
system_disk_operations{device="/dev/sdb",direction="read",} 436.0
system_disk_operations{device="/dev/sda",direction="read",} 42.0
system_disk_operations{device="/dev/sda",direction="write",} 8316.0
system_disk_operations{device="/dev/sdb",direction="write",} 79.0
system_disk_operations{device="/dev/sdc",direction="write",} 8263.0
# HELP system_disk_io System disk IO
# TYPE system_disk_io counter
system_disk_io{device="/dev/sdc",direction="read",} 4.0380928E8
system_disk_io{device="/dev/sdb",direction="read",} 2.5220096E7
system_disk_io{device="/dev/sda",direction="read",} 168960.0
system_disk_io{device="/dev/sda",direction="write",} 4.296175616E9
system_disk_io{device="/dev/sdb",direction="write",} 561152.0
system_disk_io{device="/dev/sdc",direction="write",} 1.326931968E9
# HELP system_network_io System network IO
# TYPE system_network_io counter
system_network_io{device="eth0",direction="receive",} 30172.0
system_network_io{device="eth0",direction="transmit",} 56757.0
# HELP system_memory_usage System memory usage
# TYPE system_memory_usage gauge
system_memory_usage{state="used",} 4.869128192E9
system_memory_usage{state="free",} 1.5155912704E10
# HELP system_network_errors System network errors
# TYPE system_network_errors counter
system_network_errors{device="eth0",direction="receive",} 0.0
system_network_errors{device="eth0",direction="transmit",} 0.0
As mentioned in the beginning, runtime_java_cpu_time, which ought to report the CPU time spent in seconds is broken, as it is being tracked as a series of snapshots of a value in time, while the underlying value reported by OSHI is the sum of CPU time consumed by the process. Unit conversion is also incorrect, since the underlying millisecond value is multiplied by thousand, whereas it ought to be divided. On the other hand, runtime_java_memory, which ought to track memory consumption over time, is being tracked as a continually rising sum value instead of a series of snapshots of the amount of consumed memory over time:
[https://github.com/open-telemetry/opentelemetry-java-instrumentation/issues/2231]
Conclusion
We have shown that starting to collect distributed traces and Java runtime metrics using the OpenTelemetry Java Auto Instrumentation is possible with minimal setup and requires no code adaption. Basic system- and process-metrics using the OpenTelemetry Java Auto Instrumentation are available as well, but require a few more steps to collect, including minor code adaption. This combination of tracing and metrics already provides fundamental insights into your application’s performance characteristics.