Remove duplicated instrumentation scopes and increase nr of produced OTel metrics

[brunobat]

Fixes: #46535

• Also improves test reliability, hopefully.
• More tests
• Improved documentation with metrics being supported

[quarkus-bot]

/cc @radcortez (opentelemetry)

[brunobat]

Native mode metrics need additional work.

[rsvoboda]

Reported data using the reproducer application (https://github.com/rsvoboda/code-with-quarkus-otel) look better, duplicates are gone and 0 value entries look reasonable.

Native mode needs fixing as mentioned it the comment above.

@brunobat, Will https://quarkus.io/guides/opentelemetry-metrics need adjustments?

I noticed that some metrics were removed by this change

REMOVED

jvm.memory.init
jvm.system.cpu.utilization

ADDED

jvm.cpu.context_switch
jvm.cpu.limit
jvm.cpu.longlock
jvm.network.io
jvm.network.time

Removed ones are not mandatory per https://github.com/microprofile/microprofile-telemetry/blob/2.0/spec/src/main/asciidoc/metrics.adoc spec, so it shouldn't be a big deal, but maybe worth to add entry into migration guide

[brunobat]

Reported data using the reproducer application (https://github.com/rsvoboda/code-with-quarkus-otel) look better, duplicates are gone and 0 value entries look reasonable.

Native mode needs fixing as mentioned it the comment above.

@brunobat, Will https://quarkus.io/guides/opentelemetry-metrics need adjustments?

I noticed that some metrics were removed by this change

REMOVED

jvm.memory.init
jvm.system.cpu.utilization

ADDED

jvm.cpu.context_switch
jvm.cpu.limit
jvm.cpu.longlock
jvm.network.io
jvm.network.time

Removed ones are not mandatory per https://github.com/microprofile/microprofile-telemetry/blob/2.0/spec/src/main/asciidoc/metrics.adoc spec, so it shouldn't be a big deal, but maybe worth to add entry into migration guide

Yes, I'm looking into it now. I'm cataloging what we can get on Native... It's not obvious.

[github-actions]

🙈 The PR is closed and the preview is expired.

[quarkus-bot]

Status for workflow Quarkus Documentation CI

This is the status report for running Quarkus Documentation CI on commit f6e64ed.

✅ The latest workflow run for the pull request has completed successfully.

It should be safe to merge provided you have a look at the other checks in the summary.

Warning

There are other workflow runs running, you probably need to wait for their status before merging.

[quarkus-bot]

Status for workflow Quarkus CI

This is the status report for running Quarkus CI on commit f6e64ed.

✅ The latest workflow run for the pull request has completed successfully.

It should be safe to merge provided you have a look at the other checks in the summary.

You can consult the Develocity build scans.

---

Flaky tests - Develocity

⚙️ JVM Integration Tests - JDK 17

📦 integration-tests/opentelemetry

✖ io.quarkus.it.opentelemetry.LoggingResourceTest.testException - History

• Condition with Lambda expression in io.quarkus.it.opentelemetry.LoggingResourceTest was not fulfilled within 2 minutes. - org.awaitility.core.ConditionTimeoutException

Details

org.awaitility.core.ConditionTimeoutException: Condition with Lambda expression in io.quarkus.it.opentelemetry.LoggingResourceTest was not fulfilled within 2 minutes.
	at org.awaitility.core.ConditionAwaiter.await(ConditionAwaiter.java:167)
	at org.awaitility.core.CallableCondition.await(CallableCondition.java:78)
	at org.awaitility.core.CallableCondition.await(CallableCondition.java:26)
	at org.awaitility.core.ConditionFactory.until(ConditionFactory.java:1160)
	at org.awaitility.core.ConditionFactory.until(ConditionFactory.java:1129)
	at io.quarkus.it.opentelemetry.LoggingResourceTest.testException(LoggingResourceTest.java:113)
	at java.base/java.lang.reflect.Method.invoke(Method.java:569)

---

⚙️ JVM Integration Tests - JDK 21

📦 integration-tests/opentelemetry

✖ io.quarkus.it.opentelemetry.LoggingResourceTest.testException - History

• Condition with Lambda expression in io.quarkus.it.opentelemetry.LoggingResourceTest was not fulfilled within 2 minutes. - org.awaitility.core.ConditionTimeoutException

Details

org.awaitility.core.ConditionTimeoutException: Condition with Lambda expression in io.quarkus.it.opentelemetry.LoggingResourceTest was not fulfilled within 2 minutes.
	at org.awaitility.core.ConditionAwaiter.await(ConditionAwaiter.java:167)
	at org.awaitility.core.CallableCondition.await(CallableCondition.java:78)
	at org.awaitility.core.CallableCondition.await(CallableCondition.java:26)
	at org.awaitility.core.ConditionFactory.until(ConditionFactory.java:1160)
	at org.awaitility.core.ConditionFactory.until(ConditionFactory.java:1129)
	at io.quarkus.it.opentelemetry.LoggingResourceTest.testException(LoggingResourceTest.java:113)
	at java.base/java.lang.reflect.Method.invoke(Method.java:580)

[brunobat]

@rsvoboda Updated the documentation.

[rsvoboda]

@brunobat can you share a bit about the struggles with Native mode?

Is it something Quarkus would be able to achieve using substitutions or opentelemetry library will need some changes in their codebase?

[brunobat]

@brunobat can you share a bit about the struggles with Native mode?

Is it something Quarkus would be able to achieve using substitutions or opentelemetry library will need some changes in their codebase?

We use external libraries from OTel to create these metrics and it turns out most of them don't work with native mode.

This this will require a lot of work to fix and should be made in the scope of a new issue.

Most likely we will need to either contribute fixes to upstream or instrument ourselves.

[rsvoboda]

This this will require a lot of work to fix and should be made in the scope of a new issue.

Please create one, I would like to watch it.

[brunobat]

This this will require a lot of work to fix and should be made in the scope of a new issue.

Please create one, I would like to watch it.

Done, please check #48365

[rsvoboda]

app https://github.com/rsvoboda/code-with-quarkus-otel/ with quarkus version changed to io.quarkus:quarkus-bom 999-SNAPSHOT with the patch

NATIVE

mvn clean package -DskipTests -Dnative
target/code-with-quarkus-1.0.0-SNAPSHOT-runner &
for i in {1..500}; do curl -s http://localhost:8080/hello > /dev/null ; sleep 0.001; done
sleep 10
PID=`ps aux | grep code-with-quarkus | grep 1.0.0-SNAPSHOT | awk '{print $2}'`
kill $PID

jvm.memory.limit ... I see it available, the doc says Not present

@brunobat please chek it too and adjust the documentation

JVM

mvn clean package -DskipTests
java -jar target/quarkus-app/quarkus-run.jar &
for i in {1..500}; do curl -s http://localhost:8080/hello > /dev/null ; sleep 0.001; done
sleep 10
PID=`jps -l | grep target/quarkus-app/quarkus-run.jar | cut -d" " -f 1`
kill $PID

jvm.system.cpu.utilization was previously available (e.g. in 3.23.3) why is it disabled now?

[brunobat]

@rsvoboda Native metrics will require a deeper look. They even might depend from the actual graalvm being used to compile the native image.
jvm.system.cpu.utilization was generated by the java17 instrumentation but had to be muted because that instrumentation also generated other metrics that were overlapping with the java8 instrumentation. I'm activating it explicitly when on native because the Java8 metrics are unlikely to be available on native. They mostly use MBeans.

Can we agree on merging these fixes and work on native under the scope of #48365 ?

[brunobat]

@gsmet added this to backport. We should have it in 3.20, if possible.