[kubeletstats] Calculate cpu usage on the fly behind feature gate

[ChrsMark]

Description

Note: The big diff is because of the updated expected test files and the added stats' summary sample.

This PR introduces a new way for calculating the *.cpu.usage metrics based on the discussions from open-telemetry/semantic-conventions#2418.

It adds the receiver.kubeletstats.cpuUsageScrapeBased feature gate which when enabled, container.cpu.usage, k8s.pod.cpu.usage and k8s.node.cpu.usage (and the cpu utilization metrics derived from them) are calculated as the rate of the corresponding *.cpu.time counter between consecutive scrapes, instead of being read directly from the kubelet's UsageNanoCores value.

The reasoning for this change is that (as discussed in open-telemetry/semantic-conventions#2418), the derived Kubelet metric (UsageNanoCores) can be inaccurate (see open-telemetry/semantic-conventions#2418 (comment)) given that it's calculated over a fixed interval that Kubelet sets and is not always in alignment with the scrape interval of the Collector. This patch gives better control on this rate calculation following a similar approach with hostmetrics receiver.

The time diff is calculated based on CPUStats.Time that Kubelet's API provides (The time at which these CPU stats were updated).

Link to tracking issue

Fixes #49477

Testing

Added unit tests

Documentation

Tuned

Authorship

• I, a human, wrote this pull request description myself.

[ChrsMark]

Local Testing Setup

Deploy the Collector to ship to Prometheus Kube-Stack:

Details

mode: daemonset

image:
  repository: chrismark/otelcontribcol-dev
  tag: latest
  pullPolicy: IfNotPresent

command:
  name: otelcontribcol
  extraArgs:
    - --feature-gates=kubeletstats.cpuUsageScrapeBased

presets:
  kubeletMetrics:
    enabled: true

clusterRole:
  create: true
  rules:
    - apiGroups:
        - ""
      resources:
        - nodes
        - services
        - pods
        - namespaces
      verbs:
        - get
        - list
        - watch
    - apiGroups: [ "" ]
      resources: [ "nodes/proxy"]
      verbs: [ "get" ]
    - apiGroups:
        - ""
      resources:
        - nodes/stats
      verbs:
        - get
    - nonResourceURLs:
        - "/metrics"
      verbs:
        - get

resources:
  limits:
    cpu: 1
    memory: 1Gi

ports:
  otlp:
    enabled: false
  otlp-http:
    enabled: false
  jaeger-compact:
    enabled: false
  jaeger-thrift:
    enabled: false
  jaeger-grpc:
    enabled: false
  zipkin:
    enabled: false

config:
  exporters:
    debug:
      verbosity: detailed
    prometheusremotewrite:
      endpoint: "http://kps-kube-prometheus-stack-prometheus.monitoring.svc:9090/api/v1/write"
      tls:
        insecure: true
      resource_to_telemetry_conversion:
        enabled: true

  receivers:
    kubeletstats:
      auth_type: serviceAccount
      collection_interval: 20s
      endpoint: ${env:K8S_NODE_IP}:10250
      insecure_skip_verify: true
      extra_metadata_labels:
        - container.id
      metrics:
        k8s.container.cpu_limit_utilization:
          enabled: true
        k8s.pod.cpu_limit_utilization:
          enabled: true

  service:
    extensions: [health_check]
    telemetry:
      logs:
        level: INFO
    pipelines:
      metrics:
        receivers: [ kubeletstats ]
        processors: [ k8sattributes ]
        exporters: [ debug, prometheusremotewrite ]

Deploy a target CPU stress Pod that will consume 0.5 cores:

Details

apiVersion: v1
kind: Pod
metadata:
  name: cpu-stress
  labels:
    app: cpu-stress
spec:
  containers:
    - name: stress
      image: ghcr.io/colinianking/stress-ng:latest
      # --cpu-load paces the worker's own busy/sleep duty cycle to hit the
      # target percentage, instead of demanding 100% and relying on the
      # kernel's CFS quota to clip it back down (which oscillates).
      args: ["--cpu", "1", "--cpu-load", "50", "--cpu-method", "matrixprod"]
      resources:
        requests:
          cpu: "500m"
        # Limit is above the paced target so cgroup throttling never kicks
        # in and interferes with the self-imposed 50% duty cycle.
        limits:
          cpu: "1"

Verify in the graph that the CPU Usage metric is around 0.5 cores over time:

[Copilot]

Pull request overview

This PR adds an alpha feature-gated path in kubeletstatsreceiver to compute *.cpu.usage metrics (and derived utilization metrics) as a scrape-to-scrape rate of *.cpu.time, rather than using Kubelet’s UsageNanoCores, aligning behavior with the semconv discussion and reducing scrape-interval mismatch issues.

Changes:

• Introduces receiver.kubeletstats.cpuUsageScrapeBased feature gate and generated gate registration.
• Adds a per-scraper CPUUsageCalculator and wires it through the metrics pipeline to compute usage from UsageCoreNanoSeconds deltas and Kubelet-provided sample times.
• Adds/updates unit tests and fixtures (including a second-scrape summary) and documents the new feature gate.

Reviewed changes

Copilot reviewed 14 out of 17 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
receiver/kubeletstatsreceiver/scraper.go	Stores and passes a shared CPUUsageCalculator into metrics production.
receiver/kubeletstatsreceiver/internal/kubelet/metrics.go	Adds scrape lifecycle hooks (startScrape/endScrape) and propagates the calculator to the accumulator.
receiver/kubeletstatsreceiver/internal/kubelet/accumulator.go	Passes stable per-resource keys into CPU metric emission so usage can be computed per resource across scrapes.
receiver/kubeletstatsreceiver/internal/kubelet/cpu.go	Adds feature-gated scrape-based CPU usage computation from cumulative CPU time.
receiver/kubeletstatsreceiver/internal/kubelet/cpu_usage.go	New calculator for rate computation and pruning of stale resource keys.
receiver/kubeletstatsreceiver/internal/kubelet/cpu_usage_test.go	New unit coverage for calculator behavior (first sample, rate, stale/negative cases, pruning, multi-key).
receiver/kubeletstatsreceiver/internal/kubelet/cpu_test.go	New tests validating feature-gated behavior and legacy path when gate is disabled.
receiver/kubeletstatsreceiver/scraper_test.go	Adds end-to-end scraper tests for the feature gate, including a controlled two-scrape scenario.
receiver/kubeletstatsreceiver/testdata/stats-summary-second-scrape.json	New fixture enabling a deterministic “second scrape” CPU usage rate validation.
receiver/kubeletstatsreceiver/testdata/scraper/test_scraper_cpu_usage_scrape_based_expected.yaml	New golden expected output for the first-scrape (no usage yet) gated scenario.
receiver/kubeletstatsreceiver/metadata.yaml	Declares the new feature gate in component metadata.
receiver/kubeletstatsreceiver/internal/metadata/generated_feature_gates.go	Generated feature gate registration for the receiver.
receiver/kubeletstatsreceiver/documentation.md	Documents the feature gate in the receiver docs.
receiver/kubeletstatsreceiver/go.mod	Adds direct deps needed for feature gates and test helper usage.
.chloggen/kubeletstats-cpu-usage-scrape-based.yaml	Adds a changelog entry for the new feature gate.

Files not reviewed (1)

• receiver/kubeletstatsreceiver/internal/metadata/generated_feature_gates.go: Generated file

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.