[processor/dynamic_sampling] honour incoming ot=th and ot=rv#49520
Open
MikeGoldsmith wants to merge 1 commit into
Open
[processor/dynamic_sampling] honour incoming ot=th and ot=rv#49520MikeGoldsmith wants to merge 1 commit into
MikeGoldsmith wants to merge 1 commit into
Conversation
11 tasks
jmacd
reviewed
Jul 7, 2026
| The processor honours any incoming `ot=th` (sampling threshold) and `ot=rv` (explicit randomness) already set on incoming spans by an upstream sampler, such as an SDK head sampler or a probabilistic collector processor: | ||
|
|
||
| - **Randomness.** If an incoming span carries `ot=rv`, that value is used to make the sampling decision. Otherwise the last 7 bytes of the trace ID are used, per the consistent probability spec. | ||
| - **Threshold composition.** The rule-selected sampler picks a rate `N`; the emitted `ot=th` is composed on top of any upstream threshold `T_up` so the effective absolute keep probability is `P(rand > T_up) / N`. Downstream metric consumers reading `ot=th` reconstruct the correct adjusted count for the original population without additional configuration. |
Contributor
There was a problem hiding this comment.
Is this the same as the "equalizing" mode documented in
https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/probabilisticsamplerprocessor#equalizing?
The way this reads there's a conditional probability formula lurking somewhere (true?).
|
|
||
| ### Known limitation: adaptive-sampler rate bias under upstream sampling | ||
|
|
||
| The adaptive samplers (`ema_dynamic`, `ema_throughput`, `windowed_throughput`) select their rate from the traffic they observe, assuming that input is a uniform sample of the underlying population. When an upstream stage has already sampled, the input reaching the adaptive sampler is skewed toward higher randomness values, and per-key rate math based on observed volume is biased. The emitted `ot=th` is still metric-accurate downstream (via threshold composition above); the caveat is that the rate the adaptive sampler *targets* is "of the input reaching me," not "of the original population." Follow-up in [#49517](https://github.com/open-telemetry/opentelemetry-collector-contrib/issues/49517). |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Implements the "Read incoming
ot=th" alpha item from thedynamic_samplingtracking issue.ot=thas the upstream sampling threshold;ot=rvas the randomness value when present, falling back to the last 7 bytes of the trace ID otherwise.Nwith the upstream threshold under consistent probability sampling semantics: emitted thresholdT_oursis chosen so that1/Nof the surviving upstream traffic passes, giving an effective absolute keep probability ofP(rand > T_up) / N. Downstream metric consumers readingot=threconstruct the correct adjusted count for the original population without additional configuration.UpdateTValueWithSampling's spec-mandated behaviour: a stricter incoming threshold is never lowered.processor_dynamic_sampling_incoming_tracestate_unparseableto observe cases where the incoming tracestate fails to parse on the emit path.Known limitation: the adaptive samplers (
ema_dynamic,ema_throughput,windowed_throughput) still pick their rate from observed input, which under upstream sampling is a skewed slice of the population. Emitted thresholds remain metric-accurate downstream, but per-key rate math is biased. Follow-up: #49517.Link to tracking issue
Related: #49311
Testing
combineThreshold(rate 1 passthrough, no-upstream 10x, upstream+10x composition).ot=threflects composed effective threshold when upstreamot=this present.ot=thretain that threshold rather than being lowered by the cached decision.go test -race ./...clean.Documentation
otelcol_processor_dynamic_sampling_incoming_tracestate_unparseable..chloggenentry added.Authorship