Performance

At a glance

What it does: compares the cost of creating spans under four tracing configurations.
Crates and versions: the example uses opentelemetry 0.33.1 and opentelemetry_sdk 0.33.1. The version set for this book is opentelemetry 0.33.1, opentelemetry_sdk 0.33.1, opentelemetry-otlp 0.33.1, opentelemetry-stdout 0.33.1, opentelemetry-appender-tracing 0.33.1, tracing-opentelemetry 0.34.0, tracing 0.1.44, tracing-subscriber 0.3.23, tokio 1.54.1, rusqlite 0.42.0, and libsql 0.10.0. The Rust edition is 2024.
Run command: cargo run -p perf-bench --release
Expected last line: ok: benchmark complete

Dependencies

[package]
name = "perf-bench"
version = "0.1.0"
edition = "2024"
publish = false

[dependencies]
opentelemetry = { version = "=0.33.1", features = ["trace"] }
opentelemetry_sdk = { version = "=0.33.1", features = ["trace"] }

Code

use std::future::Future;
use std::io::{self, Write};
use std::time::{Duration, Instant};

use opentelemetry::global;
use opentelemetry::trace::{Tracer, TracerProvider};
use opentelemetry_sdk::error::OTelSdkResult;
use opentelemetry_sdk::trace::{Sampler, SdkTracerProvider, SpanData, SpanExporter};

const ITERATIONS: u32 = 1_000_000;

#[derive(Debug)]
struct DiscardExporter;

impl SpanExporter for DiscardExporter {
    fn export(
        &self,
        _batch: Vec<SpanData>,
    ) -> impl Future<Output = OTelSdkResult> + Send {
        async { Ok(()) }
    }
}

fn measure(mut operation: impl FnMut()) -> Duration {
    let start = Instant::now();
    for _ in 0..ITERATIONS {
        operation();
    }
    start.elapsed()
}

fn write_result(
    output: &mut impl Write,
    name: &str,
    elapsed: Duration,
) -> io::Result<()> {
    writeln!(
        output,
        "{name}\t{}",
        elapsed.as_nanos() / u128::from(ITERATIONS)
    )
}

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let mut output = io::stdout().lock();

    let no_provider_tracer = global::tracer("perf-bench");
    let no_provider = measure(|| {
        let _span = no_provider_tracer.start("iteration");
    });

    let off_provider = SdkTracerProvider::builder()
        .with_sampler(Sampler::AlwaysOff)
        .build();
    let off_tracer = off_provider.tracer("perf-bench");
    let always_off = measure(|| {
        let _span = off_tracer.start("iteration");
    });

    let ratio_provider = SdkTracerProvider::builder()
        .with_sampler(Sampler::TraceIdRatioBased(0.01))
        .build();
    let ratio_tracer = ratio_provider.tracer("perf-bench");
    let ratio = measure(|| {
        let _span = ratio_tracer.start("iteration");
    });

    let batch_provider = SdkTracerProvider::builder()
        .with_sampler(Sampler::AlwaysOn)
        .with_batch_exporter(DiscardExporter)
        .build();
    let batch_tracer = batch_provider.tracer("perf-bench");
    let always_on_batch = measure(|| {
        let _span = batch_tracer.start("iteration");
    });

    writeln!(output, "case\tns/iteration")?;
    write_result(&mut output, "no provider", no_provider)?;
    write_result(&mut output, "provider, always off", always_off)?;
    write_result(&mut output, "provider, ratio 0.01", ratio)?;
    write_result(
        &mut output,
        "provider, always on, batch",
        always_on_batch,
    )?;

    let off_shutdown = off_provider.shutdown();
    let ratio_shutdown = ratio_provider.shutdown();
    let batch_shutdown = batch_provider.shutdown();

    off_shutdown?;
    ratio_shutdown?;
    batch_shutdown?;

    writeln!(output, "ok: benchmark complete")?;
    Ok(())
}

Run it

Run:

cargo run -p perf-bench --release

The numbers depend on the machine and build. The table format is:

case                         ns/iteration
no provider                  -
provider, always off         -
provider, ratio 0.01          -
provider, always on, batch    -
ok: benchmark complete

The dashes are placeholders, not measured results. In the program’s output the columns are separated by a tab character; the spacing shown here is for illustration.

How it works

The program measures one million iterations of a function that starts and drops a span. It uses std::time::Instant and reports integer nanoseconds per iteration.

The first case uses the global tracer without installing a provider. The next cases use explicit providers configured with the always-off sampler and a trace ID ratio sampler set to 0.01. The last case uses an always-on sampler and a batch processor whose exporter discards spans.

The batch processor can be tuned through its queue size, scheduled delay, export batch size, and export timeout. Queue size bounds spans waiting to be exported. Scheduled delay controls how long a batch can wait before export. Export batch size bounds spans sent together, and export timeout limits time spent on an export. This example uses the SDK’s default batch settings. If the queue fills, new spans can be dropped rather than waiting for space.

Span and attribute limits bound the data retained for each span. Sampling reduces the number of spans recorded and exported. Keep attribute cardinality low: values that vary on every request can create many distinct attribute values. Avoid unnecessary allocation on the hot path, especially when recording attributes.

Measure in the environment that matters to you. Results vary with compiler settings, hardware, workload, and exporter behavior; this small benchmark is not a substitute for measuring your application.

Pitfalls

Do not compare debug and release builds as if they were equivalent. Use --release for this example.
Do not treat the sample numbers as universal. Run the program on the target machine and under representative load.
Do not assume batching makes a full queue harmless. Spans can be dropped when the queue is full.
Do not put request IDs or other high-cardinality values into attributes without considering their storage and export cost.
Do not assume sampling removes all tracing overhead. Starting spans and evaluating sampling still take time.

Keyboard shortcuts