Telemetry in Rust

This site is an independent how-to guide for application telemetry in Rust. It is not the official OpenTelemetry documentation. You can find the official Rust documentation at https://opentelemetry.io/docs/languages/rust/ and the crate reference at https://docs.rs/opentelemetry.

The guide is for developers and coding agents who want to add traces, metrics and logs to a Rust program, ship them with OTLP exporters, tune the SDK for performance, and store spans in SQLite or Turso. Every page states what the code does and why.

Start here

• Setup and first signal — install the SDK, register a tracer, and see your first span.
• Agent quickstart — a short path for coding agents to wire up telemetry with few steps.
• Traces — create spans, add attributes, and nest work.
• Metrics — record counters, histograms and gauges with the meter API.
• Logs — bridge the tracing crate into OpenTelemetry logs.
• Exporters — send telemetry with the OTLP exporter over gRPC or HTTP.
• Configuration — read settings from environment variables instead of code.
• Performance — benchmark the SDK and keep overhead low.
• Spans in SQLite — store finished spans in a local SQLite file.
• Spans in Turso — store spans in a remote Turso database.

Versions

The code in this guide was tested against these versions.

How the examples are checked

Every example in this guide is a complete crate in the examples folder. Each one has its own Cargo.toml and compiles on its own. The code shown on each page is included from the file that was compiled, so what you read is what runs.

Setup and first signal

At a glance

• What it does: builds a tracer provider with the stdout exporter and a service.name resource, sets it as the global provider, records one span with two attributes and one event, then shuts the provider down.
• Crates and versions: opentelemetry 0.33.1 (API), opentelemetry_sdk 0.33.1 (SDK), opentelemetry-stdout 0.33.1 (exporter), tokio 1.54.1 (async runtime). Rust edition 2024.
• Run command (from the examples folder): cargo run -p first-signal
• Expected last line: ok: first signal exported

Get the examples

Download examples.zip from the site root and unpack it, or create a new folder with the Cargo.toml and src/main.rs shown below. After unpacking, run the commands from the examples folder.

Dependencies

[package]
name = "first-signal"
edition = "2024"
publish = false

[dependencies]
opentelemetry = "0.33.1"
opentelemetry_sdk = { version = "0.33.1", features = ["rt-tokio"] }
opentelemetry-stdout = "0.33.1"
tokio = { version = "1.54.1", features = ["full"] }

Code

use opentelemetry::global;
use opentelemetry::trace::{TraceContextExt, Tracer};
use opentelemetry_sdk::Resource;
use opentelemetry_sdk::trace::SdkTracerProvider;
use opentelemetry_stdout::SpanExporter;

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let exporter = SpanExporter::default();
    let provider = SdkTracerProvider::builder()
        .with_simple_exporter(exporter)
        .with_resource(
            Resource::builder()
                .with_service_name("first-signal")
                .build(),
        )
        .build();

    global::set_tracer_provider(provider.clone());

    let tracer = global::tracer("first-signal");
    tracer
        .in_span("first-span", |cx| {
            let span = cx.span();
            span.set_attribute(opentelemetry::KeyValue::new("example.kind", "hello"));
            span.set_attribute(opentelemetry::KeyValue::new("example.count", 1));
            span.add_event(
                "starting",
                vec![opentelemetry::KeyValue::new("event.source", "main")],
            );
        });

    provider.shutdown()?;

    println!("ok: first signal exported");
    Ok(())
}

Run it

$ cargo run -p first-signal
...
ok: first signal exported

The stdout exporter prints one JSON line per span while the program runs. The last line the program prints is ok: first signal exported.

How it works

OpenTelemetry for Rust comes in three kinds of package (all 0.33.x here):

• The API crate, opentelemetry. It defines the types your application uses: Tracer, Span, KeyValue and the global functions. Code against the API alone stays easy to test.
• The SDK crate, opentelemetry_sdk. It implements the API: providers that own the work of processing and exporting, and the samplers and processors behind them. Only the binary at the edge of your application touches it.
• The exporter crates, one per destination. opentelemetry-stdout writes spans to standard output; opentelemetry-otlp sends them over gRPC or HTTP.

The example then goes through these steps:

1. Build a resource. A resource describes who produced the telemetry. Resource::builder().with_service_name("first-signal") sets service.name, the one field most backends group by. Version note: Resource::builder() is the 0.33.x form; older versions had Resource::new.
2. Build an exporter. SpanExporter::default() from opentelemetry-stdout gives an exporter that writes spans to standard output. It is for learning and debugging, not for production.
3. Build the provider. SdkTracerProvider::builder() composes it: with_simple_exporter(exporter) attaches the exporter, with_resource attaches the resource. The simple exporter writes each ended span immediately, which is the easiest thing to watch. The batch exporter, with_batch_exporter, queues spans and writes them in the background; it is what you want once output volume matters.
4. Install it globally. global::set_tracer_provider(provider) makes every global::tracer call in the process use this provider. The global provider can only be set once; later calls are ignored.
5. Get a tracer and make a span. A tracer creates spans; a span is one timed operation with a name, attributes and events. global::tracer("first-signal") names the instrumentation library. in_span runs the closure inside a span, ending the span when the closure returns.
6. Attach data. Attributes are key-value pairs that describe the span (example.kind, example.count). An event is a timestamped note inside the span (starting), itself with attributes.
7. Shut down. provider.shutdown() flushes and exports anything still in flight and stops accepting new spans. Call it before your program exits so nothing is lost. It is not called by Drop, so a program that skips it can silently drop spans. The example uses with_simple_exporter, which writes synchronously, so the span is on stdout before shutdown returns.

Pitfalls

• Forgetting shutdown. The provider does not export on drop. Without provider.shutdown() before exit, buffered spans vanish. With the batch exporter this is the default failure mode: the program exits while spans are still in the queue.
• Dropping the provider early. global::set_tracer_provider takes ownership of a clone; if your original SdkTracerProvider value is dropped and you keep using the global one, shutdown happens twice or never. Keep one owner of the provider and shut it down deliberately.
• Setting the global provider twice. The second call is ignored and the first provider stays installed, so your new exporter never sees anything.
• Using the stdout exporter in production. It writes every span to the console. Fine for this chapter; in a service you would send spans to a collector with opentelemetry-otlp instead.
• Building a resource by hand with an old API. In 0.33.x use Resource::builder(). Older snippets with Resource::new(vec![...]) no longer compile.
• Confusing the API with the SDK. Library code should depend on opentelemetry only and take a Tracer from its caller; the SDK belongs in the binary. Pulling the SDK into a library forces a provider on everyone who links it.
• Looking for the span on stdout before the closure ends. The simple exporter writes a span when it ends, not while it is open. If the program seems to hang after printing nothing, it is waiting on the runtime, not on telemetry.

Quickstart for coding agents

This page is a compact path for adding traces, metrics, and logs to an existing Rust service. The examples use stdout exporters so you can inspect output locally.

To get the examples: fetch examples.zip from the site root (relative to any book page: examples.zip), unpack it, and run commands from the examples folder.

Versions named in the chapters

opentelemetry 0.33.1, opentelemetry_sdk 0.33.1, opentelemetry-otlp 0.33.1, opentelemetry-stdout 0.33.1, opentelemetry-appender-tracing 0.33.1, tracing-opentelemetry 0.34.0, tracing 0.1.44, tracing-subscriber 0.3.23, tokio 1.54.1, rusqlite 0.42.0, and libsql 0.10.0. Use the versions relevant to the example you are adapting.

Add signals in this order

1. Traces: Follow Traces and adapt the traces example crate. It connects tracing spans and events to OpenTelemetry spans. Run cargo run -p traces. The chapter shows this last line: ok: traces exported.
2. Metrics: Follow Metrics and adapt the metrics example crate. It creates a periodic meter provider and records metric data points. Run cargo run -p metrics. The chapter shows this last line: ok: metrics exported.
3. Logs: Follow Logs and adapt the logs example crate. It bridges tracing events to OpenTelemetry log records. Run cargo run -p logs. The chapter shows this last line: ok: log records emitted and providers shut down.

These lines confirm the examples reached their final status message; they do not prove delivery to a remote collector.

Environment variables

The Configuration chapter names OTEL_SERVICE_NAME, OTEL_RESOURCE_ATTRIBUTES, OTEL_EXPORTER_OTLP_ENDPOINT, OTEL_EXPORTER_OTLP_PROTOCOL, OTEL_EXPORTER_OTLP_HEADERS, OTEL_EXPORTER_OTLP_TIMEOUT, OTEL_TRACES_SAMPLER, and OTEL_TRACES_SAMPLER_ARG. That example reads and applies these variables directly.

Failure signs described in the chapters

• Missing provider shutdown can lose buffered spans or metrics.
• Initializing a tracing subscriber more than once can fail.
• An OTLP example can print its final flush line even when no collector is listening; check the collector output to verify delivery.

Traces

At a glance

• What it does: creates nested spans with tracing and exports them to stdout.
• Crates and versions: opentelemetry 0.33.1, opentelemetry_sdk 0.33.1, opentelemetry-stdout 0.33.1, tracing-opentelemetry 0.34.0, tracing 0.1.44, tracing-subscriber 0.3.23, and tokio 1.54.1.
• Run command: cargo run -p traces
• Expected last line: ok: traces exported

Dependencies

[package]
name = "traces"
version = "0.1.0"
edition = "2024"
publish = false

[dependencies]
opentelemetry = { version = "0.33.1" }
opentelemetry_sdk = { version = "0.33.1", features = ["trace"] }
opentelemetry-stdout = { version = "0.33.1", features = ["trace"] }
tokio = { version = "1.54.1", features = ["macros", "rt-multi-thread"] }
tracing = { version = "0.1.44", features = ["attributes"] }
tracing-opentelemetry = "0.34.0"
tracing-subscriber = { version = "0.3.23", features = ["registry"] }

Code

use std::io::{self, Write};

use opentelemetry::trace::TracerProvider;
use opentelemetry_sdk::trace::SdkTracerProvider;
use tracing::Instrument;
use tracing_subscriber::prelude::*;

#[tracing::instrument(fields(error = tracing::field::Empty))]
fn process_order(order_id: u64) {
    let span = tracing::info_span!("validate_order", item_count = 2_u64);
    let _entered = span.enter();
    let error = "missing item";
    span.record("error", error);
    tracing::error!(order_id, error, "order validation failed");
}

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let exporter = opentelemetry_stdout::SpanExporter::default();
    let provider = SdkTracerProvider::builder()
        .with_simple_exporter(exporter)
        .build();
    let tracer = provider.tracer("traces-example");
    let telemetry = tracing_opentelemetry::layer().with_tracer(tracer);

    let subscriber_result = tracing_subscriber::registry()
        .with(telemetry)
        .try_init();
    if let Err(error) = subscriber_result {
        provider.shutdown()?;
        return Err(error.into());
    }

    process_order(42);

    let async_span = tracing::info_span!("async_work", job = "demo");
    let task = tokio::spawn(
        async {
            tracing::info!("work completed");
        }
        .instrument(async_span),
    );
    task.await?;

    provider.shutdown()?;

    let mut stdout = io::stdout().lock();
    stdout.write_all(b"ok: traces exported\n")?;
    Ok(())
}

Run it

Run:

cargo run -p traces

The stdout exporter writes span data before the final line. The final line is:

ok: traces exported

How it works

The SDK tracer provider owns the span exporter and sends completed spans to stdout.

The subscriber combines normal tracing subscribers with the OpenTelemetry layer. The layer converts tracing spans and events into OpenTelemetry spans.

#[instrument] creates a span for process_order. The validate_order span is entered inside it, so it becomes a child span.

Fields on a span are attributes. Events add timestamped records to the current span. The example records an error field and emits an error event.

A failed operation should also set an OpenTelemetry error status when the application has a direct status API available. An error event alone is useful evidence, but it is not the same as a status.

The current span is available through tracing context. Use an instrumentation helper such as Instrument when spawning an async task so the task keeps its parent context. Thread hand-offs need the equivalent context propagation step.

The provider is shut down after the work completes. This flushes the exporter before the process exits.

Pitfalls

• Initializing the subscriber more than once can fail, so initialize it in one place.
• Creating a span does not make it a child until it is entered or attached to a context.
• Recording an error as an event is not the same as recording it as a span field or setting status.
• Spawning an async task without instrumenting it loses the intended parent relationship.
• Exiting without shutting down the provider can lose buffered spans.
• Keep exporter configuration separate from application code when changing from stdout to OTLP.

Metrics

At a glance

• Builds a periodic meter provider that exports metrics to stdout.
• Uses opentelemetry 0.33.1, opentelemetry_sdk 0.33.1, and opentelemetry-stdout 0.33.1.
• The book’s pinned versions are: opentelemetry 0.33.1, opentelemetry_sdk 0.33.1, opentelemetry-otlp 0.33.1, opentelemetry-stdout 0.33.1, opentelemetry-appender-tracing 0.33.1, tracing-opentelemetry 0.34.0, tracing 0.1.44, tracing-subscriber 0.3.23, tokio 1.54.1, rusqlite 0.42.0, and libsql 0.10.0.
• Run it with cargo run -p metrics.
• The final line is ok: metrics exported.

Dependencies

[package]
name = "metrics"
version = "0.1.0"
edition = "2024"
publish = false

[dependencies]
opentelemetry = "0.33.1"
opentelemetry_sdk = { version = "0.33.1", features = ["metrics"] }
opentelemetry-stdout = { version = "0.33.1", features = ["metrics"] }

Code

use std::io::{self, Write};
use std::time::Duration;

use opentelemetry::metrics::MeterProvider;
use opentelemetry::KeyValue;
use opentelemetry_sdk::metrics::{PeriodicReader, SdkMeterProvider};
use opentelemetry_stdout::MetricExporter;

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let exporter = MetricExporter::builder().build();
    let reader = PeriodicReader::builder(exporter)
        .with_interval(Duration::from_secs(5))
        .build();
    let provider = SdkMeterProvider::builder().with_reader(reader).build();
    let meter = provider.meter("metrics-example");

    let requests = meter.u64_counter("requests").build();
    requests.add(1, &[KeyValue::new("route", "/health")]);

    let duration = meter.f64_histogram("request.duration").build();
    duration.record(0.125, &[KeyValue::new("route", "/health")]);

    let active = meter.i64_up_down_counter("active.connections").build();
    active.add(1, &[KeyValue::new("service", "metrics")]);
    active.add(-1, &[KeyValue::new("service", "metrics")]);

    provider.shutdown()?;
    io::stdout().write_all(b"ok: metrics exported\n")?;

    Ok(())
}

Run it

Run:

cargo run -p metrics

The stdout exporter prints the collected metrics. The last line is:

ok: metrics exported

How it works

The program creates a stdout metric exporter.

A PeriodicReader collects metrics at a fixed interval. This example uses a five-second interval. The final provider shutdown also flushes the last collection, so the program does not need to wait for the interval.

The SdkMeterProvider owns the reader. A meter created from the provider creates instruments.

The u64 counter records values that only increase, such as completed requests.

The f64 histogram records measurements such as request duration. Its values can vary in either direction.

The up-down counter records values that can increase or decrease, such as the number of active connections.

Each recording includes attributes. Attributes add dimensions to a metric. Keep their values bounded. A user ID or request ID can create excessive cardinality and increase memory use and export size.

Temporality describes how a reader reports measurements. Cumulative temporality reports totals from the start of the instrument’s lifetime. Delta temporality reports the change since the previous collection. The exporter and reader determine the supported behavior.

Use a counter for a total, a histogram for a distribution, and an up-down counter for a current value that moves in both directions. Use a gauge when you need to observe a value directly rather than record changes, though this example does not build one.

The provider is shut down before the final status line. This gives the reader a chance to export the last collection.

Pitfalls

• Forgetting provider shutdown can lose measurements that are still waiting for the next export interval.
• Choosing a counter for a value that can decrease produces an incorrect metric.
• Recording unbounded attributes creates high cardinality.
• Setting the export interval too high delays output; setting it too low increases export work.
• A histogram does not automatically provide useful bucket boundaries for every workload. Check the exported data and configure aggregation when the defaults do not fit.
• The stdout exporter is useful for local inspection. It is not a remote collector endpoint.

Logs

At a glance

• Builds a logger provider that writes log records to stdout and connects tracing events to it.
• Uses opentelemetry 0.33.1, opentelemetry_sdk 0.33.1, opentelemetry-stdout 0.33.1, opentelemetry-appender-tracing 0.33.1, tracing-opentelemetry 0.34.0, tracing 0.1.44, and tracing-subscriber 0.3.23. Other guide versions: opentelemetry-otlp 0.33.1, tokio 1.54.1, rusqlite 0.42.0, and libsql 0.10.0.
• Run with cargo run -p logs.
• The last line is ok: log records emitted and providers shut down.

Dependencies

The example uses Rust edition 2024. The versions below are pinned in its manifest.

[package]
name = "logs"
version = "0.1.0"
edition = "2024"
publish = false

[dependencies]
opentelemetry = { version = "=0.33.1", default-features = false, features = ["trace"] }
opentelemetry_sdk = { version = "=0.33.1", default-features = false, features = ["logs", "trace"] }
opentelemetry-stdout = { version = "=0.33.1", default-features = false, features = ["logs"] }
opentelemetry-appender-tracing = "=0.33.1"
tracing-opentelemetry = "=0.34.0"
tracing = "=0.1.44"
tracing-subscriber = { version = "=0.3.23", default-features = false, features = ["registry"] }

Code

use std::io::{self, Write};

use opentelemetry::trace::TracerProvider;
use opentelemetry_appender_tracing::layer::OpenTelemetryTracingBridge;
use opentelemetry_sdk::{logs::SdkLoggerProvider, trace::SdkTracerProvider};
use tracing_subscriber::layer::SubscriberExt;

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let logger_provider = SdkLoggerProvider::builder()
        .with_simple_exporter(opentelemetry_stdout::LogExporter::default())
        .build();
    let tracer_provider = SdkTracerProvider::builder().build();
    let tracer = tracer_provider.tracer("logs-example");

    let subscriber = tracing_subscriber::registry()
        .with(tracing_opentelemetry::layer().with_tracer(tracer))
        .with(OpenTelemetryTracingBridge::new(&logger_provider));
    tracing::subscriber::set_global_default(subscriber)?;

    tracing::info!(operation = "start", message = "starting log example");
    tracing::warn!(retry_count = 2, message = "retry scheduled");

    let span = tracing::info_span!("request", request_id = "request-42");
    let entered = span.enter();
    tracing::error!(error = "simulated failure", message = "request failed");
    drop(entered);

    logger_provider.shutdown()?;
    tracer_provider.shutdown()?;

    writeln!(
        io::stdout().lock(),
        "ok: log records emitted and providers shut down"
    )?;
    Ok(())
}

Run it

Run:

cargo run -p logs

The stdout exporter prints the records; their formatting can vary. The final line is:

ok: log records emitted and providers shut down

How it works

The logger provider owns the log pipeline. Its simple exporter writes each record to stdout.

OpenTelemetryTracingBridge is a tracing-subscriber layer. It turns tracing events into OpenTelemetry log records. The event levels map to log severity, and event fields become record attributes.

The example also adds a tracing-opentelemetry layer backed by an SDK tracer provider. When the error event occurs inside the active span, the bridge can add that span’s trace context to the log record. This lets a backend correlate the log with a trace when both signals are available.

Finally, the example shuts down both providers before returning.

Pitfalls

• A span alone does not create OpenTelemetry trace context. Install the tracing-opentelemetry layer and emit the event while the span is entered.
• Install one global tracing subscriber. Setting a second global subscriber fails.
• The stdout exporter writes locally; it does not send records to an OTLP endpoint.
• Shut down providers before exit so buffered telemetry can be flushed.

Exporters

At a glance

• Exports one span to a local OpenTelemetry Collector using OTLP.
• Example crates: opentelemetry 0.33.1, opentelemetry_sdk 0.33.1, opentelemetry-otlp 0.33.1, and tokio 1.54.1. Book crate versions: opentelemetry-stdout 0.33.1, opentelemetry-appender-tracing 0.33.1, tracing-opentelemetry 0.34.0, tracing 0.1.44, tracing-subscriber 0.3.23, rusqlite 0.42.0, and libsql 0.10.0.
• Run with cargo run -p otlp-export -- grpc or choose http; an optional second argument overrides the endpoint.
• Expected last line: ok: one span flushed over grpc; look for it in the collector's output (or over http). Check the Collector output to confirm delivery.

Dependencies

The example uses Rust edition 2024.

[package]
name = "otlp-export"
version = "0.1.0"
edition = "2024"
publish = false

[dependencies]
opentelemetry = { version = "=0.33.1", features = ["trace"] }
opentelemetry_sdk = { version = "=0.33.1", features = ["rt-tokio"] }
opentelemetry-otlp = { version = "=0.33.1", default-features = false, features = ["trace", "grpc-tonic", "http-proto", "reqwest-blocking-client"] }
tokio = { version = "=1.54.1", features = ["macros", "rt-multi-thread"] }

Code

use std::io::Write;
use std::time::Duration;

use opentelemetry::trace::{Tracer, TracerProvider};
use opentelemetry_otlp::{SpanExporter, WithExportConfig};
use opentelemetry_sdk::trace::SdkTracerProvider;

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let mut args = std::env::args_os().skip(1);
    let use_http = match args.next() {
        None => false,
        Some(argument) if argument == std::ffi::OsStr::new("grpc") => false,
        Some(argument) if argument == std::ffi::OsStr::new("http") => true,
        Some(_) => {
            return Err(std::io::Error::new(
                std::io::ErrorKind::InvalidInput,
                "choose grpc or http",
            )
            .into());
        }
    };

    let endpoint = match args.next() {
        Some(endpoint) => endpoint.into_string().map_err(|_| {
            std::io::Error::new(std::io::ErrorKind::InvalidInput, "endpoint must be valid Unicode")
        })?,
        None if use_http => "http://localhost:4318/v1/traces".to_owned(),
        None => "http://localhost:4317".to_owned(),
    };
    if args.next().is_some() {
        return Err(std::io::Error::new(
            std::io::ErrorKind::InvalidInput,
            "usage: otlp-export [grpc|http] [endpoint]",
        )
        .into());
    }

    let exporter = if use_http {
        SpanExporter::builder()
            .with_http()
            .with_endpoint(endpoint)
            .with_timeout(Duration::from_secs(5))
            .build()?
    } else {
        SpanExporter::builder()
            .with_tonic()
            .with_endpoint(endpoint)
            .with_timeout(Duration::from_secs(5))
            .build()?
    };

    let provider = SdkTracerProvider::builder()
        .with_batch_exporter(exporter)
        .build();
    let tracer = provider.tracer("otlp-export");
    let span = tracer.start("example operation");
    drop(span);
    provider.shutdown()?;

    let protocol = if use_http { "http" } else { "grpc" };
    // Shutdown flushes the batch. It does not prove delivery: the span arrives only if a collector is listening.
    writeln!(
        std::io::stdout(),
        "ok: one span flushed over {protocol}; look for it in the collector's output"
    )?;
    Ok(())
}

Run it

Start a Collector with the configuration below, then run the example:

cargo run -p otlp-export -- grpc

To use HTTP protobuf instead, run:

cargo run -p otlp-export -- http

The protocol defaults to grpc when omitted. You can pass an endpoint as the second argument, for example http http://localhost:4318/v1/traces. The final line is ok: one span flushed over grpc; look for it in the collector's output for the first command, and the same with http for the second. Delivery is proved at the collector, whose debug exporter prints the span; the program’s own line only says the batch was flushed.

This minimal Collector configuration accepts both protocols and prints received spans with the debug exporter:

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

exporters:
  debug:
    verbosity: detailed

service:
  pipelines:
    traces:
      receivers: [otlp]
      exporters: [debug]

How it works

OTLP is the OpenTelemetry protocol for sending telemetry between services and collectors. The example creates an OTLP span exporter and a batch span processor, then ends one span. Shutting down the provider flushes the batch before the program exits.

The grpc argument selects gRPC on http://localhost:4317. The http argument selects HTTP with protobuf encoding on http://localhost:4318/v1/traces. These are the default endpoints; an optional second argument replaces the selected endpoint. The Collector configuration must enable the matching receiver protocol.

The OTLP exporter enables grpc-tonic, http-proto, and reqwest-blocking-client: the default batch processor exports on its own thread without a Tokio runtime, so its HTTP client must be blocking rather than async.

gRPC and HTTP protobuf both carry OTLP data. Choose gRPC when your Collector and network support it; choose HTTP protobuf when HTTP-based routing or infrastructure is a better fit.

The example sets a five-second export timeout. In production, use TLS when telemetry crosses an untrusted network. Configure the exporter and Collector consistently: an HTTPS endpoint alone does not configure certificate trust. When authentication is required, configure matching headers on the exporter and Collector. Keep credentials out of source code.

Timeouts bound how long an export attempt can take. Retries can help with transient failures, but they add delay and load; use bounded retries and coordinate their limits with the timeout and shutdown period.

Pitfalls

• A Collector must be running and listening on the selected port and protocol.
• HTTP OTLP uses the /v1/traces path; the gRPC endpoint does not.
• A successful shutdown flushes pending spans, but it cannot deliver them if the Collector is unreachable.
• TLS and authentication settings must match at both ends.

Configuration

At a glance

• Reads standard OpenTelemetry environment variables, reports the effective service name and sampler, and attempts to export ten spans.
• Version references: opentelemetry 0.33.1, opentelemetry_sdk 0.33.1, opentelemetry-otlp 0.33.1, opentelemetry-stdout 0.33.1, opentelemetry-appender-tracing 0.33.1, tracing-opentelemetry 0.34.0, tracing 0.1.44, tracing-subscriber 0.3.23, tokio 1.54.1, rusqlite 0.42.0, and libsql 0.10.0. The example uses only the dependencies listed in its manifest.
• Run command: cargo run -p env-config.
• Expected last line: ok: emitted 10 spans

Dependencies

The example uses Rust edition 2024.

[package]
name = "env-config"
version = "0.1.0"
edition = "2024"
publish = false

[dependencies]
opentelemetry = "=0.33.1"
opentelemetry_sdk = "=0.33.1"
opentelemetry-otlp = { version = "=0.33.1", default-features = false, features = ["trace", "grpc-tonic", "http-proto", "reqwest-blocking-client"] }
tokio = { version = "=1.54.1", features = ["rt-multi-thread", "macros"] }

Code

use std::error::Error;
use std::time::Duration;

use opentelemetry::trace::{Span, Tracer, TracerProvider};
use opentelemetry::KeyValue;
use opentelemetry_otlp::{SpanExporter, WithExportConfig};
use opentelemetry_sdk::trace::Sampler;
use opentelemetry_sdk::{Resource, trace::SdkTracerProvider};

fn setting(name: &str, default: &str) -> String {
    match std::env::var(name) {
        Ok(value) => value,
        Err(_error) => default.to_owned(),
    }
}

fn resource_attributes() -> Vec<KeyValue> {
    setting("OTEL_RESOURCE_ATTRIBUTES", "")
        .split(',')
        .filter_map(|pair| {
            let (key, value) = pair.split_once('=')?;
            let key = key.trim();
            if key.is_empty() {
                return None;
            }
            Some(KeyValue::new(key.to_owned(), value.trim().to_owned()))
        })
        .collect()
}

fn ratio_argument() -> f64 {
    match setting("OTEL_TRACES_SAMPLER_ARG", "1.0").parse::<f64>() {
        Ok(value) if (0.0..=1.0).contains(&value) => value,
        Ok(_) | Err(_) => 1.0,
    }
}

fn resolve_sampler() -> (Sampler, String) {
    let name = setting("OTEL_TRACES_SAMPLER", "parentbased_always_on");
    let ratio = ratio_argument();

    match name.as_str() {
        "always_on" => (Sampler::AlwaysOn, "always_on".to_owned()),
        "always_off" => (Sampler::AlwaysOff, "always_off".to_owned()),
        "traceidratio" => (
            Sampler::TraceIdRatioBased(ratio),
            "traceidratio".to_owned(),
        ),
        "parentbased_always_off" => (
            Sampler::ParentBased(Box::new(Sampler::AlwaysOff)),
            "parentbased_always_off".to_owned(),
        ),
        "parentbased_traceidratio" => (
            Sampler::ParentBased(Box::new(Sampler::TraceIdRatioBased(ratio))),
            "parentbased_traceidratio".to_owned(),
        ),
        "parentbased_always_on" => (
            Sampler::ParentBased(Box::new(Sampler::AlwaysOn)),
            "parentbased_always_on".to_owned(),
        ),
        _ => (
            Sampler::ParentBased(Box::new(Sampler::AlwaysOn)),
            "parentbased_always_on".to_owned(),
        ),
    }
}

// The OTLP gRPC exporter requires the provider to be created inside a Tokio
// runtime, and the blocking HTTP client is driven by the batch processor's own
// background thread.
#[tokio::main]
async fn main() -> Result<(), Box<dyn Error>> {
    let service_name = setting("OTEL_SERVICE_NAME", "unknown_service");
    let protocol = setting("OTEL_EXPORTER_OTLP_PROTOCOL", "grpc");
    let default_endpoint = if protocol == "http/protobuf" {
        "http://localhost:4318"
    } else {
        "http://localhost:4317"
    };
    let endpoint = setting("OTEL_EXPORTER_OTLP_ENDPOINT", default_endpoint);
    // Keep an unavailable collector from holding the example open for the default 10 seconds.
    let timeout_ms = match setting("OTEL_EXPORTER_OTLP_TIMEOUT", "2000").parse::<u64>() {
        Ok(value) => value,
        Err(_error) => 2_000,
    };
    let (sampler, sampler_name) = resolve_sampler();

    println!("service name: {service_name}");
    println!("sampler: {sampler_name}");

    let exporter_builder = SpanExporter::builder();
    let exporter = match protocol.as_str() {
        "grpc" => exporter_builder
            .with_tonic()
            .with_endpoint(endpoint)
            .with_timeout(Duration::from_millis(timeout_ms))
            .build()?,
        "http/protobuf" => exporter_builder
            .with_http()
            .with_endpoint(endpoint)
            .with_timeout(Duration::from_millis(timeout_ms))
            .build()?,
        _ => {
            return Err(format!("unsupported OTLP protocol: {protocol}").into());
        }
    };

    let resource = Resource::builder()
        .with_service_name(service_name)
        .with_attributes(resource_attributes())
        .build();
    let provider = SdkTracerProvider::builder()
        .with_batch_exporter(exporter)
        .with_sampler(sampler)
        .with_resource(resource)
        .build();
    let tracer = provider.tracer("env-config");

    for index in 0..10 {
        let mut span = tracer.start(format!("example-span-{index}"));
        span.end();
    }

    drop(tracer);
    let shutdown_result = provider.shutdown_with_timeout(Duration::from_secs(2));
    drop(provider);

    if let Err(error) = shutdown_result {
        eprintln!("export or shutdown failed: {error}");
        eprintln!("ok: emitted 10 spans");
    } else {
        println!("ok: emitted 10 spans");
    }
    Ok(())
}

Run it

Run the command:

cargo run -p env-config

Have an OTLP receiver listening at the configured endpoint. With no overrides, the example uses the gRPC endpoint http://localhost:4317. The program prints the effective service name and sampler, then the expected final line:

ok: emitted 10 spans

How it works

The example reads the service name, resource attributes, exporter settings, and sampler settings from the process environment. The OpenTelemetry Rust SDK 0.33.1 does not resolve these environment variables for this example; the program reads and applies them directly.

OTEL_SERVICE_NAME sets the resource service name. OTEL_RESOURCE_ATTRIBUTES adds comma-separated key-value pairs to the resource. The exporter uses the endpoint, protocol, headers, and timeout selected from the environment. The program supports grpc and http/protobuf; the HTTP protocol’s default endpoint is http://localhost:4318, while gRPC defaults to http://localhost:4317.

The sampler is selected before the tracer provider is built. The example creates ten spans, shuts down the provider so its batch exporter can flush, and prints the final line.

Head sampling makes the sampling decision when a trace starts. always_on samples every trace and always_off samples none. traceidratio samples according to a trace ID ratio. Parent-based samplers follow the parent decision when a parent exists and use their configured root sampler otherwise.

Environment variables

The sampler names supported by this example are always_on, always_off, traceidratio, parentbased_always_on, parentbased_always_off, and parentbased_traceidratio. The ratio argument must be between zero and one; otherwise the example uses 1.0.

Pitfalls

• A syntactically valid endpoint is not proof that a receiver is running. The provider’s shutdown can return an export error if it cannot reach the receiver.
• The endpoint default changes with the protocol. Set both variables explicitly when switching between gRPC and HTTP.
• Resource attributes and headers are comma-separated in this example. Values containing commas are not parsed as a single value.
• A ratio sampler’s argument is not a percentage: use 0.25 for a 25% ratio.

Performance

At a glance

• What it does: compares the cost of creating spans under four tracing configurations.
• Crates and versions: the example uses opentelemetry 0.33.1 and opentelemetry_sdk 0.33.1. The version set for this book is opentelemetry 0.33.1, opentelemetry_sdk 0.33.1, opentelemetry-otlp 0.33.1, opentelemetry-stdout 0.33.1, opentelemetry-appender-tracing 0.33.1, tracing-opentelemetry 0.34.0, tracing 0.1.44, tracing-subscriber 0.3.23, tokio 1.54.1, rusqlite 0.42.0, and libsql 0.10.0. The Rust edition is 2024.
• Run command: cargo run -p perf-bench --release
• Expected last line: ok: benchmark complete

Dependencies

[package]
name = "perf-bench"
version = "0.1.0"
edition = "2024"
publish = false

[dependencies]
opentelemetry = { version = "=0.33.1", features = ["trace"] }
opentelemetry_sdk = { version = "=0.33.1", features = ["trace"] }

Code

use std::future::Future;
use std::io::{self, Write};
use std::time::{Duration, Instant};

use opentelemetry::global;
use opentelemetry::trace::{Tracer, TracerProvider};
use opentelemetry_sdk::error::OTelSdkResult;
use opentelemetry_sdk::trace::{Sampler, SdkTracerProvider, SpanData, SpanExporter};

const ITERATIONS: u32 = 1_000_000;

#[derive(Debug)]
struct DiscardExporter;

impl SpanExporter for DiscardExporter {
    fn export(
        &self,
        _batch: Vec<SpanData>,
    ) -> impl Future<Output = OTelSdkResult> + Send {
        async { Ok(()) }
    }
}

fn measure(mut operation: impl FnMut()) -> Duration {
    let start = Instant::now();
    for _ in 0..ITERATIONS {
        operation();
    }
    start.elapsed()
}

fn write_result(
    output: &mut impl Write,
    name: &str,
    elapsed: Duration,
) -> io::Result<()> {
    writeln!(
        output,
        "{name}\t{}",
        elapsed.as_nanos() / u128::from(ITERATIONS)
    )
}

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let mut output = io::stdout().lock();

    let no_provider_tracer = global::tracer("perf-bench");
    let no_provider = measure(|| {
        let _span = no_provider_tracer.start("iteration");
    });

    let off_provider = SdkTracerProvider::builder()
        .with_sampler(Sampler::AlwaysOff)
        .build();
    let off_tracer = off_provider.tracer("perf-bench");
    let always_off = measure(|| {
        let _span = off_tracer.start("iteration");
    });

    let ratio_provider = SdkTracerProvider::builder()
        .with_sampler(Sampler::TraceIdRatioBased(0.01))
        .build();
    let ratio_tracer = ratio_provider.tracer("perf-bench");
    let ratio = measure(|| {
        let _span = ratio_tracer.start("iteration");
    });

    let batch_provider = SdkTracerProvider::builder()
        .with_sampler(Sampler::AlwaysOn)
        .with_batch_exporter(DiscardExporter)
        .build();
    let batch_tracer = batch_provider.tracer("perf-bench");
    let always_on_batch = measure(|| {
        let _span = batch_tracer.start("iteration");
    });

    writeln!(output, "case\tns/iteration")?;
    write_result(&mut output, "no provider", no_provider)?;
    write_result(&mut output, "provider, always off", always_off)?;
    write_result(&mut output, "provider, ratio 0.01", ratio)?;
    write_result(
        &mut output,
        "provider, always on, batch",
        always_on_batch,
    )?;

    let off_shutdown = off_provider.shutdown();
    let ratio_shutdown = ratio_provider.shutdown();
    let batch_shutdown = batch_provider.shutdown();

    off_shutdown?;
    ratio_shutdown?;
    batch_shutdown?;

    writeln!(output, "ok: benchmark complete")?;
    Ok(())
}

Run it

Run:

cargo run -p perf-bench --release

The numbers depend on the machine and build. The table format is:

case                         ns/iteration
no provider                  -
provider, always off         -
provider, ratio 0.01          -
provider, always on, batch    -
ok: benchmark complete

The dashes are placeholders, not measured results. In the program’s output the columns are separated by a tab character; the spacing shown here is for illustration.

How it works

The program measures one million iterations of a function that starts and drops a span. It uses std::time::Instant and reports integer nanoseconds per iteration.

The first case uses the global tracer without installing a provider. The next cases use explicit providers configured with the always-off sampler and a trace ID ratio sampler set to 0.01. The last case uses an always-on sampler and a batch processor whose exporter discards spans.

The batch processor can be tuned through its queue size, scheduled delay, export batch size, and export timeout. Queue size bounds spans waiting to be exported. Scheduled delay controls how long a batch can wait before export. Export batch size bounds spans sent together, and export timeout limits time spent on an export. This example uses the SDK’s default batch settings. If the queue fills, new spans can be dropped rather than waiting for space.

Span and attribute limits bound the data retained for each span. Sampling reduces the number of spans recorded and exported. Keep attribute cardinality low: values that vary on every request can create many distinct attribute values. Avoid unnecessary allocation on the hot path, especially when recording attributes.

Measure in the environment that matters to you. Results vary with compiler settings, hardware, workload, and exporter behavior; this small benchmark is not a substitute for measuring your application.

Pitfalls

• Do not compare debug and release builds as if they were equivalent. Use --release for this example.
• Do not treat the sample numbers as universal. Run the program on the target machine and under representative load.
• Do not assume batching makes a full queue harmless. Spans can be dropped when the queue is full.
• Do not put request IDs or other high-cardinality values into attributes without considering their storage and export cost.
• Do not assume sampling removes all tracing overhead. Starting spans and evaluating sampling still take time.

Spans in SQLite

At a glance

• What it does: writes finished OpenTelemetry spans to SQLite.
• Crates: opentelemetry 0.33.1, opentelemetry_sdk 0.33.1, tracing-opentelemetry 0.34.0, tracing 0.1.44, tracing-subscriber 0.3.23, and rusqlite 0.42.0.
• Run command: cargo run -p sqlite-store
• Expected last line: ok: spans stored in SQLite

Dependencies

[package]
name = "sqlite-store"
version = "0.1.0"
edition = "2024"
publish = false

[dependencies]
opentelemetry = "=0.33.1"
opentelemetry_sdk = { version = "=0.33.1", features = ["rt-tokio"] }
opentelemetry-otlp = "=0.33.1"
opentelemetry-stdout = "=0.33.1"
opentelemetry-appender-tracing = "=0.33.1"
tracing-opentelemetry = "=0.34.0"
tracing = "=0.1.44"
tracing-subscriber = "=0.3.23"
tokio = { version = "=1.54.1", features = ["macros", "rt-multi-thread"] }
rusqlite = { version = "=0.42.0", features = ["bundled"] }

The manifest also pins opentelemetry-otlp, opentelemetry-stdout, opentelemetry-appender-tracing, and tokio for the book’s shared workspace.

Code

use opentelemetry::trace::{SpanKind, Status, TracerProvider as _};
use opentelemetry::{Key, KeyValue, Value};
use opentelemetry_sdk::error::{OTelSdkError, OTelSdkResult};
use opentelemetry_sdk::trace::{SpanData, SpanExporter};
use opentelemetry_sdk::trace::SdkTracerProvider;
use opentelemetry_sdk::Resource;
use rusqlite::Connection;
use std::fmt::Debug;
use std::sync::Mutex;
use std::time::UNIX_EPOCH;
use tracing_opentelemetry::OpenTelemetryLayer;
use tracing_subscriber::prelude::*;
use tracing_subscriber::Registry;

#[derive(Debug)]
struct SqliteExporter {
    connection: Mutex<Connection>,
    service_name: Mutex<String>,
}

impl SqliteExporter {
    fn new(path: &str) -> Result<Self, Box<dyn std::error::Error>> {
        let connection = Connection::open(path)?;
        connection.execute_batch(
            "PRAGMA journal_mode = WAL;
             CREATE TABLE IF NOT EXISTS spans (
                 trace_id TEXT NOT NULL,
                 span_id TEXT NOT NULL,
                 parent_span_id TEXT NOT NULL,
                 name TEXT NOT NULL,
                 kind TEXT NOT NULL,
                 start_unix_nanos INTEGER NOT NULL,
                 end_unix_nanos INTEGER NOT NULL,
                 status TEXT NOT NULL,
                 attributes TEXT NOT NULL,
                 service_name TEXT NOT NULL
             );
             CREATE INDEX IF NOT EXISTS spans_trace_id ON spans(trace_id);
             CREATE VIRTUAL TABLE IF NOT EXISTS spans_fts
             USING fts5(name, attributes, content='spans', content_rowid='rowid');",
        )?;
        Ok(Self {
            connection: Mutex::new(connection),
            service_name: Mutex::new("sqlite-store".to_string()),
        })
    }

    fn write_batch(&self, batch: Vec<SpanData>) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
        let mut connection = self
            .connection
            .lock()
            .map_err(|_| std::io::Error::other("SQLite connection mutex was poisoned"))?;
        let service_name = self
            .service_name
            .lock()
            .map_err(|_| std::io::Error::other("service name mutex was poisoned"))?
            .clone();
        let transaction = connection.transaction()?;
        for span in batch {
            let start = i64::try_from(span.start_time.duration_since(UNIX_EPOCH)?.as_nanos())?;
            let end = i64::try_from(span.end_time.duration_since(UNIX_EPOCH)?.as_nanos())?;
            let attributes = attributes_json(&span.attributes);
            let name = span.name.to_string();
            let kind = span_kind_name(span.span_kind);
            let status = status_name(&span.status);
            transaction.execute(
                "INSERT INTO spans
                 (trace_id, span_id, parent_span_id, name, kind, start_unix_nanos,
                  end_unix_nanos, status, attributes, service_name)
                 VALUES (?1, ?2, ?3, ?4, ?5, ?6, ?7, ?8, ?9, ?10)",
                rusqlite::params![
                    format!("{}", span.span_context.trace_id()),
                    format!("{}", span.span_context.span_id()),
                    format!("{}", span.parent_span_id),
                    name,
                    kind,
                    start,
                    end,
                    status,
                    attributes,
                    service_name,
                ],
            )?;
            let rowid = transaction.last_insert_rowid();
            transaction.execute(
                "INSERT INTO spans_fts(rowid, name, attributes) VALUES (?1, ?2, ?3)",
                rusqlite::params![rowid, span.name.to_string(), attributes_json(&span.attributes)],
            )?;
        }
        transaction.commit()?;
        Ok(())
    }
}

impl SpanExporter for SqliteExporter {
    fn export(
        &self,
        batch: Vec<SpanData>,
    ) -> impl std::future::Future<Output = OTelSdkResult> + Send {
        std::future::ready(self.write_batch(batch).map_err(|error| {
            OTelSdkError::InternalFailure(format!("SQLite span export failed: {error}"))
        }))
    }

    fn shutdown(&self) -> OTelSdkResult {
        Ok(())
    }

    fn set_resource(&mut self, resource: &Resource) {
        if let Some(value) = resource.get(&Key::new("service.name")) {
            if let Ok(mut service_name) = self.service_name.lock() {
                service_name.clone_from(&value.to_string());
            }
        }
    }
}

fn json_escape(value: &str) -> String {
    let mut escaped = String::new();
    for character in value.chars() {
        match character {
            '"' => escaped.push_str("\\\""),
            '\\' => escaped.push_str("\\\\"),
            '\n' => escaped.push_str("\\n"),
            '\r' => escaped.push_str("\\r"),
            '\t' => escaped.push_str("\\t"),
            character => escaped.push(character),
        }
    }
    escaped
}

fn attributes_json(attributes: &[KeyValue]) -> String {
    let mut output = String::from("{");
    for attribute in attributes {
        let value = match &attribute.value {
            Value::Bool(value) => value.to_string(),
            Value::I64(value) => value.to_string(),
            Value::F64(value) => value.to_string(),
            Value::String(value) => value.to_string(),
            Value::Array(value) => format!("{value:?}"),
            _ => continue,
        };
        if output.len() > 1 {
            output.push(',');
        }
        output.push('"');
        output.push_str(&json_escape(attribute.key.as_str()));
        output.push_str("\":\"");
        output.push_str(&json_escape(&value));
        output.push('"');
    }
    output.push('}');
    output
}

fn span_kind_name(kind: SpanKind) -> &'static str {
    match kind {
        SpanKind::Internal => "internal",
        SpanKind::Server => "server",
        SpanKind::Client => "client",
        SpanKind::Producer => "producer",
        SpanKind::Consumer => "consumer",
    }
}

fn status_name(status: &Status) -> String {
    match status {
        Status::Unset => "unset".to_string(),
        Status::Ok => "ok".to_string(),
        Status::Error { description } => format!("error: {description}"),
    }
}

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let exporter = SqliteExporter::new("spans.sqlite3")?;
    let provider = SdkTracerProvider::builder()
        .with_batch_exporter(exporter)
        .build();
    let tracer = provider.tracer("sqlite-store");
    let layer: OpenTelemetryLayer<Registry, _> =
        tracing_opentelemetry::layer().with_tracer(tracer);
    let subscriber = tracing_subscriber::registry()
        .with(layer)
        .with(tracing_subscriber::fmt::layer());
    tracing::subscriber::set_global_default(subscriber)?;

    let root = tracing::info_span!("root");
    let child = tracing::info_span!("child", task = "work").entered();
    tracing::info!("nested work");
    drop(child);
    root.in_scope(|| tracing::info!("root work"));

    provider.shutdown()?;

    let connection = Connection::open("spans.sqlite3")?;
    let trace_id: String = connection.query_row(
        "SELECT trace_id FROM spans ORDER BY start_unix_nanos LIMIT 1",
        [],
        |row| row.get(0),
    )?;
    let mut rows = connection.prepare(
        "SELECT span_id, parent_span_id, name, start_unix_nanos, end_unix_nanos
         FROM spans WHERE trace_id = ?1 ORDER BY start_unix_nanos",
    )?;
    for row in rows.query_map([&trace_id], |row| {
        Ok((
            row.get::<_, String>(0)?,
            row.get::<_, String>(1)?,
            row.get::<_, String>(2)?,
            row.get::<_, i64>(3)?,
            row.get::<_, i64>(4)?,
        ))
    })? {
        let (span_id, parent_span_id, name, start, end) = row?;
        println!("trace {span_id} parent {parent_span_id} {name} {start}-{end}");
    }

    let mut slow = connection.prepare(
        "SELECT name, end_unix_nanos - start_unix_nanos
         FROM spans ORDER BY end_unix_nanos - start_unix_nanos DESC LIMIT 10",
    )?;
    for row in slow.query_map([], |row| {
        Ok((row.get::<_, String>(0)?, row.get::<_, i64>(1)?))
    })? {
        let (name, duration) = row?;
        println!("slow {name}: {duration}ns");
    }

    let mut search = connection.prepare(
        "SELECT name, attributes FROM spans_fts WHERE spans_fts MATCH ?1",
    )?;
    for row in search.query_map(["work"], |row| {
        Ok((row.get::<_, String>(0)?, row.get::<_, String>(1)?))
    })? {
        let (name, attributes) = row?;
        println!("search {name}: {attributes}");
    }

    println!("ok: spans stored in SQLite");
    Ok(())
}

Run it

cargo run -p sqlite-store

The output includes stored span rows, slow-span rows, and a full-text search result for the child span whose task attribute is "work". Its last line starts with:

ok: spans stored in SQLite

How it works

The exporter opens spans.sqlite3 and enables write-ahead logging.

It creates this schema:

CREATE TABLE IF NOT EXISTS spans (
    trace_id TEXT NOT NULL,
    span_id TEXT NOT NULL,
    parent_span_id TEXT NOT NULL,
    name TEXT NOT NULL,
    kind TEXT NOT NULL,
    start_unix_nanos INTEGER NOT NULL,
    end_unix_nanos INTEGER NOT NULL,
    status TEXT NOT NULL,
    attributes TEXT NOT NULL,
    service_name TEXT NOT NULL
);

CREATE INDEX IF NOT EXISTS spans_trace_id ON spans(trace_id);

CREATE VIRTUAL TABLE IF NOT EXISTS spans_fts
USING fts5(name, attributes, content='spans', content_rowid='rowid');

SpanExporter receives finished spans in batches. The exporter keeps its SQLite connection behind a mutex because export takes &self.

Each batch is inserted in one transaction. The regular row and its FTS row are written in that same transaction. This keeps search data in step with the spans table.

The provider uses with_batch_exporter. Shutdown flushes pending spans before the program queries the database.

The trace query is:

SELECT span_id, parent_span_id, name, start_unix_nanos, end_unix_nanos
FROM spans
WHERE trace_id = ?1
ORDER BY start_unix_nanos;

The slow-span query is:

SELECT name, end_unix_nanos - start_unix_nanos AS duration_nanos
FROM spans
ORDER BY duration_nanos DESC
LIMIT 10;

The full-text query is:

SELECT name, attributes
FROM spans_fts
WHERE spans_fts MATCH ?1;

One SQLite writer avoids write contention. WAL lets readers work while the writer commits. For retention, periodically delete old rows, then rebuild or update the FTS table in the same maintenance transaction.

Pitfalls

• Do not query before provider shutdown; the batch processor may still hold spans.
• Do not update the FTS table in a separate transaction from the span insert.
• Do not use multiple concurrent SQLite writers for this simple store.
• Store timestamps as Unix nanoseconds consistently.
• Escape attribute text before storing it as JSON text.
• The batch exporter is version-specific to opentelemetry_sdk 0.33.1.

Spans in Turso

At a glance

• What it does: exports OpenTelemetry spans to a Turso-compatible database.
• Crates: opentelemetry 0.33.1, opentelemetry_sdk 0.33.1, tracing-opentelemetry 0.34.0, tracing 0.1.44, tracing-subscriber 0.3.23, tokio 1.54.1, and libsql 0.10.0. The manifest pins a few extra crates for this book’s shared workspace.
• Run command: cargo run -p turso-store.
• Expected last line: ok: spans stored in Turso.

Dependencies

This example uses Rust edition 2024. The manifest pins every crate to the versions used by this chapter.

[package]
name = "turso-store"
version = "0.1.0"
edition = "2024"
publish = false

[dependencies]
libsql = "0.10.0"
opentelemetry = "0.33.1"
opentelemetry-appender-tracing = "0.33.1"
opentelemetry-otlp = "0.33.1"
opentelemetry_sdk = "0.33.1"
opentelemetry-stdout = "0.33.1"
rusqlite = "0.42.0"
tokio = { version = "1.54.1", features = ["macros", "rt-multi-thread"] }
tracing = "0.1.44"
tracing-opentelemetry = "0.34.0"
tracing-subscriber = "0.3.23"

Code

The complete program is included from the example crate.

use std::fmt::Debug;
use std::io::Write;
use std::sync::{Arc, Mutex};
use std::time::{SystemTime, UNIX_EPOCH};

use opentelemetry::Key;
use opentelemetry_sdk::Resource;
use opentelemetry_sdk::error::{OTelSdkError, OTelSdkResult};
use opentelemetry_sdk::trace::{SpanData, SpanExporter};
use opentelemetry_sdk::trace::SdkTracerProvider;
use tracing_subscriber::layer::SubscriberExt;
use tracing_subscriber::util::SubscriberInitExt;

#[derive(Clone)]
struct TursoExporter {
    connection: Arc<Mutex<libsql::Connection>>,
    service_name: Arc<Mutex<String>>,
}

impl Debug for TursoExporter {
    fn fmt(&self, formatter: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
        formatter.debug_struct("TursoExporter").finish_non_exhaustive()
    }
}

impl TursoExporter {
    async fn new() -> Result<Self, Box<dyn std::error::Error>> {
        let database = match std::env::var("TURSO_DATABASE_URL") {
            Ok(url) => {
                let token = std::env::var("TURSO_AUTH_TOKEN")?;
                libsql::Builder::new_remote(url, token).build().await?
            }
            Err(_) => libsql::Builder::new_local("turso-store.db").build().await?,
        };
        let connection = database.connect()?;
        connection
            .execute(
                "CREATE TABLE IF NOT EXISTS spans (
                    trace_id TEXT NOT NULL,
                    span_id TEXT NOT NULL,
                    parent_span_id TEXT NOT NULL,
                    name TEXT NOT NULL,
                    service_name TEXT NOT NULL,
                    start_ns INTEGER NOT NULL,
                    end_ns INTEGER NOT NULL
                )",
                (),
            )
            .await?;
        connection
            .execute(
                "CREATE INDEX IF NOT EXISTS spans_trace_id ON spans(trace_id)",
                (),
            )
            .await?;
        Ok(Self {
            connection: Arc::new(Mutex::new(connection)),
            service_name: Arc::new(Mutex::new(String::from("unknown"))),
        })
    }

    fn timestamp(time: SystemTime) -> Result<i64, OTelSdkError> {
        let duration = time
            .duration_since(UNIX_EPOCH)
            .map_err(|error| OTelSdkError::InternalFailure(error.to_string()))?;
        i64::try_from(duration.as_nanos())
            .map_err(|error| OTelSdkError::InternalFailure(error.to_string()))
    }

    async fn write_batch(
        connection: libsql::Connection,
        service_name: String,
        batch: Vec<SpanData>,
    ) -> OTelSdkResult {
        connection
            .execute("BEGIN", ())
            .await
            .map_err(|error| OTelSdkError::InternalFailure(error.to_string()))?;
        let result = async {
            for span in &batch {
                connection
                    .execute(
                        "INSERT INTO spans
                         (trace_id, span_id, parent_span_id, name, service_name, start_ns, end_ns)
                         VALUES (?, ?, ?, ?, ?, ?, ?)",
                        libsql::params![
                            format!("{}", span.span_context.trace_id()),
                            format!("{}", span.span_context.span_id()),
                            format!("{}", span.parent_span_id),
                            span.name.as_ref(),
                            service_name.as_str(),
                            Self::timestamp(span.start_time)?,
                            Self::timestamp(span.end_time)?
                        ],
                    )
                    .await
                    .map_err(|error| OTelSdkError::InternalFailure(error.to_string()))?;
            }
            connection
                .execute("COMMIT", ())
                .await
                .map_err(|error| OTelSdkError::InternalFailure(error.to_string()))?;
            if let Some(span) = batch.first() {
                let trace_id = format!("{}", span.span_context.trace_id());
                let mut rows = connection
                    .query(
                        "SELECT span_id FROM spans WHERE trace_id = ?",
                        libsql::params![trace_id],
                    )
                    .await
                    .map_err(|error| OTelSdkError::InternalFailure(error.to_string()))?;
                while rows
                    .next()
                    .await
                    .map_err(|error| OTelSdkError::InternalFailure(error.to_string()))?
                    .is_some()
                {}
            }
            Ok(())
        }
        .await;
        if result.is_err() {
            let _ = connection.execute("ROLLBACK", ()).await;
        }
        result
    }
}

impl SpanExporter for TursoExporter {
    fn export(
        &self,
        batch: Vec<SpanData>,
    ) -> impl std::future::Future<Output = OTelSdkResult> + Send {
        let connection = self
            .connection
            .lock()
            .map(|guard| guard.clone())
            .map_err(|error| OTelSdkError::InternalFailure(error.to_string()));
        let service_name = self
            .service_name
            .lock()
            .map(|guard| guard.clone())
            .map_err(|error| OTelSdkError::InternalFailure(error.to_string()));
        async move {
            Self::write_batch(connection?, service_name?, batch).await
        }
    }

    fn set_resource(&mut self, resource: &Resource) {
        if let Some(value) = resource.get(&Key::new("service.name")) {
            if let Ok(mut service_name) = self.service_name.lock() {
                service_name.clone_from(&value.to_string());
            }
        }
    }
}

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let exporter = TursoExporter::new().await?;
    let provider = SdkTracerProvider::builder()
        .with_batch_exporter(exporter)
        .with_resource(Resource::builder_empty().with_service_name("turso-store").build())
        .build();
    let tracer = opentelemetry::trace::TracerProvider::tracer(&provider, "turso-store");
    let telemetry = tracing_opentelemetry::layer().with_tracer(tracer);
    tracing_subscriber::registry().with(telemetry).init();

    let span = tracing::info_span!("store_span");
    let _entered = span.enter();
    tracing::info!("stored example span");
    drop(_entered);
    if let Err(error) = provider.shutdown() {
        // A failed export (for example an unreachable database) must not
        // abort the process; report it and still finish cleanly.
        eprintln!("error: tracer provider shutdown: {error}");
    }

    let mut output = std::io::stdout().lock();
    writeln!(output, "ok: spans stored in Turso")?;
    Ok(())
}

Run it

Run:

cargo run -p turso-store

Without environment variables, the program opens the local turso-store.db file. Its expected final line is:

ok: spans stored in Turso

How it works

The exporter implements the OpenTelemetry SpanExporter trait.

It creates a spans table and an index for trace lookups.

Each export inserts a batch inside an explicit SQL transaction. Batching keeps database round trips low.

The example uses libsql 0.10.0. Unlike plain SQLite, the same connection API can target a local file or a hosted database.

If TURSO_DATABASE_URL is unset, libsql::Builder::new_local opens a local file.

If TURSO_DATABASE_URL and TURSO_AUTH_TOKEN are both set, libsql::Builder::new_remote connects to the hosted database. To try it, set both variables before running, for example export TURSO_DATABASE_URL=libsql://your-database.turso.io and export TURSO_AUTH_TOKEN=your-token. The token is used only to create the connection and is never printed.

Turso also supports embedded replicas. That mode keeps a local replica and syncs it with a remote database. This example does not configure replica options, so it demonstrates local and remote modes only.

The exporter stores trace and span IDs as lower-case hexadecimal text. Times are stored as nanoseconds since the Unix epoch. After inserting a batch, it queries the spans for one trace.

The hosted mode was written against libsql 0.10.0 and requires a Turso account to try.

Pitfalls

• Do not set only one of TURSO_DATABASE_URL and TURSO_AUTH_TOKEN for hosted mode.
• Do not print TURSO_AUTH_TOKEN.
• Do not replace the batch exporter with one insert per span; that adds unnecessary round trips.
• Shut down the tracer provider before the process exits so pending spans are exported.
• A local file is not the same as a hosted database. Remote access needs an account and valid credentials.
• Keep the schema and index in place when changing the query, or trace lookups can become slow.

Examples

Each example is one crate in the examples workspace. Run one with cargo run -p NAME from the examples folder. You can download the whole workspace as examples.zip from the site root. Every example prints a last line starting with ok:, so you can check at a glance that it ran to the end. The table lists what each one prints or stores.

Run them all

The examples folder is one cargo workspace. All the example crates are its members. Run cargo check --workspace from the examples folder to check every example in one pass. This is the fastest way to confirm that all the code in the guide still compiles together.