Distributed Tracing in .NET: Following Requests Across Services

When a user clicks a button and the request travels through an API gateway, a backend service, a database, and a cache before returning a response, how do you understand what happened? Distributed tracing gives you a complete timeline of every operation, across every service, tied together by a single trace ID.

The Building Blocks

.NET's distributed tracing is built on two types from System.Diagnostics:

• Activity — represents a single unit of work (a span in OpenTelemetry terminology)
• ActivitySource — a factory that creates activities

ASP.NET Core, HttpClient, and database drivers create activities automatically. The framework also handles context propagation — passing the trace ID between services via HTTP headers using the W3C Trace Context standard.

How Context Propagation Works

When Service A calls Service B via HTTP, the following happens automatically:

1. Service A has an active Activity with a trace ID, say abc123
2. HttpClient adds a traceparent header to the outgoing request: 00-abc123-def456-01
3. Service B's ASP.NET Core middleware reads this header
4. Service B creates a new Activity as a child of the incoming context
5. Both activities share the same trace ID abc123

This chain continues through every service call. The result is a tree of spans, all linked by a common trace ID.

Trace abc123
├── GET /checkout (frontend, 450ms)
│   ├── POST /orders (order-api, 320ms)
│   │   ├── SELECT ... (postgresql, 15ms)
│   │   ├── PUBLISH order.created (rabbitmq, 5ms)
│   │   └── SET order:123 (redis, 2ms)
│   └── GET /inventory (inventory-api, 80ms)
│       └── SELECT ... (postgresql, 12ms)

Creating Custom Spans

Automatic instrumentation covers HTTP, gRPC, and database calls. For your own business logic, create an ActivitySource:

public static class Telemetry
{
    public static readonly ActivitySource Source = new("MyApp.Checkout");
}

Use it to create spans around meaningful operations:

public async Task<CheckoutResult> ProcessCheckoutAsync(Cart cart)
{
    using var activity = Telemetry.Source.StartActivity("ProcessCheckout");
    activity?.SetTag("cart.item_count", cart.Items.Count);
    activity?.SetTag("cart.customer_id", cart.CustomerId);

    // Validate stock
    using (Telemetry.Source.StartActivity("ValidateStock"))
    {
        foreach (var item in cart.Items)
        {
            var available = await _inventoryClient
                .CheckStockAsync(item.ProductId);
            if (!available)
            {
                activity?.SetStatus(ActivityStatusCode.Error,
                    $"Product {item.ProductId} out of stock");
                throw new OutOfStockException(item.ProductId);
            }
        }
    }

    // Process payment
    using (var paymentSpan = Telemetry.Source.StartActivity("ProcessPayment"))
    {
        paymentSpan?.SetTag("payment.amount", cart.Total);
        await _paymentService.ChargeAsync(cart.CustomerId, cart.Total);
    }

    // Create order
    using (Telemetry.Source.StartActivity("CreateOrder"))
    {
        return await _orderService.CreateAsync(cart);
    }
}

Recording Errors

When an exception occurs, record it on the span:

try
{
    await ProcessPaymentAsync(amount);
}
catch (PaymentFailedException ex)
{
    Activity.Current?.SetStatus(ActivityStatusCode.Error, ex.Message);
    Activity.Current?.AddEvent(new ActivityEvent("PaymentFailed",
        tags: new ActivityTagsCollection
        {
            { "exception.type", ex.GetType().Name },
            { "exception.message", ex.Message }
        }));
    throw;
}

The Aspire dashboard and production tracing tools will highlight these error spans, making failed requests easy to identify.

Adding Events to Spans

Spans can contain events — timestamped annotations that mark significant moments:

using var activity = Telemetry.Source.StartActivity("ProcessOrder");

activity?.AddEvent(new ActivityEvent("InventoryValidated"));

// ... process payment ...

activity?.AddEvent(new ActivityEvent("PaymentProcessed",
    tags: new ActivityTagsCollection
    {
        { "payment.transaction_id", transactionId }
    }));

Events appear as markers on the span's timeline, giving you precise timing of each step without creating separate child spans.

Baggage: Passing Context Beyond Headers

Sometimes you need to pass context that is not a span attribute — for example, a customer tier that affects how every downstream service handles the request:

Activity.Current?.SetBaggage("customer.tier", "premium");

Baggage propagates to all downstream services via HTTP headers. Any service in the chain can read it:

var tier = Activity.Current?.GetBaggageItem("customer.tier");

Use baggage sparingly — it adds overhead to every HTTP call in the trace.

Sampling

In production, tracing every single request generates enormous volumes of data. Configure sampling to capture a representative subset:

builder.Services.AddOpenTelemetry()
    .WithTracing(tracing =>
    {
        tracing.SetSampler(new TraceIdRatioBasedSampler(0.1)); // 10%
    });

For error investigation, consider always sampling failed requests using a custom sampler or tail-based sampling in your collector.

Distributed tracing turns opaque, multi-service request flows into clear, inspectable timelines. .NET provides the primitives, ASP.NET Core provides the automatic instrumentation, and Aspire provides the dashboard to visualise it all.