Observability

Modern .NET has observability built into the runtime. The System.Diagnostics APIs -- Activity, ActivitySource, and Meter -- are the primitives the framework itself uses to describe what your code is doing, and they ship in the base class library with no extra dependency. OpenTelemetry (OTel) is the vendor-neutral standard that sits on top: it listens to those primitives, batches the signals, and exports them to whatever backend you run. You instrument against the .NET APIs, and OTel handles collection and shipping.

The big win is that .NET's own instrumentation already speaks OTel. ASP.NET Core emits a span for every inbound request, HttpClient emits one for every outbound call, and EF Core emits one for every database query -- all without a line of manual code. Because those spans propagate context across process boundaries, you get a distributed trace stitched together across services almost for free, then export it to any backend -- Jaeger, Tempo, Honeycomb, Azure Monitor, Grafana -- over OTLP. Your job is to add a thin layer of business-specific spans, metrics, and logs on top of that foundation.

The Three Signals

OpenTelemetry defines three signals, and each maps cleanly onto a .NET primitive. Traces are Activity objects produced by an ActivitySource -- a span is just an Activity with a start, an end, and a parent. Metrics are instruments (counters, histograms, gauges) created from a Meter. Logs are plain ILogger calls with structured properties. The point of OTel is that it correlates all three: every log line and every metric exemplar carries the trace and span ID of the operation that produced it, so you can pivot from a slow trace to its logs to the metrics it moved without leaving the operation.

Wiring Up OpenTelemetry

Registration happens once in the host builder. You configure a resource (the identity of the service that shows up on every signal), then opt into tracing and metrics, adding the auto-instrumentation packages for the frameworks you use plus your own sources, and finally an exporter.

builder.Services.AddOpenTelemetry()
    .ConfigureResource(r => r.AddService("orders-api"))
    .WithTracing(t => t
        .AddAspNetCoreInstrumentation()
        .AddHttpClientInstrumentation()
        .AddEntityFrameworkCoreInstrumentation()
        .AddSource("Orders")              // your custom ActivitySource
        .AddOtlpExporter())
    .WithMetrics(m => m
        .AddAspNetCoreInstrumentation()
        .AddRuntimeInstrumentation()
        .AddMeter("Orders")
        .AddOtlpExporter());

The auto-instrumentation lines are doing most of the work. AddAspNetCoreInstrumentation gives you a server span per request with route, status code, and timing; AddHttpClientInstrumentation gives you a client span per outbound call, and crucially injects the traceparent header so the next service continues the same trace; AddEntityFrameworkCoreInstrumentation gives you a span per SQL command with the statement text. AddRuntimeInstrumentation on the metrics side emits GC, thread pool, and allocation counters that you would otherwise have to scrape by hand.

AddOtlpExporter is the shipping mechanism. By default it sends OTLP over gRPC to localhost:4317, which is the convention for a local OpenTelemetry Collector sidecar, but everything is overridable through standard environment variables -- OTEL_EXPORTER_OTLP_ENDPOINT, OTEL_EXPORTER_OTLP_HEADERS, OTEL_SERVICE_NAME -- so you point at a different backend per environment without touching code. In production the usual shape is: app exports to a Collector, the Collector handles batching, retries, fan-out, and sampling, and your service stays ignorant of which backend is on the other end.

Custom Tracing with ActivitySource

Auto-instrumentation tells you that a request happened and that it hit the database, but it cannot name your business operations. When you want a span around "process this order" -- a unit of work that spans several calls and deserves its own line on the trace -- you create a static ActivitySource and start an activity from it.

private static readonly ActivitySource Source = new("Orders");

public async Task<Order> ProcessAsync(string id, CancellationToken ct)
{
    using var activity = Source.StartActivity("ProcessOrder");
    activity?.SetTag("order.id", id);
    try
    {
        return await _store.GetAsync(id, ct)
            ?? throw new OrderNotFoundException(id);
    }
    catch (Exception ex)
    {
        activity?.SetStatus(ActivityStatusCode.Error);
        activity?.AddException(ex);
        throw;
    }
}

Several details earn their place here. The using is what ends the span: when activity is disposed at the end of the method, its duration is recorded and it is handed to the exporter. StartActivity returns null when nothing is listening to the source -- when this source is not registered or the trace was sampled out -- which is why every call is null-conditional (activity?). That null path is the zero-cost path: if no one cares, you pay nothing beyond a null check, so you can leave instrumentation in hot code without guilt. Tags set via SetTag become span attributes, which are indexed and searchable in the backend, so order.id lets you find the exact trace for a specific order later.

Context propagation is the part that makes this distributed rather than local. The current Activity flows automatically across every await via AsyncLocal, so a span you start in one method is the parent of spans started in the methods it awaits, with no plumbing. It also flows across process boundaries: the HttpClient and gRPC instrumentation serialize the current trace and span ID into the W3C traceparent header on the way out, and the receiving service's ASP.NET Core instrumentation reads it back in and continues the same trace. That single header is what turns a pile of per-service spans into one connected distributed trace.

Custom Metrics with Meter

Traces tell you about individual operations; metrics tell you about aggregate behavior over time. You create a static Meter and ask it for instruments, then record against those instruments at the call site.

private static readonly Meter Meter = new("Orders");
private static readonly Counter<long> OrdersProcessed =
    Meter.CreateCounter<long>("orders.processed");
private static readonly Histogram<double> ProcessingTime =
    Meter.CreateHistogram<double>("orders.processing.duration", "ms");

// at the call site
OrdersProcessed.Add(1, new KeyValuePair<string, object?>("status", "completed"));

The instrument type encodes how the value behaves, and picking the right one matters. A Counter<T> only ever goes up and is meant for cumulative totals -- requests served, orders processed -- which the backend differentiates into a rate. An UpDownCounter<T> can move in both directions, for things like queue depth or active connections. A Histogram<T> records a distribution of values and is what you want for latencies and sizes, because it lets the backend compute percentiles rather than just an average. An ObservableGauge<T> is sampled on collection via a callback rather than pushed, which suits values you can read on demand like cache size or pool utilization.

Tags are passed at record time, not at creation time, and they become the dimensions you slice by -- here status lets you break the count into completed versus failed. This is also where the most expensive mistake lives: every distinct combination of tag values is a separate time series, so tagging by a bounded set like status or region is fine, but tagging by an unbounded value like order ID or user ID multiplies your series without limit and will overwhelm the backend. Keep dimensions low-cardinality by construction.

Structured Logging

ILogger is the logging signal, and the rule that makes it useful is to log with message templates rather than interpolated strings. The named placeholders are captured as structured properties, so {OrderId} is stored as a queryable field rather than baked into an opaque string -- you can filter on it in the backend, which is impossible once it has been flattened by interpolation.

// structured: orderId is a captured property, not just text
logger.LogInformation("Processed order {OrderId} in {Elapsed}ms", id, elapsed);

[LoggerMessage(Level = LogLevel.Warning, Message = "Payment declined for {OrderId}")]
public static partial void PaymentDeclined(this ILogger logger, string orderId);

The template form is the everyday default. The [LoggerMessage] source generator is the high-performance form: it generates the logging method at compile time, avoids boxing the arguments, and skips the formatting work entirely when the log level is disabled, which makes it the right choice for hot paths where the per-call overhead of the template form would show up in a profile. Both forms produce the same structured output -- the difference is cost, not capability.

When OTel logging is wired in, every log record is automatically stamped with the current trace and span ID. That stamp is the bridge between signals: a log line emitted inside an Activity carries that activity's IDs, so from a single log entry in your backend you can jump straight to the full distributed trace it belongs to, and from a slow span you can pull up exactly the logs it produced.

Health Checks

Health is a separate concern from tracing, served by Microsoft.Extensions.Diagnostics.HealthChecks. You register checks and expose them as endpoints, splitting liveness (is the process alive and should it be restarted) from readiness (is it able to serve traffic right now, given dependencies like the database). Orchestrators and load balancers consume these: Kubernetes restarts a pod that fails its liveness probe but only withholds traffic from one that fails readiness.

builder.Services.AddHealthChecks()
    .AddCheck("self", () => HealthCheckResult.Healthy(), tags: ["live"])
    .AddDbContextCheck<AppDbContext>(tags: ["ready"]);

app.MapHealthChecks("/health/live",
    new() { Predicate = c => c.Tags.Contains("live") });
app.MapHealthChecks("/health/ready",
    new() { Predicate = c => c.Tags.Contains("ready") });

The tag predicate is what keeps the two probes meaningfully distinct. Liveness should check almost nothing -- if it depends on the database, a database blip will trigger pod restarts that make the outage worse. Readiness is where dependency checks belong, so a service with a degraded downstream is pulled out of rotation without being killed.

Sharp Edges

The failures here cluster around cost and registration. Sampling is the first: tracing 100% of requests is fine in development and ruinous at scale, both in exporter overhead and backend bill, so production runs a sampler -- head-based (decide at the start of the trace, cheap but blind to outcome) or tail-based (decide after the trace completes, so you can keep all the errors and slow ones, at the cost of buffering in the Collector). Cardinality is the second and is the one that quietly takes a backend down: putting user IDs, request IDs, or any unbounded value into metric tags creates a runaway number of time series -- keep those on spans and logs, where high cardinality is free, and off metrics, where it is not.

The rest are smaller but recurring. Interpolated log messages ($"Processed {id}") destroy the structured properties and leave you with strings you cannot query -- always use the template form. Synchronous or unbatched exporters add latency to the request path; route everything through the Collector with batching so shipping happens off the hot path. And the quiet failure that catches everyone once: a custom ActivitySource or Meter emits absolutely nothing unless its name is registered with AddSource or AddMeter. The code runs, the spans and metrics are created, and they vanish into the void -- there is no error, just silence -- until you add the one registration line that tells OTel to listen.