Distributed Tracing

Trace requests through Ray Serve, Ray Data, and your own application code with OpenTelemetry.

Ray applications produce distributed traces at the application layer (FastAPI, vLLM, your code). Ray-internal tracing of tasks and actors is gated behind a feature flag.

Instrument FastAPI

from opentelemetry import trace
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
from opentelemetry.instrumentation.fastapi import FastAPIInstrumentor
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor

provider = TracerProvider(resource=Resource(attributes={"service.name": "ray-app"}))
provider.add_span_processor(BatchSpanProcessor(OTLPSpanExporter(endpoint="https://otel:4318/v1/traces")))
trace.set_tracer_provider(provider)

# In your Serve app:
FastAPIInstrumentor.instrument_app(api)

Spans propagate through FastAPI handlers and across await DeploymentHandle.remote(...) calls when the OpenTelemetry context is set.

Cross-service propagation

Use the traceparent HTTP header (W3C trace context). FastAPI’s instrumentation reads and writes it automatically; for raw clients, attach the header manually.

Custom spans

tracer = trace.get_tracer(__name__)

@serve.deployment
class Service:
    async def __call__(self, request):
        with tracer.start_as_current_span("preprocess"):
            x = preprocess(request)
        with tracer.start_as_current_span("predict"):
            y = self.model(x)
        return y

Where to send traces

Any OpenTelemetry-compatible backend: Tempo, Jaeger, Honeycomb, Datadog, Lightstep, etc. Configure with the OTLPSpanExporter endpoint.

Ray-internal spans

For low-level Ray spans, set RAY_BACKEND_LOG_LEVEL=debug and configure an OTLP exporter. This is rarely needed unless you’re debugging Ray itself.

Next steps

Metrics

Pair traces with metrics.

State API

Programmatic cluster inspection.