Monitoring Ray Serve

Metrics, logs, and dashboards for Ray Serve.

Metrics

Serve exports Prometheus-compatible metrics on port 8080 (configurable). Key metrics:

Metric	Description
ray_serve_num_http_requests_total	Counter of HTTP requests received.
ray_serve_request_latency_ms	Histogram of request latency.
ray_serve_num_ongoing_requests	In-flight requests per replica.
ray_serve_num_replicas	Current replica count per deployment.
ray_serve_replica_starts_total	Counter of replica restarts.
ray_serve_deployment_queued_requests	Requests waiting for a replica.

Dashboard

The Ray dashboard’s Serve tab shows:

• Live replica counts per deployment
• Per-replica QPS and latency
• Recent deployment history (rolling updates, restarts)
• Per-deployment logs

Logging

Each replica logs to /tmp/ray/session_*/logs/serve/. Configure log level per deployment:

@serve.deployment(logging_config={"log_level": "INFO", "encoding": "JSON"})
class Service:
    ...

For structured logging, set encoding="JSON".

Tracing

Use OpenTelemetry middleware on your FastAPI app:

from opentelemetry.instrumentation.fastapi import FastAPIInstrumentor

FastAPIInstrumentor.instrument_app(api)

Spans propagate across DeploymentHandle calls when the OpenTelemetry context is forwarded.

Custom metrics

Use prometheus_client from inside a deployment:

from prometheus_client import Counter

predictions_total = Counter("predictions_total", "Predictions made")

@serve.deployment
class Predictor:
    def __call__(self, request):
        predictions_total.inc()
        return self.model(request)

Next steps

Production guide

Where these metrics fit in production.

Observability

Cluster-wide observability.