Logging

Configure, route, and aggregate Ray logs.

Each Ray process writes logs to /tmp/ray/session_latest/logs/. Workers’ stdout and stderr are forwarded to the driver process by default.

Locations

• dashboard.log, dashboard_agent.log
• gcs_server.out / gcs_server.err
• monitor.out (autoscaler)
• raylet.out / raylet.err
• worker-*.out / worker-*.err
• serve/ (Ray Serve)

Driver forwarding

ray.init(log_to_driver=True)   # default
ray.init(log_to_driver=False)  # silent driver

Log dedup

Set RAY_DEDUP_LOGS=1 (default in newer versions) to collapse identical lines from many workers into a single line at the driver.

Per-task and per-actor loggers

import logging
import ray

logger = logging.getLogger("my_app")

@ray.remote
def task():
    logger.info("hello")

Logs land in the worker’s .out file and are forwarded to the driver.

Aggregating across nodes

Run a log shipper (fluentbit, Vector, Datadog Agent) on each node tailing the Ray log directory. Ship to your platform’s log backend (CloudWatch, GCP Logging, Loki, ELK, Datadog).

Rotation

Configure with environment variables:

RAY_ROTATION_MAX_BYTES=1073741824   # 1 GiB per file
RAY_ROTATION_BACKUP_COUNT=10        # keep 10 rotations

Sensitive data

Argument values are logged when tasks fail. Don’t pass secrets as task arguments — use environment variables, runtime envs, or external secret stores.

Next steps

Profiling

py-spy and memray.

State API

Inspect the cluster programmatically.