Logging on VM Clusters

Where Ray writes logs and how to aggregate them across nodes.

Where logs live

/tmp/ray/session_latest/logs/ on every node. Subdirectories include:

• dashboard.log, dashboard_agent.log — dashboard
• gcs_server.out — GCS
• monitor.out — autoscaler
• raylet.out — per-node raylet
• worker-*.out / worker-*.err — Python worker stdout/stderr

Per-job logs

Logs from ray.init (driver) and your tasks/actors are routed through the raylet. By default, worker stdout/stderr is forwarded to the driver and printed in your terminal.

Disable with:

ray.init(log_to_driver=False)

Or per-task:

@ray.remote
class Quiet:
    def __init__(self):
        import sys
        sys.stdout = open("/dev/null", "w")

Log aggregation

Run a log shipper on every node (fluentbit, Vector, Datadog Agent) tailing /tmp/ray/session_latest/logs/.

Sample fluentbit input:

[INPUT]
    Name              tail
    Path              /tmp/ray/session_latest/logs/*.out,/tmp/ray/session_latest/logs/*.err
    Tag               ray.<filename>
    Refresh_Interval  5

Dedup

RAY_DEDUP_LOGS=1 collapses identical log lines from many workers into a single line in the driver output. On for the driver only — workers still log everything to disk.

Rotation

RAY_ROTATION_MAX_BYTES=1073741824 RAY_ROTATION_BACKUP_COUNT=10 ray start ...

Sensitive data

Worker logs may include argument values. Don’t pass secrets as task arguments — use environment variables or runtime envs instead.

Next steps

Observability

Logs, metrics, and tracing together.

Best practices

Logging in large clusters.