Set up a relay for CosmicAC

Route CosmicAC job-agent connections through a relay when your cluster network blocks direct connections.

Set up a relay when your cluster network blocks direct connections between CosmicAC job agents and cosmicac-wrk-server-k8s-nvidia.

CosmicAC components normally connect directly over the Holepunch peer-to-peer stack. Use a relay when either of the following occurs:

Communication failure: cosmicac-wrk-agent-inference or cosmicac-wrk-agent-instance can't reach cosmicac-wrk-server-k8s-nvidia. Job statuses stop updating, and the last communication can be minutes old. A job can also remain in Creating or Starting after CosmicAC creates it.
Access issues: users can't access GPU containers on a specific cluster.

Prerequisites

You need the following before you start:

A running CosmicAC deployment. See Installation.
A host that runs Node.js and npm and is reachable from both the deployment host and the job virtual machines on the cluster.
An open UDP port on the relay host that no firewall blocks. The relay listens on port 49737 by default.
Up to 200 MB of RAM and minimal CPU available on the relay host.

Install the relay service

On the relay host, install blind-relay-service:

npm install -g blind-relay-service

Start the relay

Start the relay with a persistent storage directory:

blind-relay --storage /var/lib/blind-relay

Use the same storage directory each time you restart the relay. The relay creates its key pair from the Corestore in this directory. If you start the relay with a different storage directory, it can generate a different key, so the key configured in CosmicAC no longer matches and jobs cannot reach the relay.

If you do not specify --storage, the relay uses ./corestore in the directory where you run the command.

To use a different UDP port, add --port <num>.

The relay displays its storage path and key:

Using corestore storage at /var/lib/blind-relay
Server listening on <relay-key>

Copy the key from the Server listening on line. You need it in the next step.

Configure the relay key

On the deployment host, add the relay key to .env so cosmicac-wrk-server-k8s-nvidia passes it to the jobs it creates:

K8S_COMMON_CONFIG__interconnect__relays='["<relay-key>"]'

The value is a JSON array. See Config overrides for information about the variable format.

Apply the configuration and restart the worker

Apply the updated configuration, then restart the worker:

task apply-wrk-server-k8s-nvidia-common-config
task restart SERVICES="cosmicac-wrk-server-k8s-nvidia"

Existing jobs do not receive the relay key

Only jobs created after the worker restarts use the relay. The worker passes the relay key to a job when it creates the job.

To recreate a stuck job, delete it:

cosmicac jobs delete <jobId>

Then create the job again.

Verify the relay

Create a new job, then list the jobs:

cosmicac jobs list

The new job should leave Creating and continue updating its status.