1 of 2

Quick Starts

Run Your First Custom Inference Workload

This quick start provides a step-by-step walkthrough for running and querying a custom inference workload.

An inference workload provides the setup and configuration needed to deploy your trained model for real-time or batch predictions. It includes specifications for the container image, data sets, network settings, and resource requests required to serve your models.

Prerequisites

Before you start, make sure:

You have created a project or have one created for you.
The project has an assigned quota of at least 1 GPU.
is properly installed by your administrator.

Browse to the provided NVIDIA Run:ai user interface and log in with your credentials.

To use the API, you will need to obtain a token as shown in .

Go to Workload manager → Workloads.
Click +NEW WORKLOAD and select Inference
Select under which cluster to create the workload

In this step, you'll test the deployed model by sending a request to the inference server. To do this, you'll launch a general-purpose workload, typically a Training or Workspace workload, to run the Triton demo client. You'll first retrieve the workload address, which serves as the model’s inference serving endpoint. Then, use the client to send a sample request and verify that the model is responding correctly.

Go to the Workload manager → Workloads.
Click COLUMNS and select Connections.
Select the link under the Connections column for the inference workload created in

Select the inference workload you created in and go to the tab to see various GPU and inference metrics graphs rise.
Manage and monitor your newly created workload using the table.

Run Your First Custom Inference Workload

This quick start provides a step-by-step walkthrough for running and querying a custom inference workload.

Prerequisites

Before you start, make sure:

You have created a project or have one created for you.
The project has an assigned quota of at least 1 GPU.
is properly installed by your administrator.

Browse to the provided NVIDIA Run:ai user interface and log in with your credentials.

To use the API, you will need to obtain a token as shown in .

Go to Workload manager → Workloads.
Click +NEW WORKLOAD and select Inference
Select under which cluster to create the workload

Go to the Workload manager → Workloads.
Click COLUMNS and select Connections.
Select the link under the Connections column for the inference workload created in

Select the inference workload you created in and go to the tab to see various GPU and inference metrics graphs rise.
Manage and monitor your newly created workload using the table.

Create an environment for your workload

Click +NEW ENVIRONMENT
Enter a name for the environment. The name must be unique.
Enter the Image URL - runai.jfrog.io/demo/example-triton-server
Set the inference serving endpoint to HTTP and the container port to 8000
Click CREATE ENVIRONMENT

The newly created environment will be selected automatically

Select the ‘half-gpu’ compute resource for your workload (GPU devices: 1)

If ‘half-gpu’ is not displayed in the gallery, follow the below steps:
- Click +NEW COMPUTE RESOURCE
- Enter half-gpu as the name for the compute resource. The name must be unique.
- Set GPU devices per pod - 1
- Set GPU memory per device
  - Select % (of device) - Fraction of a GPU device’s memory
  - Set the memory Request - 50 (the workload will allocate 50% of the GPU memory)
- Optional: set the CPU compute per pod - 0.1 cores (default)
- Optional: set the CPU memory per pod - 100 MB (default)
- Click CREATE COMPUTE RESOURCE

The newly created compute resource will be selected automatically

Under Replica autoscaling:

Set a minimum of 1 replica and maximum of 2 replicas
Set the conditions for creating a new replica to Concurrency (Requests) and the value to 3
Set when the replicas should be automatically scaled down to zero to After 5 minutes of inactivity. When automatic scaling to zero is enabled, the minimum number of replicas set in the previous step, automatically changes to 0.

Copy the following command to your terminal. Make sure to update the below parameters. For more details, see Inferences API:

curl -L 'https://<COMPANY-URL>/api/v1/workloads/inferences' \ 
-H 'Content-Type: application/json' \
-H 'Authorization: Bearer <TOKEN>' \
-d '{ 
    "name": "workload-name", 
    "useGivenNameAsPrefix": true,
    "projectId": "<PROJECT-ID>", 
    "clusterId": "<CLUSTER-UUID>", 
    "spec": {
        "image": "runai.jfrog.io/demo/example-triton-server",
        "servingPort": {
            "protocol": "http",
            "container": 8000
        },
        "autoscaling": {
            "minReplicas": 0,
            "maxReplicas": 2,
            "metric": "concurrency",
            "metricThreshold": 3,
            "scaleToZeroRetentionSeconds": 300
        },
        "compute": {
            "cpuCoreRequest": 0.1,
            "gpuRequestType": "portion",
            "cpuMemoryRequest": "100M",
            "gpuDevicesRequest": 1,
            "gpuPortionRequest": 0.5
        }
    }
}'

<COMPANY-URL> - The link to the NVIDIA Run:ai user interface
<TOKEN> - The API access token obtained in Step 1
<PROJECT-ID> - The ID of the Project the workload is running on. You can get the Project ID via the API.
<CLUSTER-UUID> - The unique identifier of the Cluster. You can get the Cluster UUID via the API.

Create an environment for your workload

Click +NEW ENVIRONMENT
Enter a name for the environment. The name must be unique.
Enter the Image URL - runai.jfrog.io/demo/example-triton-client
Set the runtime settings for the environment. Click +COMMAND & ARGUMENTS and add the following:
- Enter the command: perf_analyzer
- Enter the arguments: -m inception_graphdef -p 3600000 -u <INFERENCE-ENDPOINT>. Make sure to replace the inference endpoint with the Address you retrieved above.
Click CREATE ENVIRONMENT

The newly created environment will be selected automatically

Select the ‘cpu-only’ compute resource for your workspace

If ‘cpu-only’ is not displayed in the gallery, follow the below steps:
- Click +NEW COMPUTE RESOURCE
- Enter cpu-only as the name for the compute resource. The name must be unique.
- Set GPU devices per pod - 0
- Set CPU compute per pod - 0.1 cores
- Set the CPU memory per pod - 100 MB (default)
- Click CREATE COMPUTE RESOURCE

The newly created compute resource will be selected automatically

Copy the following command to your terminal. Make sure to update the below parameters according. For more details, see Trainings API:

curl -L 'https://<COMPANY-URL>/api/v1/workloads/trainings' \ 
-H 'Content-Type: application/json' \
-H 'Authorization: Bearer <TOKEN>' \ 
-d '{ 
    "name": "workload-name", 
    "projectId": "<PROJECT-ID>", 
    "clusterId": "<CLUSTER-UUID>",
    "spec": {  
        "image": "runai.jfrog.io/demo/example-triton-client",
        "command": "perf_analyzer",
        "args": "-m inception_graphdef -p 3600000 -u <INFERENCE-ENDPOINT>",
        "compute": {
            "cpuCoreRequest":0.1,
            "cpuMemoryRequest": "100M",
        }
    }
}

<COMPANY-URL> - The link to the NVIDIA Run:ai user interface
<TOKEN> - The API access token obtained in Step 1
<PROJECT-ID> - The ID of the Project the workload is running on. You can get the Project ID via the API.
<CLUSTER-UUID> - The unique identifier of the Cluster. You can get the Cluster UUID via the API.
<INFERENCE-ENDPOINT> - You can get the inference endpoint from the urls parameter via the API.

Quick Starts

Run Your First Custom Inference Workload

Prerequisites

Quick Starts

Run Your First Custom Inference Workload

Prerequisites

Step 1: Logging In

Step 2: Submitting an Inference Workload

Step 3: Querying the Inference Server

Next Steps