Defining a Resource Interface

This section describes how to extend NVIDIA Run:ai to support ML frameworks, tools, or Kubernetes resources by defining and registering them through the Workload Types API and a corresponding Resource Interface (RI) definition. See the Quick start templates for ready-to-use examples of Resource Interface configurations.

An RI is a declarative workload contract that describes how NVIDIA Run:ai should interpret a custom resource. It defines how the platform identifies the workload’s structure, locates its pods, tracks its state, and applies scheduling, monitoring, and optimization logic.

Without an RI, NVIDIA Run:ai cannot reliably schedule, monitor, or optimize a custom resource, because it lacks the information required to traverse the resource’s schema and interpret its semantics.

Workload Types API

Use this endpoint POST /api/v1/workload-types to register a new workload type. Each request must include the workload’s metadata and one or more Resource Interface (RI) definitions, one per supported CRD version (for example, v1, v1beta1). See Workload Types API for more details.

Field	Description	Example
categoryId	The unique identifier of the workload category. See List workload categories API.	046b6c7f-...
priorityId	The unique identifier of the workload priority. See Get workload priorities API.	046b6c7f-...
name	The unique name of the workload type. This value must match the Kubernetes Kind that represents the workload type.	Deployment
group	The Kubernetes API group associated with the workload resource.	apps
resourceInterfaces	One or more RI definitions (one per CRD version, e.g. v1, v1beta1). Lists the versions of the custom resource definition (CRD) supported for this workload type, such as v1, v1beta1, or v1alpha1. This enables the platform to correctly parse, interpret, and manage manifests for this workload type according to the specific structure and schema associated with each listed version.	See Resource interface structure

What is a Resource Interface?

In Kubernetes, a workload is not a standalone execution unit (pod). It comprises various components, including an ingress entry point, a collection of pods, and storage.

Defining a Resource Interface (RI) enables NVIDIA Run:ai to interpret, schedule, monitor, and optimize custom Kubernetes workloads by describing how the CRD’s structure maps to workload behavior. An RI tells the platform:

• What resources represent the workload (CRD group, version, kind)

• How to find pods and their spec fields

• How to map status conditions to unified states

• How to interpret component hierarchy and scheduling rules

Note

The RI defines the contract between a CRD and the NVIDIA Run:ai platform. It does not create the CRD; it explains how NVIDIA Run: should treat it.

Resource Interface Structure

A Resource Interface is a structured YAML or JSON object that describes how a workload is represented in Kubernetes. The following sections describe each required component of the RI definition.

Minimum Requirements

Contract Summary: Required vs Optional

Section	Required	Purpose
rootComponent	Yes	Defines the primary resource (CRD) that represents the workload
statusDefinition	Yes (under rootComponent)	Maps the CRD’s status fields to NVIDIA Run:ai canonical workload states (for example, running, failed)
specDefinition	Yes (where pods exist)	Specifies where NVIDIA Run:ai can locate the pod specifications associated with the workload
childComponents	Optional (only if workload has owned resources)	Describes subordinate Kubernetes resources that are created and managed by the root CRD
additionalChildKinds	Optional	Declares additional resource kinds (GVKs) that the workload creates or controls.
optimizationInstructions	Optional	Instructs scheduler behaviors (e.g., gang scheduling).
scaleDefinition	Optional	Defines how the workload or its components can be scaled

Each required field has explicit minimal semantics and a clear reason for being required. Optional fields unlock additional platform capabilities.

A minimal RI must define a rootComponent with a name, full GVK (group, version, kind) and a statusDefinition section that describes its runtime state.

rootComponent:
  name: "minimal"
  kind:
    group: "minimal.org"
    version: "v1"
    kind: "minimalRI"
  statusDefinition:
    statusMappings:
      running:
        byConditions:
        - type: "Running"
          status: "True"

Root Component

Every Resource Interface begins with a rootComponent definition:

• Must specify the full Kubernetes GVK (group, version, kind)

• Must include a statusDefinition to describe the workload’s runtime state

rootComponent:
  name: "pytorchjob"
  kind:
    group: "kubeflow.org"
    version: "v1"
    kind: "PyTorchJob"
  statusDefinition:
    statusMappings:
      running:
        byConditions:
        - type: "Running"
          status: "True"

Child Components

childComponents represent resources owned by the root component.

• Must include ownerRef (points to parent component)

• Usually includes a specDefinition

• All paths must be absolute, starting from the CRD root

childComponents:
- name: "worker"
  ownerName: "pytorchjob"
  kind:
    group: "apps"
    version: "v1"
    kind: "StatefulSet"
  specDefinition:
    podTemplateSpecPath: ".spec.pytorchReplicaSpecs.Worker.template"

Paths

All paths within an RI are written in jq syntax. The jq query language provides both path navigation and various query capabilities, and is widely used in the Kubernetes ecosystem. When defining paths:

• Use the correct jq type for each property (path/query)

• Provide default values where necessary

Spec Definitions

specDefinition determines how NVIDIA Run:ai locates pod specifications within the CRD. Several mutually exclusive definition types are supported, such as podTemplateSpecPath, podSpecPath, metaDataPath and fragmentedPodDefinition.

Pattern	Use Case	Example
podTemplateSpecPath	CRD embeds a full pod template	.spec.pytorchReplicaSpecs.Master.template
podSpecPath and metadataPath	CRD directly embeds a podSpec and/or an objectMeta	.spec.jobTemplate.spec .spec.jobTemplate.metadata
fragmentedPodSpecDefinition	Pod fields scattered across CRD	specDefinition: fragmentedPodDefinition: labelsPath: ".spec.labels" annotationsPath: ".spec.annotations" resourcesPath: ".spec.resources" schedulerNamePath: ".spec.schedulerName"

Component Instances

A component’s spec definition might point to multiple specs (in map/array format). In those cases, it’s crucial to be able to distinguish between each instance of that component.

If a component produces multiple instances (arrays or maps), define instanceIdPath to identify each instance uniquely.

# Array of specs
instanceIdPath: ".spec.jobs[].name"

# Map of specs (use map keys as instance IDs)
instanceIdPath: ".spec.jobs | to_entries[] | .key"

Pod Selectors

Use podSelector to associate pods with components or instances. This is required when multiple components define pods or when a component manages multiple instances. All selectors of each kind (component, instance) must be mutually exclusive within their scope. Paths in pod selectors refer to paths in the pod JSON/YAML.

• componentTypeSelector - A key and value selector that associates pod to the current component. If the value is not provided, only key existence is checked:

  podSelector:
    componentTypeSelector:
      keyPath: '.metadata.labels["training.kubeflow.org/replica-type"]'
      value: "master"

• componentInstanceSelector - A path on the pod that holds its matching instance id:

  podSelector:
    componentInstanceSelector:
      idPath: '.metadata.labels["jobset.sigs.k8s.io/replicatedjob-name"]'

Status Definitions

The statusDefinition maps CRD conditions or phases to the generic Resource Interface (RI) statuses. A status definition is required for the rootComponent. For each generic status, you can define how it is evaluated using one or more of the following mechanisms:

Conditions and Phases

• Statuses can be derived from CRD conditions, phases, or both.

• If both conditions and phases are defined, both are evaluated when determining the status.

• When using conditions or phases, you must first define a conditionsDefinition or phaseDefinition.

• Multiple, independent definitions can be provided for the same generic status.

• When defining a status using byConditions, all specified conditions must be met (AND logic).

statusDefinition:
        conditionsDefinition:
          path: ".status.conditions"
          typeFieldName: "type"
          statusFieldName: "status"
        statusMappings:
          initializing:
          - byConditions:
            - type: "Created"
              status: "True"
            - type: "Running"
              status: "False"
          running:
          - byConditions:
            - type: "Running"
              status: "True"
            - type: "Succeeded"
              status: "False"
            - type: "Failed"
              status: "False"
          completed:
          - byConditions:
            - type: "Succeeded"
              status: "True"
          failed:
          - byConditions:
            - type: "Failed"
              status: "True"

Matched Expressions

Matched expressions consist of:

• An expression

• An expected result

If the evaluated expression output matches the expected result, the condition is considered met and the corresponding status is applied.

Examples:

Running: []boltv1alpha1.StatusMatcher{
{
	ByExpression: &boltv1alpha1.ExpressionMatcher{
		Expression:     ".status.phase",
		ExpectedResult: "running",
	},
},

Running: []boltv1alpha1.StatusMatcher{
{
	ByExpression: &boltv1alpha1.ExpressionMatcher{
		Expression:     ".status.readyReplicas == .status.readyReplicas",
		ExpectedResult: "true",
	},
},

Additional Child Kinds

List any additional GVKs created or managed by the CRD but not defined explicitly under childComponents. This is essential for permission management so that your CRD can be managed correctly.

  additionalChildKinds:
   - group: apps
     version: v1
     kind: Deployment
   - group: leaderworkerset.x-k8s.io
     version: v1
     kind: LeaderWorkerSet

Optimization Instructions

Optimization instructions define how NVIDIA Run:ai schedules or groups pods. All paths in the configuration refer to pod YAML or JSON fields.

Supported Instruction Types

The supported instruction type is gangScheduling, which instructs the Scheduler on how to group related pods:

• Each pod-group definition contains a list of the included members

• Each defined member can provide a list of distinct keys to group pods by and a list of filters to determine which pods should be included

optimizationInstructions:
  gangScheduling:
    podGroups:
    - name: "job"
        members:
        - componentName: "master"
          groupByKeyPaths: 
          - '.metadata.labels["training.kubeflow.org/job-name"]'
        - componentName: "worker"
          groupByKeyPaths:
          - '.metadata.labels["training.kubeflow.org/job-name"]'

Grouping examples

The following definitions are equivalent when the master and worker share the same job-name label:

optimizationInstructions:
  gangScheduling:
    podGroups:
    - name: "job"
        members:
        - componentName: "master"
          groupByKeyPaths: 
          - '.metadata.labels["training.kubeflow.org/job-name"]'
        - componentName: "worker"
          groupByKeyPaths:
          - '.metadata.labels["training.kubeflow.org/job-name"]'

optimizationInstructions:
  gangScheduling:
    podGroups:
    - name: "job"
        members:
        - componentName: "job"
          groupByKeyPaths: 
          - '.metadata.labels["training.kubeflow.org/job-name"]'

Using default values

When multiple groups are possible, use a default value to cover cases where a single group is used (the used pattern: -{name}-{index}):

optimizationInstructions:
   gangScheduling:
     podGroups:
     - name: "group"
       members:
       - componentName: "group"
         groupByKeyPaths:
         - '.metadata.labels["leaderworkerset.sigs.k8s.io/name"]'
         - '.metadata.labels["leaderworkerset.sigs.k8s.io/group-index"] // "0"'

Filters

When different components are not named components (array/map of specs in the same component) you can use filters to form different pod groups. In this example, for a CRD that defines multiple jobs but all under the job component, we use jq query as filter to identify pods that use NVIDIA GPUs, and queries with hard-coded values as grouping keys.

optimizationInstructions:
  gangScheduling:
    podGroups:
    - name: "gpu-jobs"
        members:
        - componentName: "job"
          filters:
          - 'any(.spec.jobs[].spec.containers[]; (.resources.limits["nvidia.com/gpu"] // 0) > 0)'
          groupByKeyPaths: 
          - 'gpu'
    - name: "no-gpu-jobs"
        members:
        - componentName: "job"
          filters:
          - 'any(.spec.jobs[].spec.containers[]; (.resources.limits["nvidia.com/gpu"] // 0) == 0)'
          groupByKeyPaths: 
          - 'no-gpu'

Best Practices

Before submitting a new Resource Interface, confirm:

• All kinds use full GVK (group, version, kind)

• The rootComponent includes a valid statusDefinition

• Absolute paths only in specDefinitions

• No duplicate child kinds; don’t list explicitly defined components in additionalChildKinds

• Mutually exclusive podSelectors in multi-component workloads

• Null-safe jq expressions (e.g. // 0 defaults)

• Target actual components in optimizationInstructions, not just the root CRD

• Status conditions match real framework APIs

• All childComponents has ownerRef directed to existing components and there are no ownership cycles

Example: KServe Inference Service

{
  "spec": {
    "structureDefinition": {
      "rootComponent": {
        "name": "inferenceservice",
        "kind": {
          "group": "serving.kserve.io",
          "version": "v1beta1",
          "kind": "InferenceService"
        },
        "specDefinition": {
          "fragmentedPodSpecDefinition": {
            "resourcesPath": ".spec.domain.resources",
            "priorityClassNamePath": ".spec.priorityClassName",
            "nodeAffinityPath": ".spec.affinity.nodeAffinity"
          }
        },
        "statusDefinition": {
          "conditionsDefinition": {
            "path": ".status.conditions",
            "typeFieldName": "type",
            "statusFieldName": "status"
          },
          "statusMappings": {
            "running": [
              {
                "byConditions": [
                  {
                    "type": "PredictorReady",
                    "status": "True"
                  },
                  {
                    "type": "RoutesReady",
                    "status": "True"
                  },
                  {
                    "type": "LatestDeploymentReady",
                    "status": "True"
                  }
                ]
              }
            ],
            "failed": [
              {
                "byConditions": [
                  {
                    "type": "PredictorReady",
                    "status": "False"
                  },
                  {
                    "type": "PredictorConfigurationReady",
                    "status": "False"
                  },
                  {
                    "type": "RoutesReady",
                    "status": "False"
                  }
                ]
              }
            ]
          }
        }
      },
      "childComponents": [
        {
          "name": "predictor",
          "kind": {
            "group": "apps",
            "version": "v1",
            "kind": "Deployment"
          },
          "ownerRef": "inferenceservice",
          "specDefinition": {
            "podSpecPath": ".spec.predictor",
            "metadataPath": ".spec.predictor",
            "fragmentedPodSpecDefinition": {
              "containerPath": ".spec.predictor.model"
            }
          },
          "scaleDefinition": {
            "minReplicasPath": ".spec.predictor.minReplicas",
            "maxReplicasPath": ".spec.predictor.maxReplicas"
          },
          "podSelector": {
		 "componentTypeSelector": {
              "keyPath": ".metadata.labels[\"component\"]",
              "value": "predictor"
            }
          }
        },
        {
          "name": "transformer",
          "kind": {
            "group": "apps",
            "version": "v1",
            "kind": "Deployment"
          },
          "ownerRef": "inferenceservice",
          "specDefinition": {
            "podSpecPath": ".spec.transformer",
            "metadataPath": ".spec.transformer"
          },
          "scaleDefinition": {
            "minReplicasPath": ".spec.transformer.minReplicas",
            "maxReplicasPath": ".spec.transformer.maxReplicas"
          },
          "podSelector": {
		 "componentTypeSelector": {
              "keyPath": ".metadata.labels[\"component\"]",
              "value": "transformer"
            }
          }
        }
      ]
    },
    "optimizationInstructions": {
      "gangScheduling": {
        "podGroups": [
          {
            "name": "service",
            "members": [
              {
                "componentName": "predictor",
                "groupByKeyPaths": [
                  ".metadata.labels[\"serving.kserve.io/inferenceservice\"]"
                ]
              },
              {
                "componentName": "transformer",
                "groupByKeyPaths": [
                  ".metadata.labels[\"serving.kserve.io/inferenceservice\"]"
                ]
              }
            ]
          }
        ]
      }
    }
  }
}