Endpoints

Interacting with RunPod's endpoints is a core feature of the SDK, enabling the execution of tasks and the retrieval of results. This section covers the synchronous and asynchronous execution methods, along with checking the status of operations.

Prerequisites

Before using the RunPod JavaScript, ensure that you have:

Installed the RunPod JavaScript SDK.
Configured your API key.

Set your Endpoint Id

Set your RunPod API key and your Endpoint Id as environment variables.

const { RUNPOD_API_KEY, ENDPOINT_ID } = process.env;
import runpodSdk from "runpod-sdk";

const runpod = runpodSdk(RUNPOD_API_KEY);
const endpoint = runpod.endpoint(ENDPOINT_ID);

This allows all calls to pass through your Endpoint Id with a valid API key.

In most situations, you'll set a variable name endpoint on the Endpoint class. This allows you to use the following methods or instances variables from the Endpoint class:

health
purge_queue
runSync
run

Run the Endpoint

Run the Endpoint with the either the asynchronous run or synchronous runSync method.

Choosing between asynchronous and synchronous execution hinges on your task's needs and application design.

Run synchronously

To execute an endpoint synchronously and wait for the result, use the runSync method on your endpoint. This method blocks the execution until the endpoint run is complete or until it times out.

JavaScript
Output

const { RUNPOD_API_KEY, ENDPOINT_ID } = process.env;
import runpodSdk from "runpod-sdk";

const runpod = runpodSdk(RUNPOD_API_KEY);
const endpoint = runpod.endpoint(ENDPOINT_ID);
const result = await endpoint.runSync({
  "input": {
    "prompt": "Hello, World!",
  },
});

console.log(result);

{
  delayTime: 18,
  executionTime: 36595,
  id: 'sync-d050a3f6-791a-4aff-857a-66c759db4a06-u1',
  output: [ { choices: [Array], usage: [Object] } ],
  status: 'COMPLETED',
  started: true,
  completed: true,
  succeeded: true
}

Run asynchronously

Asynchronous execution allows for non-blocking operations, enabling your code to perform other tasks while waiting for an operation to complete.

For non-blocking operations, use the run method on the endpoint. This method allows you to start an endpoint run and then check its status or wait for its completion at a later time.

JavaScript
Output

const { RUNPOD_API_KEY, ENDPOINT_ID } = process.env;
import runpodSdk from "runpod-sdk";

const runpod = runpodSdk(RUNPOD_API_KEY);
const endpoint = runpod.endpoint(ENDPOINT_ID);
const result = await endpoint.run({
  "input": {
    "prompt": "Hello, World!",
  },
});

console.log(result);

{
  "id": "d4e960f6-073f-4219-af24-cbae6b532c31-u1",
  "status": "IN_QUEUE"
}

Get results from an asynchronous run

The following example shows how to get the results of an asynchronous run.

JavaScript
Output

const { RUNPOD_API_KEY, ENDPOINT_ID } = process.env;
import runpodSdk from "runpod-sdk";

const sleep = (ms) => new Promise((resolve) => setTimeout(resolve, ms));

async function main() {
  const runpod = runpodSdk(RUNPOD_API_KEY);
  const endpoint = runpod.endpoint(ENDPOINT_ID);
  const result = await endpoint.run({
    input: {
      prompt: "Hello, World!",
    },
  });

  console.log(result);
  console.log("run response");
  console.log(result);

  const { id } = result; // Extracting the operation ID from the initial run response

  // Check the status in a loop, similar to the working example
  for (let i = 0; i < 20; i++) {
    // Increase or decrease the loop count as necessary
    const statusResult = await endpoint.status(id);
    console.log("status response");
    console.log(statusResult);

    if (
      statusResult.status === "COMPLETED"
      || statusResult.status === "FAILED"
    ) {
      // Once completed or failed, log the final status and break the loop
      if (statusResult.status === "COMPLETED") {
        console.log("Operation completed successfully.");
        console.log(statusResult.output);
      } else {
        console.log("Operation failed.");
        console.log(statusResult);
      }
      break;
    }

    // Wait for a bit before checking the status again
    await sleep(5000);
  }
}

main();

run response
{ id: 'c671a352-78e6-4eba-b2c8-2ea537c00897-u1', status: 'IN_QUEUE' }
status response
{
  delayTime: 19,
  id: 'c671a352-78e6-4eba-b2c8-2ea537c00897-u1',
  status: 'IN_PROGRESS',
  started: true,
  completed: false,
  succeeded: false
}
status response
{
  delayTime: 19,
  executionTime: 539,
  id: 'c671a352-78e6-4eba-b2c8-2ea537c00897-u1',
  output: [ { choices: [Array], usage: [Object] } ],
  status: 'COMPLETED',
  started: true,
  completed: true,
  succeeded: true
}
Operation completed successfully.
[ { choices: [ [Object] ], usage: { input: 5, output: 16 } } ]

Poll the status of an asynchronous run

Uses await endpoint.status(id) to check the status of the operation repeatedly until it either completes or fails. After each check, the function waits for 5 seconds (or any other suitable duration you choose) before checking the status again, using the sleep function. This approach ensures your application remains responsive and doesn't overwhelm the Runpod endpoint with status requests.

JavaScript
Output

const { RUNPOD_API_KEY, ENDPOINT_ID } = process.env;
import runpodSdk from "runpod-sdk";

// Function to pause execution for a specified time
const sleep = (ms) => new Promise((resolve) => setTimeout(resolve, ms));

async function main() {
  try {
    const runpod = runpodSdk(RUNPOD_API_KEY);
    const endpoint = runpod.endpoint(ENDPOINT_ID);
    const result = await endpoint.run({
      input: {
        prompt: "Hello, World!",
      },
    });

    const { id } = result;
    if (!id) {
      console.error("No ID returned from endpoint.run");
      return;
    }

    // Poll the status of the operation until it completes or fails
    let isComplete = false;
    while (!isComplete) {
      const status = await endpoint.status(id);
      console.log(`Current status: ${status.status}`);

      if (status.status === "COMPLETED" || status.status === "FAILED") {
        isComplete = true; // Exit the loop
        console.log(`Operation ${status.status.toLowerCase()}.`);

        if (status.status === "COMPLETED") {
          console.log("Output:", status.output);
        } else {
          console.error("Error details:", status.error);
        }
      } else {
        await sleep(5000); // Adjust the delay as needed
      }
    }
  } catch (error) {
    console.error("An error occurred:", error);
  }
}

main();

Current status: IN_QUEUE
Current status: IN_PROGRESS
Current status: COMPLETED
Operation completed.
Hello, World!

Stream

Stream allows you to stream the output of an Endpoint run. To enable streaming, your handler must support the "return_aggregate_stream": True option on the start method of your Handler. Once enabled, use the stream method to receive data as it becomes available.

JavaScript
Output
Handler

const { RUNPOD_API_KEY, ENDPOINT_ID } = process.env;
import runpodSdk from "runpod-sdk";

async function main() {
  const runpod = runpodSdk(RUNPOD_API_KEY);
  const endpoint = runpod.endpoint(ENDPOINT_ID);
  const result = await endpoint.run({
    input: {
      prompt: "Hello, World!",
    },
  });

  console.log(result);

  const { id } = result;
  for await (const result of endpoint.stream(id)) {
    console.log(`${JSON.stringify(result, null, 2)}`);
  }
  console.log("done streaming");
}

main();

{ id: 'cb68890e-436f-4234-955d-001db6afe972-u1', status: 'IN_QUEUE' }
{
  "output": "H"
}
{
  "output": "e"
}
{
  "output": "l"
}
{
  "output": "l"
}
{
  "output": "o"
}
{
  "output": ","
}
{
  "output": " "
}
{
  "output": "W"
}
{
  "output": "o"
}
{
  "output": "r"
}
{
  "output": "l"
}
{
  "output": "d"
}
{
  "output": "!"
}
done streaming

You must define your handler to support the "return_aggregate_stream": True option on the start method.

from time import sleep
import runpod


def handler(job):
    job_input = job["input"]["prompt"]

    for i in job_input:
        sleep(1)  # sleep for 1 second for effect
        yield i


runpod.serverless.start(
    {
        "handler": handler,
        "return_aggregate_stream": True,  # Ensures aggregated results are streamed back
    }
)

note

The maximum size for a payload that can be sent using yield to stream results is 1 MB.

Health check

Monitor the health of an endpoint by checking its status, including jobs completed, failed, in progress, in queue, and retried, as well as the status of workers.

JavaScript
Output

const { RUNPOD_API_KEY, ENDPOINT_ID } = process.env;
import runpodSdk from "runpod-sdk";

const runpod = runpodSdk(RUNPOD_API_KEY);
const endpoint = runpod.endpoint(ENDPOINT_ID);

const health = await endpoint.health();
console.log(health);

{
  "jobs": {
    "completed": 72,
    "failed": 1,
    "inProgress": 6,
    "inQueue": 0,
    "retried": 1
  },
  "workers": {
    "idle": 4,
    "initializing": 0,
    "ready": 4,
    "running": 1,
    "throttled": 0
  }
}

Status

Use the status method and specify the id of the run to get the status of a run.

JavaScript
Output

const { RUNPOD_API_KEY, ENDPOINT_ID } = process.env;
import runpodSdk from "runpod-sdk";

async function main() {
  try {
    const runpod = runpodSdk(RUNPOD_API_KEY);
    const endpoint = runpod.endpoint(ENDPOINT_ID);
    const result = await endpoint.run({
      input: {
        prompt: "Hello, World!",
      },
    });

    const { id } = result;
    if (!id) {
      console.error("No ID returned from endpoint.run");
      return;
    }

    const status = await endpoint.status(id);
    console.log(status);
  } catch (error) {
    console.error("An error occurred:", error);
  }
}

main();

{
  "delayTime": 18,
  "id": "792b1497-b2c8-4c95-90bf-4e2a6a2a37ff-u1",
  "status": "IN_PROGRESS",
  "started": true,
  "completed": false,
  "succeeded": false
}

Cancel

You can cancel a Job request by using the cancel() function on the run request. You might want to cancel a Job because it's stuck with a status of IN_QUEUE or IN_PROGRESS, or because you no longer need the result.

JavaScript
Output

const { RUNPOD_API_KEY, ENDPOINT_ID } = process.env;
import runpodSdk from "runpod-sdk";

async function main() {
  try {
    const runpod = runpodSdk(RUNPOD_API_KEY);
    const endpoint = runpod.endpoint(ENDPOINT_ID);
    const result = await endpoint.run({
      input: {
        prompt: "Hello, World!",
      },
    });

    const { id } = result;
    if (!id) {
      console.error("No ID returned from endpoint.run");
      return;
    }

    const cancel = await endpoint.cancel(id);
    console.log(cancel);
  } catch (error) {
    console.error("An error occurred:", error);
  }
}

main();

{
  "id": "5fb6a8db-a8fa-41a1-ad81-f5fad9755f9e-u1",
  "status": "CANCELLED"
}

Timeout

To set a timeout on a run, pass a timeout value to the run method. Time is measured in milliseconds.

JavaScript
Output

const { RUNPOD_API_KEY, ENDPOINT_ID } = process.env;
import runpodSdk from "runpod-sdk";

const runpod = runpodSdk(RUNPOD_API_KEY);
const endpoint = runpod.endpoint(ENDPOINT_ID);
const result = await endpoint.run({
  "input": {
    "prompt": "Hello, World!",
  },
}, 5000);

console.log(result);

{
  "id": "43309f93-0422-4eac-92cf-e385dee36e99-u1",
  "status": "IN_QUEUE"
}

Execution policy

You can set the maximum time to wait for a response from the endpoint in the policy parameter.

JavaScript
Output

const { RUNPOD_API_KEY, ENDPOINT_ID } = process.env;
import runpodSdk from "runpod-sdk";

const runpod = runpodSdk(RUNPOD_API_KEY);
const endpoint = runpod.endpoint(ENDPOINT_ID);
const result = await endpoint.run({
  "input": {
    "prompt": "Hello, World!",
  },
  policy: {
    executionTimeout: 5000,
  },
});

console.log(result);

{
  "id": "21bd3763-dcbf-4091-84ee-85b80907a020-u1",
  "status": "IN_QUEUE"
}

For more information, see Execution policy.

Purge queue

You can purge all jobs from a queue by using the purgeQueue() function.

purgeQueue() doesn't affect Jobs in progress.

JavaScript
Output

const { RUNPOD_API_KEY, ENDPOINT_ID } = process.env;
import runpodSdk from "runpod-sdk";

async function main() {
  try {
    const runpod = runpodSdk(RUNPOD_API_KEY);
    const endpoint = runpod.endpoint(ENDPOINT_ID);
    await endpoint.run({
      input: {
        prompt: "Hello, World!",
      },
    });

    const purgeQueue = await endpoint.purgeQueue();
    console.log(purgeQueue);
  } catch (error) {
    console.error("An error occurred:", error);
  }
}

main();

{
  "removed": 1,
  "status": "completed"
}

Prerequisites​

Set your Endpoint Id​

Run the Endpoint​

Run synchronously​

Run asynchronously​

Get results from an asynchronous run​

Poll the status of an asynchronous run​

Stream​

Health check​

Status​

Cancel​

Timeout​

Execution policy​

Purge queue​