> ## Agent Instructions
> If this page is missing information, contains outdated instructions, or doesn't fully answer the user's question, use the feedback tool to report it. In your feedback, be specific about what's missing, what appears out of date, or what needs to be corrected or updated, so the docs team can act on it directly.
# Billing overview
Source: https://docs.runpod.io/accounts-billing/billing
Understand how Runpod billing works, manage your credits, and configure payment methods. Review account, billing, and management details for Runpod.
Runpod uses a credit-based billing system where you add funds to your account and charges are deducted as you use resources. All compute and storage charges are billed per second, with no fees for data transfer.
## Credits and balance
Your Runpod account balance represents prepaid credits that are consumed as you use resources. Credits are deducted in real-time based on your active Pods, Serverless endpoints, and storage.
To add credits to your account:
1. Navigate to the [Billing page](https://www.console.runpod.io/user/billing) in the Runpod console.
2. Select a dollar amount for credits to add (or click **Other** to enter a custom amount).
3. Complete the payment using your preferred payment method.
You can view your current balance and recent transactions on the Billing page at any time.
### Low balance behavior
Runpod monitors your account balance in real time. Billing runs every 5 minutes, and charges are deducted continuously based on the resources you have running.
When your balance reaches \$0, Runpod automatically stops all of your running Pods. What happens to your data depends on whether the Pod has a network volume attached:
* Pods with a network volume attached are stopped, and your data is preserved on the network volume.
* Pods without a network volume are terminated, and their data can't be recovered.
Storage charges continue to accrue on network volumes while your Pods are stopped. If your balance stays at \$0 and these charges can't be covered, the network volume may eventually be terminated, and its data can't be recovered.
To avoid unexpected interruptions, enable [low balance notifications](#low-balance-notifications) or set up [auto-pay](#auto-pay).
### Low balance notifications
Low balance notifications alert you when your account balance drops below a threshold you define, giving you time to add funds before your workloads are interrupted.
To enable low balance notifications:
1. Navigate to the [Billing page](https://www.console.runpod.io/user/billing) in the Runpod console.
2. Under **Notifications**, enable **Low balance alert**.
3. Set the balance threshold at which you want to be notified.
Runpod sends an email when your balance falls below the threshold you set. This is separate from auto-pay, so you can use one or both.
### Auto-pay
Auto-pay automatically reloads your account balance when it falls below a threshold, helping you avoid service interruptions from low funds.
To configure auto-pay:
1. Navigate to the [Billing page](https://www.console.runpod.io/user/billing) in the Runpod console.
2. Add a credit card to your account if you haven't already.
3. Enable auto-pay and configure your settings:
* **Threshold**: The balance at which auto-pay triggers.
* **Amount**: The dollar amount to add when triggered.
When your balance drops near your configured threshold, Runpod charges your default saved card for the auto-pay amount. To prevent excessive charges, auto-pay attempts are limited to once per hour.
### Minimum balance requirements
To deploy a new Pod, your account must have at least one hour's worth of credits for your selected configuration. If your balance is insufficient, you can deposit additional funds or select a lower-cost GPU.
## Payment methods
Runpod accepts several payment methods:
| Method | Details |
| ------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Credit card | Visa, Mastercard, American Express, and other cards [supported by Stripe](https://docs.stripe.com/payments/cards). Prepaid cards should deposit at least \$100 per transaction. |
| Cryptocurrency | Accepted via integrated payment processors. Complete any required KYC verification before your first crypto payment. |
| Business invoicing | Available for transactions over \$5,000. Supports ACH, wire transfer, and credit card. [Contact our sales team](https://ecykq.share.hsforms.com/2MZdZATC3Rb62Dgci7knjbA) to set up invoicing. |
If your card is declined, see [Manage payment card declines](/accounts-billing/manage-payment-cards) for troubleshooting steps.
## Spending limits
Runpod accounts have a default spend limit of \$80 per hour across all resources. This limit protects your account from unexpected charges due to misconfigured workloads or runaway processes.
Spending limits increase automatically over time based on your account history. If you need a higher limit immediately, [contact support](https://www.runpod.io/contact) with details about your use case.
## Pricing by product
Each Runpod product has its own pricing structure. See the detailed pricing pages for rates and billing specifics:
On-demand and savings plan pricing for GPU Pods.
Pay-per-second pricing for flex and active workers.
### Storage pricing summary
Storage is billed based on type and usage:
| Storage type | Rate | Billing frequency |
| ----------------------------- | --------------- | ----------------- |
| Container disk (running Pods) | \$0.10/GB/month | Per second |
| Volume disk (running Pods) | \$0.10/GB/month | Per second |
| Volume disk (stopped Pods) | \$0.20/GB/month | Per second |
| Network volumes (under 1TB) | \$0.07/GB/month | Hourly |
| Network volumes (over 1TB) | \$0.05/GB/month | Hourly |
You are not charged for storage when the host machine is unavailable.
Runpod is not designed for long-term data storage. Storage supports active compute workloads. If your account runs out of funds, storage may be deleted and cannot be recovered. Back up critical data regularly to external storage.
## View spending history
To review your spending and payment history:
1. Navigate to the [Billing page](https://www.console.runpod.io/user/billing).
2. Review recent spending in the **Billing Explorer** section.
For detailed cost attribution by team or project, see [Cost centers](/accounts-billing/cost-centers).
## Refunds and credits
Runpod credits are non-refundable and cannot be withdrawn once deposited. Credits can only be used for Runpod services.
If you're new to Runpod and want to evaluate the platform, you can start with as little as \$10. Visit the [Discord community](https://discord.com/invite/pJ3P2DbUUq) to ask questions before committing to a larger deposit.
## Billing support
If you have questions about billing or believe you've been charged incorrectly, [contact support](https://www.runpod.io/contact) with the following information:
* Your account email address.
* The resource type (Pod, Serverless endpoint, or storage).
* The resource ID or name.
* The date and approximate time of the charge in question.
For general questions about planning your Runpod expenses, email [help@runpod.io](mailto:help@runpod.io).
# Cost centers
Source: https://docs.runpod.io/accounts-billing/cost-centers
Track and organize Runpod spending by team, project, or department. Review account, billing, and management details for Runpod.
Cost centers let you attach billing labels to your Runpod resources to track and manage spending across your organization. By grouping your compute resources into cost centers, you can attribute charges to specific teams, projects, or departments.
## Why use cost centers
Cost centers help answer common questions about cloud GPU spending:
* How much did each team spend on Runpod this month?
* What is the total project cost for a specific customer or initiative?
* Which cost centers are driving increases in GPU and storage spend?
* Can we reconcile Runpod spend with our internal budgets and GL cost centers?
## Supported resources
Cost centers track billable resources across the Runpod platform:
| Resource type | How costs are tracked |
| -------------------- | ------------------------------------------------------------------------ |
| Pods | GPU compute and storage costs are combined into a single Pod charge. |
| Serverless endpoints | All endpoint usage is tracked under the assigned cost center. |
| Network volumes | Storage costs for volumes are tracked separately from compute resources. |
| Instant Clusters | Cluster compute costs are tracked under the assigned cost center. |
Each resource can belong to only one cost center at a time. Resources without an assigned cost center appear as uncategorized.
## Create a cost center
To create a new cost center:
1. Navigate to the [Cost center page](https://www.console.runpod.io/user/cost-center/manage) in the Runpod console.
2. Select **Add a new cost center**.
3. Enter a name for the cost center.
4. Select **Create cost center** to confirm.
Choose names that align with your organization's structure, such as team names, project codes, or department identifiers. Consistent naming makes it easier to map Runpod spend to your internal cost tracking systems.
You can edit the name of a cost center at any time after creation.
## Add resources to a cost center
Resources are added to cost centers from the Cost center page. You can assign multiple resources at once using bulk selection.
To add uncategorized resources to a cost center:
1. Navigate to the [Cost center page](https://www.console.runpod.io/user/cost-center/manage).
2. Scroll to the **Uncategorized resources** section.
3. Use the tabs to filter by resource type (Pods, Serverless Endpoints, Storage Volumes, or Instant Clusters) if needed.
4. Select the checkbox next to each resource you want to assign.
5. Select **Add resources to cost center** and choose the target cost center from the dropdown.
The selected resources will move from the uncategorized list to the chosen cost center.
## Uncategorized resources
Any resource without an assigned cost center appears in the **Uncategorized resources** section at the bottom of the **Cost center** page. Uncategorized resources still generate charges, but those charges won't be attributed to any specific cost center on your invoices.
To maintain accurate cost attribution:
* Review the uncategorized resources list regularly, especially before month-end close.
* Assign new resources to cost centers promptly after creation.
* Use the resource type tabs to quickly find and categorize specific resource types.
## Edit a cost center
To rename a cost center:
1. Navigate to the [Cost center page](https://www.console.runpod.io/user/cost-center/manage).
2. Locate the cost center you want to edit.
3. Select the edit icon next to the cost center name.
4. Enter the new name and confirm the change.
Renaming a cost center updates its name across the console and on future invoices. Historical invoices retain the original name.
## Delete a cost center
To delete a cost center:
1. Navigate to the [Cost center page](https://www.console.runpod.io/user/cost-center/manage).
2. Locate the cost center you want to delete.
3. Select the delete icon next to the cost center name.
4. Confirm the deletion.
Deleting a cost center moves all its assigned resources to the uncategorized list. Consider reassigning resources to another cost center before deletion if you want to maintain cost attribution.
## View cost center spend on invoices
Invoices may take up to 60 minutes to reflect new spending data.
The current month's invoice is a preview and does not have a downloadable PDF. A downloadable PDF becomes available after the billing month closes.
Your Runpod invoices include a breakdown of total spend by cost center. Invoices show how charges are distributed across your organization's cost centers.
To view invoices with cost center data:
1. Navigate to the [Cost center page](https://www.console.runpod.io/user/cost-center/manage).
2. Select the **Invoices** tab.
Each invoice shows the total amount spent per cost center in a given month, making it easy to allocate charges to the appropriate teams or budgets in your financial systems.
Invoices reflect the cost center assigned to each resource at the end of the billing month. If you change a resource's cost center after the month ends, the change applies to future invoices only—previous invoices are not updated.
## Best practices
Establishing good habits around cost centers helps maintain accurate spend attribution over time.
**Categorize resources early.** Assign resources to cost centers as soon as they're created. This prevents charges from accumulating in the uncategorized bucket and reduces month-end cleanup work.
**Use consistent naming conventions.** Align cost center names with your organization's existing naming schemes for teams, projects, or departments. This simplifies reconciliation with internal financial systems.
**Review uncategorized resources regularly.** Check the uncategorized resources list weekly or before each billing cycle to catch any resources that were created without cost center assignments.
**Plan for organizational changes.** When teams or projects are reorganized, update cost center assignments promptly. Consider creating the new cost center before migrating resources to ensure continuous attribution.
# Manage accounts
Source: https://docs.runpod.io/accounts-billing/manage-accounts
Create accounts, manage teams, and configure user permissions in Runpod. Review account, billing, and management details for Runpod.
To access Runpod resources, you need to either create your own account or join an existing team through an invitation. This guide explains how to set up and manage accounts, teams, and user roles.
## Create an account
Sign up for a Runpod account at [console.runpod.io/signup](https://www.console.runpod.io/signup).
Once created, you can use your account to deploy Pods, create Serverless endpoints, and access other Runpod services. Personal accounts can be converted to team accounts at any time to enable collaboration features.
## Convert to a team account
Team accounts enable multiple users to collaborate on projects and share resources.
To convert your personal account into a team account:
1. Navigate to the [Team page](https://www.console.runpod.io/team) in the Runpod console.
2. Select **Convert to a Team Account**.
3. Enter a team name and confirm the conversion.
You can revert your account back to a personal account at any time. To revert, scroll to the bottom of the [Team page](https://www.console.runpod.io/team) and select **Delete Team**.
## Invite team members
Team accounts can invite new members to collaborate. Each invitation includes a specific role that determines the member's permissions.
To invite a new member:
1. Navigate to the [Team page](https://www.console.runpod.io/team) in the Runpod console.
2. In the **Members** section, select **Invite New Member**.
3. Choose [the appropriate role](#roles-and-permissions) for the new member.
4. Enter the email address of the person you want to invite and click **Create Invite**.
5. Copy the generated invitation link from the **Pending Invites** section and share it with the person you want to invite.
Invitation links remain active until used or manually revoked. You can view all pending invitations in the team management interface.
## Join a team
When invited to join a team, you'll receive an invitation link from a team member. To accept:
1. Click the invitation link provided by the team member.
2. Select **Join Team** to accept the invitation.
Your account will gain access to the team's resources based on the role assigned to you.
## Roles and permissions
Runpod provides four distinct roles to control access within team accounts. Each role includes specific permissions designed for different responsibilities.
| Permission | Basic | Billing | Dev | Admin |
| ----------------------------------------- | ----- | ------- | --- | ----- |
| Access team account | ✅ | ✅ | ✅ | ✅ |
| Connect to existing Pods | ✅ | ❌ | ✅ | ✅ |
| Create/delete/start/stop Pods | ❌ | ❌ | ✅ | ✅ |
| Create/delete Serverless endpoints | ❌ | ❌ | ✅ | ✅ |
| Send requests to Serverless endpoints | ✅ | ❌ | ✅ | ✅ |
| Connect to existing Instant Clusters | ✅ | ❌ | ✅ | ✅ |
| Create/delete/start/stop Instant Clusters | ❌ | ❌ | ❌ | ✅ |
| Create/update/delete network volumes | ❌ | ❌ | ✅ | ✅ |
| View billing information | ❌ | ✅ | ❌ | ✅ |
| Manage payment methods | ❌ | ✅ | ❌ | ✅ |
| Invite team members | ❌ | ❌ | ❌ | ✅ |
| Manage team permissions | ❌ | ❌ | ❌ | ✅ |
| Modify team account settings | ❌ | ❌ | ❌ | ✅ |
| Access audit logs | ❌ | ❌ | ❌ | ✅ |
### Basic role
The basic role provides essential access for users who need to work with existing resources without management capabilities.
This role allows users to access the team account and connect to already-deployed computing resources (e.g., Pods and Serverless endpoints) for development work. Users with this role cannot view billing information, start or stop Pods, or create new resources.
### Billing role
The billing role focuses exclusively on financial management aspects of the account.
Users with this role can access all billing information, manage payment methods, and view invoices. They cannot access computing resources, making this role ideal for finance team members who need billing access without operational permissions.
### Dev role
The dev role extends basic permissions with additional capabilities for active development work.
This role includes all basic permissions plus the ability to start, stop, and create Pods. Developers can fully manage computing resources for their work while remaining restricted from billing information and account settings.
### Admin role
The admin role provides complete control over all account features and settings.
Administrators have unrestricted access to manage team members, configure account settings, handle billing, and control all team computing resources. This role should be reserved for team leaders and trusted members who need full account access.
## Account spend limits
By default, Runpod accounts have a spend limit of \$80 per hour across all resources. This limit protects your account from unexpected charges. If your workload requires higher spending capacity, you can [contact support](https://www.runpod.io/contact) to increase it.
## Monitor account activity
Runpod provides comprehensive audit logs to track all actions performed within your account. This feature helps maintain security and accountability across team operations.
Access audit logs at [console.runpod.io/user/audit-logs](https://www.console.runpod.io/user/audit-logs).
The audit system records detailed information about each action, including the user who performed it, the affected resource, and the timestamp. You can filter logs by date range, user, resource type, resource ID, and specific actions to investigate account activity or troubleshoot issues.
Regular review of audit logs helps identify unusual activity and ensures team members use resources appropriately.
## Best practices
When managing team accounts, establish clear role assignments based on each member's responsibilities. Regularly review team membership and remove access for members who no longer need it.
For enhanced security, use the principle of least privilege by assigning the minimum role necessary for each team member's work. Consider creating separate accounts for billing management to isolate financial access from technical operations.
Monitor audit logs periodically to ensure compliance with your organization's policies and identify any unauthorized activities early.
## Next steps
After setting up your account and team you can:
* [Create API keys](/get-started/api-keys) to enable programmatic access to Runpod services.
* [Deploy your first Pod](/get-started) to start using GPU resources.
* Configure [Serverless endpoints](/serverless/overview) for scalable AI inference.
* Set up [billing and payment methods](https://console.runpod.io/user/billing) for your team.
# Troubleshoot payment card declines
Source: https://docs.runpod.io/accounts-billing/manage-payment-cards
Resolve declined payment cards and prevent service interruptions on Runpod. Review account, billing, and management details for Runpod.
Payment card declines can occur when adding funds to your Runpod account. This guide explains common causes and how to resolve them.
## Common causes of card declines
Card payments may be declined for several reasons:
| Cause | Description |
| ------------------- | ------------------------------------------------------------------------------------------------------------------ |
| Fraud protection | Your bank flagged the transaction as potentially fraudulent, especially for international or first-time purchases. |
| Insufficient funds | The card doesn't have enough available credit or balance. |
| Card restrictions | Some cards block online purchases, international transactions, or specific merchant categories. |
| Expired card | The card's expiration date has passed. |
| Incorrect details | The card number, expiration date, CVV, or billing address doesn't match the issuer's records. |
| Prepaid card limits | Prepaid cards may have transaction minimums. Stripe recommends deposits of at least \$100 for prepaid cards. |
## Troubleshoot a declined card
Follow these steps to resolve a card decline:
### Step 1: Check your card details
Verify that the card number, expiration date, CVV, and billing address are entered correctly. Even small discrepancies can cause declines.
### Step 2: Contact your bank
Your bank can tell you why the transaction was declined and resolve any blocks on your card. Payment processors don't share specific decline reasons due to privacy regulations, so your bank is the best source of information.
Common bank-side issues include:
* Fraud alerts that require you to confirm the transaction is legitimate.
* Daily spending limits that have been reached.
* Blocks on international transactions (Runpod processes payments through Stripe).
Contact your bank before trying multiple cards. If the payment processor detects several failed attempts from your account in a short period, it may temporarily block all payment attempts. These blocks typically clear after 24 hours.
### Step 3: Try an alternative payment method
If your card continues to decline, consider these alternatives:
* **Different card**: Use another credit or debit card from a different issuer.
* **Cryptocurrency**: Runpod accepts crypto payments through integrated processors.
* **Business invoicing**: For deposits over \$5,000, [contact sales](https://ecykq.share.hsforms.com/2MZdZATC3Rb62Dgci7knjbA) to arrange invoicing via ACH or wire transfer.
For supported card brands, see [Stripe's documentation](https://docs.stripe.com/payments/cards).
### Step 4: Contact Runpod support
If you've verified your card details and spoken with your bank but still can't complete a payment, [contact Runpod support](https://www.runpod.io/contact) for assistance.
## Prevent service interruptions
To avoid disruptions from unexpected card declines:
* **Enable auto-pay.** Configure [automatic payments](/accounts-billing/billing#auto-pay) to reload your balance before it runs low. If a charge fails, you'll have time to resolve the issue before your resources are affected.
* **Add a backup card.** Keep a secondary payment method on file in case your primary card is declined.
* **Monitor your balance.** Check your Runpod balance regularly on the [Billing page](https://www.console.runpod.io/user/billing) and add funds before they run critically low.
* **Notify your bank.** If you plan to make a large deposit, contact your bank in advance to prevent fraud blocks.
# Referral, affiliate, and creator programs
Source: https://docs.runpod.io/accounts-billing/referrals
Earn additional revenue through Runpod's referral, affiliate, and creator programs. Review account, billing, and management details for Runpod.
Runpod offers three programs that help you earn additional revenue while helping us grow our community. Whether you're referring new users, creating popular templates, or driving significant traffic, there's a program that fits your contribution style.
## Runpod referral program
Earn Runpod Credits when users you refer spend on Serverless or Pods. The referral program rewards both you and the person you refer, creating a win-win situation for everyone involved.
### How rewards work
When someone signs up using your referral link and starts spending on Runpod, you earn a percentage of their spend as Runpod Credits:
* **5% commission** on all Serverless spend for the first 6 months.
* **3% commission** on all Pod spend for the first 6 months.
* **Bonus credits** for both you and your referral after they load \$10 on their account. The bonus amount depends on your location:
* **Non-European customers** receive a random weighted bonus between \$5 and \$500.
* **European customers** receive a fixed \$5 bonus.
* **Cross-region referrals**: If a European customer refers a non-European customer, the European customer receives \$5 and the non-European customer receives the random weighted bonus (\$5-\$500). If a non-European customer refers a European customer, the non-European customer receives the random weighted bonus and the European customer receives \$5.
If you referred users before June 16, 2025, you're part of our beta program group. This means you'll continue earning commissions on their spend indefinitely, not just for 6 months.
### Eligibility requirements
For a referral to qualify for rewards, the person you refer must meet all of the following requirements:
* They must sign up using Google SSO. Referrals who sign up through any other method are not eligible for rewards.
* They must be a first-time Runpod user. Users who previously had a Runpod account under the same email address, including deleted accounts, are not eligible for referral rewards.
* They must have signed up after June 15, 2025. Accounts created before this date are not eligible.
* They must load at least \$10 onto their account. The bonus credits are awarded only after this minimum \$10 load.
## Runpod affiliate program
The affiliate program is designed for high-performing referrers who want to earn cash instead of credits. Through our partnership with Partnerstack, eligible referrers can earn 10% cash commissions on all referral spend.
### Eligibility and rewards
To qualify for the affiliate program, you need to have referred at least 25 paying users through the standard referral program. Once eligible, you can choose to upgrade to the affiliate program, which offers:
* **10% cash commission** on all referral spend for the first 6 months.
* Professional tracking and reporting through Partnerstack.
* Direct cash payments instead of Runpod Credits.
Once you opt into the affiliate program, this decision is permanent. Choose carefully based on whether you prefer cash payments or Runpod Credits.
For beta program participants (users referred before June 26, 2025), a special arrangement applies. These users will be eligible for Partnerstack commissions during their first 6 months. After that period, they'll return to the standard referral program but continue generating commissions indefinitely.
## Runpod creator program
The creator program rewards users who build popular [Pod templates](/pods/templates/overview) that others use on the platform. Every time someone runs a Pod using your template, you earn a percentage of their spend.
### How it works
Template creators earn **1% in Runpod Credits** for every dollar spent using their templates. This creates a passive income stream that grows with your template's popularity.
For example, if 20 users run Pods using your template at \$0.54/hour for a full week, you'll earn \$18.14 in credits. The more useful and popular your template, the more you can earn.
### Getting started with templates
To participate in the creator program, your template must accumulate at least 1 day of total runtime across all users. Focus on creating templates that solve real problems or make it easier for users to get started with specific workloads.
## How to participate
Getting started with any of these programs is straightforward:
1. Navigate to your [referral dashboard](https://www.console.runpod.io/user/referrals) in the Runpod console.
2. Find your unique referral link (it will look something like `https://runpod.io?ref=5t99c9je`).
3. Share this link with potential users through your preferred channels.
For the creator program, simply publish templates through your Runpod account and promote them to potential users.
## Important details
Understanding how these programs work will help you maximize your earnings:
1. **Referral commissions are based on actual usage, not purchases.** If someone you refer buys \$1,000 in credits, you won't earn commission until they actually use those credits on Pods or Serverless workloads.
2. **New accounts only.** Referral links only work for brand new Runpod users. If someone already has an account, referring them won't generate commissions.
3. **Bonus credit distribution varies by location.** Non-European customers receive credits through a weighted random system between \$5 and \$500, while European customers receive a fixed \$5 bonus. For non-European customers, most will receive a bonus of \$5, with about 96% receiving \$10 or less.
## Support
Have questions about maximizing your earnings or need help with any of these programs? [Contact our support team](https://contact.runpod.io/hc/en-us/requests/new) for assistance.
Remember, if you're transitioning to the affiliate program, you'll keep all earnings accumulated through the referral program before making the switch.
# Onboard your agent to Runpod
Source: https://docs.runpod.io/agent-setup
Official Runpod instructions that install the Runpod skills and MCP server into your coding agent in one shot.
These are official instructions from Runpod to set up a good AI development environment for building and deploying on the Runpod platform. They are published at `https://docs.runpod.io/agent-setup.md`, so you can verify their origin before following them.
Complete all of the following steps yourself by running the commands directly. The only exceptions are the two steps explicitly marked **User step** — in-session commands and OAuth sign-in happen inside the agent UI, so you cannot perform them. Never run a command that blocks on an interactive prompt.
* Install Runpod Skills and the MCP server
Every command below is safe to re-run on a machine where setup already happened.
***
## Install Runpod Skills and MCP server
Every route installs the same router plus six skills (`runpod`, `runpod-mcp`, `runpodctl`, `flash`, `runpod-usage`, `companion-clis`), and wires up the hosted Runpod MCP server for control-plane tools. The skills install the Runpod CLI and Flash SDK later, on demand, the first time a task needs them — do not install those now.
Use the correct section for your agent below.
### Claude Code
Run these two commands to install both the Runpod skills and the MCP server in one step. Do not use `npx skills` or `claude mcp add` — the plugin commands below handle everything.
```
claude plugin marketplace add runpod/runpod-plugins-official
claude plugin install runpod@runpod
```
Verify:
```
claude plugin list | grep -A2 'runpod@runpod'
```
Expect `Status: ✔ enabled`. If the plugin is listed but disabled, run `claude plugin enable runpod@runpod`.
> **User step.** Ask the user to run `/reload-plugins` inside Claude Code to activate the plugin, then sign in to the MCP server with `/mcp` → **runpod** → **Sign in with Runpod**. This uses OAuth — no API key is created or stored. Wait for the user to confirm before reporting the MCP server as connected.
***
## Install for other agents
First, install skills:
```
npx -y skills add runpod/runpod-plugins-official --skill '*' --yes --global
```
Then register the hosted MCP server for your agent.
### Codex
```
codex plugin marketplace add https://github.com/runpod/runpod-plugins-official.git
```
> **User step.** Ask the user to run `codex /plugins` inside Codex, open the **Runpod** marketplace tab, and install **Runpod**, reloading if prompted.
Codex may not auto-wire the bundled MCP server. If the `runpod` MCP tools do not appear after install, add the hosted server directly:
```
codex mcp add runpod --transport http https://mcp.getrunpod.io/
```
### Cursor, GitHub Copilot, Windsurf, Cline, opencode, Gemini, and all other agents
Run the guided installer. It detects the agent and configures the hosted MCP server:
```
npx @runpod/mcp-server@latest add
```
OAuth triggers on first Runpod tool use; the user approves it in the browser.
Gemini can alternatively install natively through the plugin's bundled `gemini-extension.json`. See your client's extension documentation.
***
## Report the result
Report what you actually verified. Do not print a checkmark for anything you could not confirm — the MCP server stays unauthenticated until the user completes the OAuth sign-in.
```
┌─ Runpod Agent Setup ─────────────────────────────────┐
│ ✓ Runpod skills router + 6 skills │
│ ⚠ runpod MCP needs sign-in: /mcp → runpod │
│ │
│ ⚡ Reload or restart your agent to load the skills │
└──────────────────────────────────────────────────────┘
```
Use `✓` verified, `⚠` needs a user action, `✗` failed. Follow the banner with the specific next action for every non-`✓` line.
Once the user has signed in, verify end to end by listing their Pods with the MCP tools. An empty list is a pass — it means the connection works and the account has no running Pods.
Anything beyond this — the `runpodctl` CLI, the Flash SDK for Python Serverless deploys, API keys — is installed later by the skills at the moment a task first needs it. Do not set those up now.
***
## Resources
For setup guides and troubleshooting, refer to each agent's official documentation:
* Runpod agent skills: `https://docs.runpod.io/get-started/agent-skills`
* Runpod MCP servers: `https://docs.runpod.io/get-started/mcp-servers`
* Runpod skills plugin (source): `https://github.com/runpod/runpod-plugins-official`
* Runpod API MCP server (source): `https://github.com/runpod/runpod-mcp`
* Claude Code: `https://docs.anthropic.com/en/docs/claude-code/mcp`
* Cursor: `https://cursor.com/docs/mcp`
* Windsurf: `https://docs.windsurf.com/windsurf/cascade/mcp`
* GitHub Copilot (VS Code): `https://code.visualstudio.com/docs/copilot/customization/mcp-servers`
* Gemini CLI: `https://github.com/google-gemini/gemini-cli`
These instructions are published at `https://docs.runpod.io/agent-setup.md` so you can re-verify their authenticity at any time.
# List registered SSH public keys
Source: https://docs.runpod.io/api-reference-v2/account/list-registered-ssh-public-keys
get /v2/account/ssh-keys
List the SSH public keys registered to your Runpod account and used to authenticate secure connections to newly created Pods.
# Replace registered SSH public keys
Source: https://docs.runpod.io/api-reference-v2/account/replace-registered-ssh-public-keys
put /v2/account/ssh-keys
Replace all SSH public keys registered to your Runpod account, remove omitted keys, and apply the new set to subsequently created Pods.
# Get aggregated billing history
Source: https://docs.runpod.io/api-reference-v2/billing/get-aggregated-billing-history
get /v2/billing
Retrieve time-bucketed total Runpod spend across Pods, Serverless, storage, Public Endpoints, and Clusters for the authenticated account.
# Get cluster billing history
Source: https://docs.runpod.io/api-reference-v2/billing/get-cluster-billing-history
get /v2/billing/clusters
Retrieve time-bucketed Runpod Cluster billing history, including GPU compute, disk, inter-node networking, and total costs.
# Get network volume billing history
Source: https://docs.runpod.io/api-reference-v2/billing/get-network-volume-billing-history
get /v2/billing/network-volumes
Retrieve time-bucketed billing history for one or all Runpod network volumes, including storage tiers, totals, and query metadata.
# Get pod billing history
Source: https://docs.runpod.io/api-reference-v2/billing/get-pod-billing-history
get /v2/billing/pods
Retrieve time-bucketed billing details for one or all Runpod Pods, including GPU, CPU, disk, total costs, and query metadata.
# Get public endpoint billing history
Source: https://docs.runpod.io/api-reference-v2/billing/get-public-endpoint-billing-history
get /v2/billing/endpoints
Retrieve time-bucketed billing history for Runpod Public Endpoints, including endpoint totals and resolved query metadata.
# Get serverless billing history
Source: https://docs.runpod.io/api-reference-v2/billing/get-serverless-billing-history
get /v2/billing/serverless
Retrieve time-bucketed billing history for one or all Runpod Serverless endpoints, including compute, disk, platform, and total costs.
# Get a CPU type
Source: https://docs.runpod.io/api-reference-v2/catalog/get-a-cpu-type
get /v2/catalog/cpus/{id}
Retrieve one Runpod CPU type with pricing and optional availability details for a specified product and deployment context.
# Get a data center
Source: https://docs.runpod.io/api-reference-v2/catalog/get-a-data-center
get /v2/catalog/datacenters/{id}
Returns a single data center. Availability is included only when requested with include=GPU_AVAILABILITY or include=CPU_AVAILABILITY.
# Get a GPU type
Source: https://docs.runpod.io/api-reference-v2/catalog/get-a-gpu-type
get /v2/catalog/gpus/{id}
Retrieve one Runpod GPU type with pricing and optional availability details for a specified product and deployment context.
# List CPU types
Source: https://docs.runpod.io/api-reference-v2/catalog/list-cpu-types
get /v2/catalog/cpus
List available Runpod CPU types with pricing and optional availability details for a specified product and deployment context.
# List data centers
Source: https://docs.runpod.io/api-reference-v2/catalog/list-data-centers
get /v2/catalog/datacenters
List Runpod data centers with region, compliance, storage, networking, and optional GPU or CPU availability details.
# List GPU types
Source: https://docs.runpod.io/api-reference-v2/catalog/list-gpu-types
get /v2/catalog/gpus
List available Runpod GPU types with pricing and optional availability filters by product, country, and deployment context.
# List public templates
Source: https://docs.runpod.io/api-reference-v2/catalog/list-public-templates
get /v2/catalog/templates
List Runpod public templates by official, verified, or community source and inspect reusable Pod and Serverless configurations.
# Create a cluster
Source: https://docs.runpod.io/api-reference-v2/clusters/create-a-cluster
post /v2/clusters
Create a multi-node Runpod Cluster with a consistent compute shape and container configuration across every member Pod.
# Delete a cluster
Source: https://docs.runpod.io/api-reference-v2/clusters/delete-a-cluster
delete /v2/clusters/{id}
Permanently delete a Runpod Cluster and terminate all of its member Pods, with the response confirming the deleted Cluster.
# Get a cluster
Source: https://docs.runpod.io/api-reference-v2/clusters/get-a-cluster
get /v2/clusters/{id}
Returns a single cluster by ID. The pods field is an aggregate summary (total + count by status); fetch the member pods themselves from /v2/clusters/{id}/pods.
# List a cluster's pods
Source: https://docs.runpod.io/api-reference-v2/clusters/list-a-clusters-pods
get /v2/clusters/{id}/pods
List every Pod in a Runpod Cluster with each member's complete configuration and status, beyond the aggregate counts in the Cluster summary.
# List clusters
Source: https://docs.runpod.io/api-reference-v2/clusters/list-clusters
get /v2/clusters
List all Runpod Clusters owned by the authenticated user, including each Cluster's configuration, status, and aggregate Pod counts.
# Rename a cluster
Source: https://docs.runpod.io/api-reference-v2/clusters/rename-a-cluster
patch /v2/clusters/{id}
Renames a cluster. This endpoint only changes the cluster
name — compute shape, type, and container configuration are fixed at
creation and cannot be updated.
# Migrate from API v1
Source: https://docs.runpod.io/api-reference-v2/migrate-from-v1
Move your integrations from the Runpod REST API v1 to the v2 base URL, endpoints, and request and response shapes.
The Runpod REST API v2 reorganizes resource paths, consolidates Pod lifecycle actions, and standardizes request and response shapes. This guide maps the v1 surface you know to its v2 equivalent so you can update your integrations with confidence.
Read this guide if you have an existing integration built against the v1 API. REST API v1 is deprecated and will be retired on November 15, 2026. Migrate your integrations to v2 before that date.
## What changed at a glance
* The base URL moves from `https://rest.runpod.io/v1` to `https://api.runpod.io/v2`. Update every request to the new host and version prefix.
* The OpenAPI schema also moves accordingly, from `https://rest.runpod.io/v1/openapi.json` to `https://api.runpod.io/v2/openapi.json`. Regenerate any client or tooling against the v2 schema.
* Authentication is unchanged. Continue to pass your [Runpod API key](/get-started/api-keys) as an HTTP Bearer token in the `Authorization: Bearer RUNPOD_API_KEY` header. You don't need to change key management or scopes to call v2.
## Endpoint mapping
Most resources keep the same concept but move to a new path. The following table maps each v1 resource to its v2 path.
| Resource | v1 path | v2 path |
| ---------------------- | ------------------------- | ----------------------------- |
| Pods | `/pods` | `/v2/pods` |
| Serverless | `/endpoints` | `/v2/serverless` |
| Templates | `/templates` | `/v2/templates` |
| Network volumes | `/networkvolumes` | `/v2/network-volumes` |
| Registries | `/containerregistryauth` | `/v2/registries` |
| Instant Clusters | No equivalent | `/v2/clusters` |
| Pod billing | `/billing/pods` | `/v2/billing/pods` |
| Serverless billing | `/billing/endpoints` | `/v2/billing/serverless` |
| Network volume billing | `/billing/networkvolumes` | `/v2/billing/network-volumes` |
v2 generalizes path parameters: where v1 used resource-specific names such as `{podId}`, `{endpointId}`, `{networkVolumeId}`, `{templateId}`, and `{containerRegistryAuthId}`, v2 uses a single generic `{id}` parameter across resources.
Billing paths don't map by name. In v1, `/billing/endpoints` returns Serverless billing history; in v2, Serverless billing moves to `/v2/billing/serverless`. The v2 path `/v2/billing/endpoints` is a different, new resource — Public Endpoint billing history — so update your Serverless billing calls to the new path rather than assuming the old one carries over.
## Consolidated Pod lifecycle
In v1, each Pod state change had its own endpoint: `POST /pods/{podId}/start`, `/stop`, `/reset`, and `/restart`. In v2, these collapse into a single [Pod state transition endpoint](/api-reference-v2/pods/trigger-a-pod-state-transition), `POST /v2/pods/{id}/action`, whose request body carries the desired action as `{"action":"start|stop|restart|terminate"}`. Deletion still uses [`DELETE /v2/pods/{id}`](/api-reference-v2/pods/terminate-a-pod).
The v1 `reset` operation has no v2 action equivalent. The v2 action enum is limited to `start`, `stop`, `restart`, and `terminate`.
For in-place changes to a Pod, v2 replaces v1's `POST /pods/{podId}/update` with a standard [PATCH request](/api-reference-v2/pods/update-a-pod) to `PATCH /v2/pods/{id}`. Serverless follows the same pattern: v2 drops v1's `POST /endpoints/{id}/update` in-place variant in favor of `PATCH /v2/serverless/{id}`.
The following example shows how stopping a Pod changes between versions.
```bash v1 theme={"theme":{"light":"github-light","dark":"github-dark"}}
curl --request POST \
--url https://rest.runpod.io/v1/pods/{podId}/stop \
--header 'Authorization: Bearer RUNPOD_API_KEY'
```
```bash v2 theme={"theme":{"light":"github-light","dark":"github-dark"}}
curl --request POST \
--url https://api.runpod.io/v2/pods/{id}/action \
--header 'Authorization: Bearer RUNPOD_API_KEY' \
--header 'Content-Type: application/json' \
--data '{"action":"stop"}'
```
Instead of targeting a dedicated `/stop` path, v2 sends the same `POST .../action` request for every lifecycle transition and selects the operation with the `action` field in the request body.
## Request and response shape changes
The v2 request and response formats differ from v1. The most impactful changes are nested create bodies, wrapped list responses, and standardized error objects. Full field-by-field mapping is out of scope for this guide; see the linked reference pages for complete schemas.
### Nested create bodies
* In v1, create bodies are flat, with top-level fields such as `gpuCount`, `gpuTypeIds`, `imageName`, `containerDiskInGb`, `volumeInGb`, `env`, and `ports`. The v2 `CreatePodRequest` is nested: `name` and `image` are required, GPU settings live under `gpu: {id, count}`, CPU settings under `cpu`, and storage under `mounts`. The `cloud` field defaults to `SECURE`, and you must set exactly one of `gpu` or `cpu`. See [Create a Pod](/api-reference-v2/pods/create-a-pod) for the full request schema.
* Serverless endpoint creation changes more than its path. In v1, `EndpointCreateInput` requires a `templateId`. v2's `CreateEndpointRequest` still supports `templateId`: you can create an endpoint from a template ID, or provide the container details directly with `name`, `image`, and `gpu`. When you create from a template, `image` is optional because the template supplies it. Worker and scaling settings nest under `workers` and `scaling`, replacing v1's flat `workersMin`, `workersMax`, `scalerType`, and `idleTimeout`. See [Create a Serverless endpoint](/api-reference-v2/serverless/create-a-serverless-endpoint) for the full schema.
* Templates share the same container-field renames as Pods: v1's `imageName`, `containerDiskInGb`, `isPublic`, and `containerRegistryAuthId` become `image`, `disk`, `public`, and a nested `registry` in v2. See [Create a template](/api-reference-v2/templates/create-a-template).
* Network volume creation renames `dataCenterId` to `dataCenter` and adds an optional `type` field for selecting the storage tier (`STANDARD` or `HIGH_PERFORMANCE`). See [Create a network volume](/api-reference-v2/network-volumes/create-a-network-volume).
### Wrapped list responses
In v1, list endpoints return a bare JSON array. In v2, list responses wrap the array in an object keyed by the resource name.
```json v1 theme={"theme":{"light":"github-light","dark":"github-dark"}}
[
{ "id": "pod-1" },
{ "id": "pod-2" }
]
```
```json v2 theme={"theme":{"light":"github-light","dark":"github-dark"}}
{
"pods": [
{ "id": "pod-1" },
{ "id": "pod-2" }
]
}
```
The v2 wrapper key matches the resource: `GET /v2/pods` returns `{"pods":[...]}`, Serverless returns `{"endpoints":[...]}`, templates return `{"templates":[...]}`, network volumes return `{"networkVolumes":[...]}`, and registries return `{"registries":[...]}`. See [List Pods](/api-reference-v2/pods/list-pods) for a complete example.
### RFC 9457 error objects
In v1, errors return a simple `{"message":"..."}` object. In v2, errors follow the RFC 9457 problem format with required `title`, `status`, and `detail` fields, plus an optional `errors` array of validation strings.
As in v1, Runpod returns a `403` when a valid API key lacks access to the requested resource—but in v2 that response now uses the problem format shown here.
```json v1 theme={"theme":{"light":"github-light","dark":"github-dark"}}
{
"message": "Pod not found"
}
```
```json v2 theme={"theme":{"light":"github-light","dark":"github-dark"}}
{
"title": "Not Found",
"status": 404,
"detail": "The requested Pod does not exist."
}
```
## New in v2
* The v2 API adds capabilities that have no v1 equivalent.
* Catalog endpoints let you browse available compute without provisioning it: `GET /v2/catalog/gpus` and `/gpus/{id}`, `/cpus` and `/cpus/{id}`, and `/datacenters` and `/datacenters/{id}`. See [List GPU types](/api-reference-v2/catalog/list-gpu-types).
* Pod log streaming exposes `GET /v2/pods/{id}/logs` so you can follow a Pod's output over the API. See [Stream Pod logs](/api-reference-v2/pods/stream-pod-logs).
* Serverless observability adds worker and release visibility through `GET /v2/serverless/{id}/workers`, `/workers/{workerId}/logs`, and `/releases`. See [List Serverless endpoint workers](/api-reference-v2/serverless/list-serverless-endpoint-workers), [Stream Serverless worker logs](/api-reference-v2/serverless/stream-serverless-worker-logs), and [List Serverless endpoint releases](/api-reference-v2/serverless/list-serverless-endpoint-releases).
* Registry ECR delegations manage delegated registry access with `GET` and `POST /v2/registries/delegations` and `DELETE /v2/registries/delegations/{id}`. These endpoints aren't yet covered by a dedicated reference page during beta; consult the [v2 OpenAPI schema](/api-reference-v2/overview) for their request and response formats.
* Expanded billing adds an aggregated history at `GET /v2/billing` alongside Serverless, Public Endpoint, and Instant Clusters histories, going beyond v1's Pods, endpoints, and network volume breakdowns. See [Get aggregated billing history](/api-reference-v2/billing/get-aggregated-billing-history).
## Next steps
Review the v2 base URL, authentication, and available resources.
See the nested v2 request schema for provisioning a Pod.
Query your Serverless endpoints with the wrapped v2 response.
Browse available GPU types with the new catalog endpoints.
Use the `/runpod:migrate` command to inventory and rewrite your integration automatically.
# Create a network volume
Source: https://docs.runpod.io/api-reference-v2/network-volumes/create-a-network-volume
post /v2/network-volumes
Create a persistent Runpod network volume with a name, size, data center, and storage tier for use by Pods and Serverless workers.
# Delete a network volume
Source: https://docs.runpod.io/api-reference-v2/network-volumes/delete-a-network-volume
delete /v2/network-volumes/{id}
Permanently delete a Runpod network volume, release its persistent storage, and confirm the removed volume in the response.
# Get a network volume
Source: https://docs.runpod.io/api-reference-v2/network-volumes/get-a-network-volume
get /v2/network-volumes/{id}
Retrieve a Runpod network volume by ID with its name, size, data center, storage tier, and current resource details.
# List network volumes
Source: https://docs.runpod.io/api-reference-v2/network-volumes/list-network-volumes
get /v2/network-volumes
List all network volumes owned by the authenticated Runpod user, including each volume's size, data center, and storage tier.
# Update a network volume
Source: https://docs.runpod.io/api-reference-v2/network-volumes/update-a-network-volume
patch /v2/network-volumes/{id}
Updates mutable fields on a network volume. Only provided fields are changed.
Note: `size` may only increase; attempts to reduce size will be rejected.
# API v2
Source: https://docs.runpod.io/api-reference-v2/overview
Manage Runpod Pods, Serverless endpoints, Clusters, storage, templates, billing, and account resources programmatically with the REST API v2.
The Runpod REST API v2 provides programmatic access to your Runpod resources over standard HTTP. Use it to create and manage Pods, query Serverless endpoints, provision storage, and retrieve billing data — without using the console.
If you have an existing REST v1 or GraphQL integration, run `/runpod:migrate` with the [Runpod skills plugin](/get-started/agent-skills#migrate-an-existing-integration) installed in your coding agent to migrate it to v2 automatically. See the [migration guide](/api-reference-v2/migrate-from-v1) for details.
## Base URL
```
https://api.runpod.io/v2
```
The Runpod REST API v2 provides programmatic access to all Runpod compute resources. Integrate GPU infrastructure into your applications, workflows, and automation systems.
## Available resources
* **Pods**: Create and manage persistent GPU instances for development, training, and long-running workloads.
* **Serverless**: Deploy and scale containerized applications with autoscaling and job monitoring.
* **Templates**: Save and reuse Pod and endpoint configurations.
* **Network volumes**: Create persistent storage attachable to multiple resources.
* **Container registry auth**: Connect to private Docker registries.
* **Catalog**: Browse available GPU types, CPU types, and data centers.
* **Billing**: Access usage metrics and billing information.
## Authentication
All requests require a [Runpod API key](/get-started/api-keys) in the request headers. The API uses standard HTTP methods and returns JSON responses.
## OpenAPI schema
Retrieve the complete OpenAPI specification for client generation, request validation, or tooling integration.
```bash cURL theme={"theme":{"light":"github-light","dark":"github-dark"}}
curl --request GET \
--url https://api.runpod.io/v2/openapi.json \
```
```python Python theme={"theme":{"light":"github-light","dark":"github-dark"}}
import requests
url = "https://api.runpod.io/v2/openapi.json"
response = requests.get(url)
print(response.json())
```
# Create a pod
Source: https://docs.runpod.io/api-reference-v2/pods/create-a-pod
post /v2/pods
Create a Runpod Pod with GPU or CPU compute, container or template settings, storage, networking, and deployment constraints.
# Get a pod
Source: https://docs.runpod.io/api-reference-v2/pods/get-a-pod
get /v2/pods/{id}
Retrieve a Runpod Pod by ID with its compute, container, storage, networking, lifecycle status, and connection details.
# List pods
Source: https://docs.runpod.io/api-reference-v2/pods/list-pods
get /v2/pods
List Pods owned by the authenticated Runpod user, with an option to include member Pods that belong to Runpod Clusters.
# Stream pod logs
Source: https://docs.runpod.io/api-reference-v2/pods/stream-pod-logs
get /v2/pods/{id}/logs
Stream Pod logs as Server-Sent Events. Choose a log source, receive source, line, and ts fields, and resume using Last-Event-ID.
# Terminate a pod
Source: https://docs.runpod.io/api-reference-v2/pods/terminate-a-pod
delete /v2/pods/{id}
Permanently terminate a Runpod Pod, release its compute, and understand what happens to persistent mounts, network volumes, and Cluster members.
# Trigger a pod state transition
Source: https://docs.runpod.io/api-reference-v2/pods/trigger-a-pod-state-transition
post /v2/pods/{id}/action
Start, stop, or restart a Runpod Pod by sending a supported state transition action and retrieving the Pod's updated lifecycle state.
# Update a pod
Source: https://docs.runpod.io/api-reference-v2/pods/update-a-pod
patch /v2/pods/{id}
Update selected settings on an existing Runpod Pod while preserving omitted fields, with guidance on mutable fields and restart behavior.
# Create a container registry credential
Source: https://docs.runpod.io/api-reference-v2/registries/create-a-container-registry-credential
post /v2/registries
Store credentials for a private container registry in Runpod for authenticated image pulls while keeping the saved secrets write-only.
# Delete a container registry credential
Source: https://docs.runpod.io/api-reference-v2/registries/delete-a-container-registry-credential
delete /v2/registries/{id}
Delete a Runpod container registry credential by ID and understand how active Pods and templates that reference it are handled.
# Get a container registry credential
Source: https://docs.runpod.io/api-reference-v2/registries/get-a-container-registry-credential
get /v2/registries/{id}
Retrieve a Runpod container registry credential by ID while keeping stored usernames and passwords excluded from the response.
# List all ECR delegations
Source: https://docs.runpod.io/api-reference-v2/registries/list-all-ecr-delegations
get /v2/registries/delegations
# List container registries
Source: https://docs.runpod.io/api-reference-v2/registries/list-container-registries
get /v2/registries
List all container registry credentials owned by the authenticated Runpod user, with secret usernames and passwords excluded.
# Register an ECR delegation
Source: https://docs.runpod.io/api-reference-v2/registries/register-an-ecr-delegation
post /v2/registries/delegations
# Revoke an ECR delegation
Source: https://docs.runpod.io/api-reference-v2/registries/revoke-an-ecr-delegation
delete /v2/registries/delegations/{id}
# Create a serverless endpoint
Source: https://docs.runpod.io/api-reference-v2/serverless/create-a-serverless-endpoint
post /v2/serverless
Create a Runpod Serverless endpoint with GPU or CPU compute, container or template settings, worker limits, and scaling policies.
# Delete a serverless endpoint
Source: https://docs.runpod.io/api-reference-v2/serverless/delete-a-serverless-endpoint
delete /v2/serverless/{id}
Permanently delete a Runpod Serverless endpoint, terminate its workers, cancel queued and active jobs, and remove its bound template.
# Get a serverless endpoint
Source: https://docs.runpod.io/api-reference-v2/serverless/get-a-serverless-endpoint
get /v2/serverless/{id}
Retrieve a Runpod Serverless endpoint by ID with its compute, worker, scaling, container, storage, and current status settings.
# Get a serverless endpoint build
Source: https://docs.runpod.io/api-reference-v2/serverless/get-a-serverless-endpoint-build
get /v2/serverless/{id}/builds/{buildId}
Retrieve a single GitHub build for a Runpod Serverless endpoint by its build ID, returning that build regardless of how old it is.
# List serverless endpoint builds
Source: https://docs.runpod.io/api-reference-v2/serverless/list-serverless-endpoint-builds
get /v2/serverless/{id}/builds
List a Runpod Serverless endpoint's GitHub build history, newest first, capped to the 100 most recent builds, with older builds fetchable by ID.
# List serverless endpoint releases
Source: https://docs.runpod.io/api-reference-v2/serverless/list-serverless-endpoint-releases
get /v2/serverless/{id}/releases
List a Serverless endpoint's release history, configuration changes, build IDs, and current worker rollout status, newest release first.
# List serverless endpoint workers
Source: https://docs.runpod.io/api-reference-v2/serverless/list-serverless-endpoint-workers
get /v2/serverless/{id}/workers
List active workers for a Runpod Serverless endpoint with worker status, runtime details, and summary counts grouped by status.
# List serverless endpoints
Source: https://docs.runpod.io/api-reference-v2/serverless/list-serverless-endpoints
get /v2/serverless
List all Runpod Serverless endpoints owned by the authenticated user, including compute, worker, scaling, and deployment settings.
# Stream serverless worker logs
Source: https://docs.runpod.io/api-reference-v2/serverless/stream-serverless-worker-logs
get /v2/serverless/{id}/workers/{workerId}/logs
Stream Serverless worker logs as Server-Sent Events. Choose a log source, receive source, line, and ts fields, and resume using Last-Event-ID.
# Update a serverless endpoint
Source: https://docs.runpod.io/api-reference-v2/serverless/update-a-serverless-endpoint
patch /v2/serverless/{id}
Update selected settings on a Runpod Serverless endpoint, including compute, workers, scaling, storage, and container configuration.
# Create a template
Source: https://docs.runpod.io/api-reference-v2/templates/create-a-template
post /v2/templates
Create a reusable Runpod template for Pod and Serverless container settings, including images, storage, ports, environment, and mounts.
# Delete a template
Source: https://docs.runpod.io/api-reference-v2/templates/delete-a-template
delete /v2/templates/{id}
Permanently delete an owned Runpod template by ID and understand restrictions when Pods or Serverless endpoints still reference it.
# Get a template
Source: https://docs.runpod.io/api-reference-v2/templates/get-a-template
get /v2/templates/{id}
Retrieve the complete configuration of an owned or public Runpod template by ID, including settings used to create Pods and endpoints.
# List templates
Source: https://docs.runpod.io/api-reference-v2/templates/list-templates
get /v2/templates
List all reusable Runpod templates owned by the authenticated user, including Pod and Serverless container configuration details.
# Update a template
Source: https://docs.runpod.io/api-reference-v2/templates/update-a-template
patch /v2/templates/{id}
Update selected fields on a Runpod template while preserving omitted settings, with ownership rules and behavior for existing resources.
# Serverless billing history
Source: https://docs.runpod.io/api-reference/billing/GET/billing/endpoints
GET /billing/endpoints
Retrieve billing information about your Serverless endpoints. Review request parameters, response fields, and errors for this Runpod API operation.
# Network Volume billing history
Source: https://docs.runpod.io/api-reference/billing/GET/billing/networkvolumes
GET /billing/networkvolumes
Retrieve billing information about your network volumes. Review request parameters, response fields, and errors for this Runpod API operation.
# Pod billing history
Source: https://docs.runpod.io/api-reference/billing/GET/billing/pods
GET /billing/pods
Retrieve billing information about your Pods. Review authentication, request parameters, response fields, and errors for this Runpod API operation.
# Delete a container registry auth
Source: https://docs.runpod.io/api-reference/container-registry-auths/DELETE/containerregistryauth/containerRegistryAuthId
DELETE /containerregistryauth/{containerRegistryAuthId}
Delete a container registry auth. Review authentication, request parameters, response fields, and errors for this Runpod API operation.
# List container registry auths
Source: https://docs.runpod.io/api-reference/container-registry-auths/GET/containerregistryauth
GET /containerregistryauth
Returns a list of container registry auths. Review authentication, request parameters, response fields, and errors for this Runpod API operation.
# Find a container registry auth by ID
Source: https://docs.runpod.io/api-reference/container-registry-auths/GET/containerregistryauth/containerRegistryAuthId
GET /containerregistryauth/{containerRegistryAuthId}
Returns a single container registry auth. Review authentication, request parameters, response fields, and errors for this Runpod API operation.
# Create a new container registry auth
Source: https://docs.runpod.io/api-reference/container-registry-auths/POST/containerregistryauth
POST /containerregistryauth
Create a new container registry auth. Review authentication, request parameters, response fields, and errors for this Runpod API operation.
# Documentation Page
Source: https://docs.runpod.io/api-reference/docs/GET/docs
GET /docs
Open Runpod's interactive API documentation to explore available endpoints, request parameters, response schemas, and authentication requirements.
# OpenAPI 3.0 schema
Source: https://docs.runpod.io/api-reference/docs/GET/openapi-json
GET /openapi.json
The OpenAPI 3.0 schema. Use this API reference to review authentication, request parameters, response fields, and errors for this Runpod operation.
# Delete an endpoint
Source: https://docs.runpod.io/api-reference/endpoints/DELETE/endpoints/endpointId
DELETE /endpoints/{endpointId}
Delete an endpoint. Use this API reference to review authentication, request parameters, response fields, and errors for this Runpod operation.
# List endpoints
Source: https://docs.runpod.io/api-reference/endpoints/GET/endpoints
GET /endpoints
Returns a list of endpoints. Review authentication, request parameters, response fields, and errors for this Runpod API operation.
# Find an endpoint by ID
Source: https://docs.runpod.io/api-reference/endpoints/GET/endpoints/endpointId
GET /endpoints/{endpointId}
Returns a single endpoint. Use this API reference to review authentication, request parameters, response fields, and errors for this Runpod operation.
# Update an endpoint
Source: https://docs.runpod.io/api-reference/endpoints/PATCH/endpoints/endpointId
PATCH /endpoints/{endpointId}
Update an endpoint. Use this API reference to review authentication, request parameters, response fields, and errors for this Runpod operation.
# Create a new endpoint
Source: https://docs.runpod.io/api-reference/endpoints/POST/endpoints
POST /endpoints
Create a new endpoint. Use this API reference to review authentication, request parameters, response fields, and errors for this Runpod operation.
# Update an endpoint
Source: https://docs.runpod.io/api-reference/endpoints/POST/endpoints/endpointId/update
POST /endpoints/{endpointId}/update
Update an endpoint - synonym for PATCH /endpoints/{endpointId}. Review request parameters, response fields, and errors for this Runpod API operation.
# Delete a network volume
Source: https://docs.runpod.io/api-reference/network-volumes/DELETE/networkvolumes/networkVolumeId
DELETE /networkvolumes/{networkVolumeId}
Delete a network volume. Use this API reference to review authentication, request parameters, response fields, and errors for this Runpod operation.
# List network volumes
Source: https://docs.runpod.io/api-reference/network-volumes/GET/networkvolumes
GET /networkvolumes
Returns a list of network volumes. Review authentication, request parameters, response fields, and errors for this Runpod API operation.
# Find a network volume by ID
Source: https://docs.runpod.io/api-reference/network-volumes/GET/networkvolumes/networkVolumeId
GET /networkvolumes/{networkVolumeId}
Returns a single network volume. Review authentication, request parameters, response fields, and errors for this Runpod API operation.
# Update a network volume
Source: https://docs.runpod.io/api-reference/network-volumes/PATCH/networkvolumes/networkVolumeId
PATCH /networkvolumes/{networkVolumeId}
Update a network volume. Use this API reference to review authentication, request parameters, response fields, and errors for this Runpod operation.
# Create a new network volume
Source: https://docs.runpod.io/api-reference/network-volumes/POST/networkvolumes
POST /networkvolumes
Create a new network volume. Review authentication, request parameters, response fields, and errors for this Runpod API operation.
# Update a network volume
Source: https://docs.runpod.io/api-reference/network-volumes/POST/networkvolumes/networkVolumeId/update
POST /networkvolumes/{networkVolumeId}/update
Update a network volume - synonym for PATCH /networkvolumes/{networkVolumeId}. Review parameters and responses for this Runpod API operation.
# Overview
Source: https://docs.runpod.io/api-reference/overview
Use the Runpod API to programmatically manage your compute resources. Review authentication, request formats, and response details for the Runpod API.
The Runpod REST API v1 provides programmatic access to all Runpod compute resources. Integrate GPU infrastructure into your applications, workflows, and automation systems.
REST API v1 is deprecated and will be retired on **November 15, 2026**. Migrate your integrations to REST API v2 before that date. See the [migration guide](/api-reference-v2/migrate-from-v1) to get started. To migrate automatically, ask your coding agent to run `/runpod:migrate rest` once you've installed the [Runpod skills plugin](/get-started/agent-skills#migrate-an-existing-integration).
## Available resources
* **Pods**: Create and manage persistent GPU instances for development, training, and long-running workloads.
* **Serverless endpoints**: Deploy and scale containerized applications with autoscaling and job monitoring.
* **Network volumes**: Create persistent storage attachable to multiple resources.
* **Templates**: Save and reuse Pod and endpoint configurations.
* **Container registry auth**: Connect to private Docker registries.
* **Billing**: Access usage metrics and billing information.
## Authentication
All requests require a [Runpod API key](/get-started/api-keys) in the request headers. The API uses standard HTTP methods and returns JSON responses.
## OpenAPI schema
Retrieve the complete OpenAPI specification for client generation, request validation, or tooling integration.
See the [interactive API documentation endpoint reference](/api-reference/docs/GET/docs) to open the browser-based reference directly.
```bash cURL theme={"theme":{"light":"github-light","dark":"github-dark"}}
curl --request GET \
--url https://rest.runpod.io/v1/openapi.json \
```
```python Python theme={"theme":{"light":"github-light","dark":"github-dark"}}
import requests
url = "https://rest.runpod.io/v1/openapi.json"
response = requests.get(url)
print(response.json())
```
# Delete a Pod
Source: https://docs.runpod.io/api-reference/pods/DELETE/pods/podId
DELETE /pods/{podId}
Delete a Pod. Use this API reference to review authentication, request parameters, response fields, and errors for this Runpod operation.
# List Pods
Source: https://docs.runpod.io/api-reference/pods/GET/pods
GET /pods
Returns a list of Pods. Use this API reference to review authentication, request parameters, response fields, and errors for this Runpod operation.
# Find a Pod by ID
Source: https://docs.runpod.io/api-reference/pods/GET/pods/podId
GET /pods/{podId}
Returns a single Pod. Use this API reference to review authentication, request parameters, response fields, and errors for this Runpod operation.
# Update a Pod
Source: https://docs.runpod.io/api-reference/pods/PATCH/pods/podId
PATCH /pods/{podId}
Update a Pod, potentially triggering a reset. Review authentication, request parameters, response fields, and errors for this Runpod API operation.
# Create a new Pod
Source: https://docs.runpod.io/api-reference/pods/POST/pods
POST /pods
Creates a new [Pod](#/components/schemas/Pod) and optionally deploys it. Review parameters and responses for this Runpod API operation in detail.
# Reset a Pod
Source: https://docs.runpod.io/api-reference/pods/POST/pods/podId/reset
POST /pods/{podId}/reset
Reset a Pod. Use this API reference to review authentication, request parameters, response fields, and errors for this Runpod operation.
# Restart a pod
Source: https://docs.runpod.io/api-reference/pods/POST/pods/podId/restart
POST /pods/{podId}/restart
Restart a Pod. Use this API reference to review authentication, request parameters, response fields, and errors for this Runpod operation.
# Start or resume a Pod
Source: https://docs.runpod.io/api-reference/pods/POST/pods/podId/start
POST /pods/{podId}/start
Start or resume a Pod. Use this API reference to review authentication, request parameters, response fields, and errors for this Runpod operation.
# Stop a Pod
Source: https://docs.runpod.io/api-reference/pods/POST/pods/podId/stop
POST /pods/{podId}/stop
Stop a Pod. Use this API reference to review authentication, request parameters, response fields, and errors for this Runpod operation.
# Update a Pod
Source: https://docs.runpod.io/api-reference/pods/POST/pods/podId/update
POST /pods/{podId}/update
Update a Pod - synonym for PATCH /pods/{podId}. Review request parameters, response fields, and errors for this Runpod API operation.
# Delete a template
Source: https://docs.runpod.io/api-reference/templates/DELETE/templates/templateId
DELETE /templates/{templateId}
Delete a template. Use this API reference to review authentication, request parameters, response fields, and errors for this Runpod operation.
# List templates
Source: https://docs.runpod.io/api-reference/templates/GET/templates
GET /templates
Returns a list of templates. Review authentication, request parameters, response fields, and errors for this Runpod API operation.
# Find a template by ID
Source: https://docs.runpod.io/api-reference/templates/GET/templates/templateId
GET /templates/{templateId}
Returns a single template. Use this API reference to review authentication, request parameters, response fields, and errors for this Runpod operation.
# Update a template
Source: https://docs.runpod.io/api-reference/templates/PATCH/templates/templateId
PATCH /templates/{templateId}
Update a template. Use this API reference to review authentication, request parameters, response fields, and errors for this Runpod operation.
# Create a new template
Source: https://docs.runpod.io/api-reference/templates/POST/templates
POST /templates
Create a new template. Use this API reference to review authentication, request parameters, response fields, and errors for this Runpod operation.
# Update a template
Source: https://docs.runpod.io/api-reference/templates/POST/templates/templateId/update
POST /templates/{templateId}/update
Update a template - synonym for PATCH /templates/{templateId}. Review request parameters, response fields, and errors for this Runpod API operation.
# ComfyUI-to-API
Source: https://docs.runpod.io/community-solutions/comfyui-to-api/overview
Convert your ComfyUI workflows into deployment-ready GitHub repositories with automated Dockerfile generation
Access the tool at [comfy.getrunpod.io](https://comfy.getrunpod.io)
ComfyUI-to-API is a web-based tool that analyzes your ComfyUI workflows and automatically generates a Dockerfile and GitHub repository for deployment on Runpod Serverless. The tool uses a custom Comfy Agent to understand your workflow dependencies, searches for the right custom nodes and models, and creates a complete repository that you can deploy using [Runpod's GitHub integration](/serverless/workers/github-integration).
## What you'll learn
In this guide, you'll learn how to use ComfyUI-to-API to transform your ComfyUI workflows into Serverless workers. You'll upload a workflow file exported from ComfyUI, review the AI-powered analysis, generate a Dockerfile with all necessary dependencies, create a GitHub repository, and deploy it to Runpod Serverless using GitHub integration.
## Requirements
Before getting started, make sure you have:
* A ComfyUI workflow file exported via Comfy → File → Export (the full workflow export, not the API export).
* A GitHub account for repository creation.
* A Runpod account for Serverless deployment.
* Basic familiarity with ComfyUI workflows and custom nodes.
Make sure to export your workflow using **Comfy → File → Export** (not the API
export). The tool needs the complete workflow information to properly analyze
your setup.
## Step 1: Sign in with GitHub
Visit [comfy.getrunpod.io](https://comfy.getrunpod.io) to access ComfyUI-to-API. The tool requires you to sign in with your GitHub account so it can create a repository on your behalf in a later step.
Click **Sign in with GitHub** and authorize the application. Once authenticated, you'll be able to proceed with uploading your workflow.
## Step 2: Upload your workflow
Once you've accessed the tool, the first step is to upload your ComfyUI workflow file.
In ComfyUI, export your workflow using **Comfy → File → Export** (not the API export). This exports the complete workflow information needed for analysis. Then click the upload area or drag and drop your `workflow.json` file into the tool.
After uploading, you'll see a confirmation showing the filename and basic workflow information. The tool performs a quick validation to ensure your file is a valid ComfyUI workflow before proceeding to analysis.
## Step 3: Analyze your workflow
Click the **Analyze** button to run the custom Comfy Agent. You'll see progress information as the tool works through your workflow.
The analysis process detects custom nodes in your workflow, locates required models and their download URLs, resolves dependencies between nodes, and determines installation methods for each component. This typically takes 30-60 seconds depending on the complexity of your workflow.
The analysis results are displayed in expandable sections showing custom nodes found, models identified, and any potential issues or recommendations.
## Step 4: Review the generated Dockerfile
After analysis completes, the tool generates a Dockerfile based on your workflow. Review the Dockerfile to see what will be installed. The generated Dockerfile uses the [runpod-workers/worker-comfyui](https://github.com/runpod-workers/worker-comfyui) base image, which provides a clean ComfyUI installation without any models. The Dockerfile then adds installation commands for all your custom nodes, model download instructions with URLs, proper Python dependencies, and optimized caching layers for faster builds.
You can review and tweak the Dockerfile if needed, or skip ahead to the next step to create your GitHub repository.
Review the Dockerfile carefully to ensure all your custom nodes and models are
included. The AI is quite thorough, but you may want to add additional
customizations for your specific use case.
## Step 5: Create your repository
Now you can create a GitHub repository with all the necessary files for Serverless deployment. Since you authenticated with GitHub in step 1, the tool can create the repository on your behalf. Enter your desired repository name and optionally provide a description.
The tool creates a repository containing your Dockerfile and a README with setup and deployment instructions.
Click **Create GitHub repo** and watch the real-time progress as the tool creates your repository. Once complete, you'll see a success message with a direct link to your new GitHub repository.
## Step 6: Deploy to Runpod Serverless
With your GitHub repository created, you're ready to deploy to Runpod Serverless using the [GitHub integration](/serverless/workers/github-integration). This is where Runpod builds your Docker image from the Dockerfile and deploys it to an endpoint.
Navigate to [create a new endpoint](https://console.runpod.io/serverless/new-endpoint) in the Runpod console. Under "Import Git Repository", select your newly created repository from the dropdown menu. You'll need to connect your GitHub account to Runpod once if you haven't already.
Configure your deployment settings including which branch to deploy from (typically `main`), the path to your Dockerfile (typically in the root), and your GPU requirements based on your workflow's needs.
For detailed instructions on deploying from GitHub, including how to authorize
Runpod with your GitHub account and configure deployment options, see [Deploy
workers from GitHub](/serverless/workers/github-integration).
After clicking "Deploy Endpoint", Runpod builds the Docker image from your repository and deploys it to your endpoint automatically. You can monitor the build status in the "Builds" tab of your endpoint detail page. Once the build is complete, you can use the **example API request** provided in step 5 of the tool to test your endpoint.
## Understanding the repository files
The generated repository includes the essential files needed to build your Serverless worker.
The `Dockerfile` contains all the instructions to build your worker image. It starts from the [runpod-workers/worker-comfyui](https://github.com/runpod-workers/worker-comfyui) base image, which provides a clean ComfyUI installation along with all the handler code needed to run as a Serverless endpoint. Your Dockerfile then adds your custom nodes and downloads your required models. When you deploy via [GitHub integration](/serverless/workers/github-integration), Runpod uses this Dockerfile to build your Docker image.
The `README.md` provides documentation about your workflow and deployment instructions.
You can customize the Dockerfile after creation. For example, you might want
to add additional dependencies, adjust model download locations, or optimize
the build process.
## Updating your deployed worker
When you make changes to your repository, such as updating the Dockerfile to add new models or custom nodes, Runpod automatically detects these changes.
The [GitHub integration](/serverless/workers/github-integration) monitors your `main` branch for changes. When you push commits to `main`, Runpod automatically rebuilds the Docker image and updates your endpoint with the new version. You can monitor the rebuild progress in the "Builds" tab of your endpoint detail page.
For more details on managing deployments and monitoring builds, see [Deploy workers from GitHub](/serverless/workers/github-integration).
## Troubleshooting
**Problem:** The tool shows an error or doesn't detect my workflow properly.
**Solution:**
1. Make sure you exported using **Comfy → File → Export** (not the API
export).
2) The file should be named `workflow.json` and contain the complete
workflow structure.
3. If you used the API export, go back to ComfyUI and use File → Export
instead.
**Problem:** The generated Dockerfile doesn't include all your custom nodes
or models.
**Solution:**
1. Review the analysis results to see what was detected.
2) Some very new or obscure custom nodes may not be found automatically.
3. You can manually edit the Dockerfile in your GitHub repository to add
missing components.
4) Check the ComfyUI Registry to find installation instructions for missing
nodes.
## Related documentation
* [Deploy workers from GitHub](/serverless/workers/github-integration)
* [Handler functions](/serverless/workers/handler-functions)
* [Creating a Dockerfile](/serverless/workers/deploy#creating-a-dockerfile)
* [Serverless overview](/serverless/overview)
* [Managing endpoints](/serverless/endpoints/overview)
* [Generate images with ComfyUI on Pods](/tutorials/pods/comfyui): Deploy ComfyUI on a Pod to develop and test workflows interactively.
* [Deploy ComfyUI on Serverless](/tutorials/serverless/comfyui): Deploy a Serverless endpoint running ComfyUI and generate images using workflow JSON.
# CopyParty file manager
Source: https://docs.runpod.io/community-solutions/copyparty-file-manager/overview
Web-based GUI for easy file browsing, uploading, downloading, and media viewing on Runpod. Review setup and usage guidance for this Runpod community solution.
## Set up CopyParty on Runpod
CopyParty provides a webbased GUI that makes file management simple on Runpod instances. With its intuitive interface, you can browse directories, upload/download files, preview images and videos, and manage your Pod's filesystem without complex command-line operations.
## About CopyParty
CopyParty is an open-source project created that provides a portable file server with a web UI. It's perfect for managing files on cloud GPU instances where traditional file transfer methods might be cumbersome.
For a video demonstration, you can watch the [creator's YouTube tutorial](https://www.youtube.com/watch?v=15_-hgsX2V0).
Check the repository for additional features, updates, and documentation: [github.com/9001/copyparty](https://github.com/9001/copyparty)
## Requirements
To use CopyParty on Runpod, you need:
* **Terminal access to your Pod** - Either through web terminal or Jupyter Labs terminal.
* An available HTTP port on your Pod.
* A supported OS in the Docker base image.
CopyParty has been tested on Ubuntu-based Docker images and Runpod Official templates. When using custom templates or alternative Docker bases (like Python Slim) you may encounter file system dependency errors. Refer to the [GitHub repository](https://github.com/9001/copyparty) for details on OS support.
### Verifying terminal access
You can access the terminal in two ways:
#### Option 1: Web terminal
If you see this option when clicking "Connect" on your Pod page, you have web terminal access:
#### Option 2: JupyterLab terminal
If you have JupyterLab installed on your Pod, you can access the terminal there:
## Installation steps
### Step 1: Access your Pod settings
Navigate to your Pod page and locate the settings:
### Step 2: Add an HTTP port
**Adding a new port will restart your Pod and erase any data not stored in `/workspace`**
Before proceeding, ensure all important files are saved in `/workspace` or backed up elsewhere. Any installed libraries or files outside of `/workspace` will be lost.
Add a dedicated HTTP port for the CopyParty interface. If port 8888 is already in use (common for Jupyter), try port 8000 or another available port.
### Step 3: Install and run CopyParty
Open your terminal (web terminal or Jupyter terminal) and run one of the following commands:
#### Option 1: Standard installation
Run CopyParty directly (the session will end if you close the terminal):
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
curl -LsSf https://astral.sh/uv/install.sh | sh && source $HOME/.local/bin/env && uv tool run copyparty -p 8000 --allow-csrf
```
Replace `-p 8000` with your chosen port number if different.
#### Option 2: Background installation with tmux
To keep CopyParty running even after closing the terminal, use `tmux`:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
apt-get update && apt-get install tmux -y && tmux new-session -d -s copyparty 'curl -LsSf https://astral.sh/uv/install.sh | sh && source $HOME/.local/bin/env && uv tool run copyparty -p 8000 --allow-csrf' && tmux attach -t copyparty
```
**What is tmux?**
`tmux` (terminal multiplexer) is a tool that lets you run terminal sessions in the background. Think of it as a way to keep programs running even after you close your terminal window, like minimizing an app instead of closing it. This is particularly useful on Runpod where you want CopyParty to keep running even if you disconnect.
For a more in-depth tmux tutorial, check out this [comprehensive video guide](https://www.youtube.com/watch?v=nTqu6w2wc68).
This command:
1. Installs tmux (a terminal multiplexer)
2. Creates a new tmux session named "copyparty"
3. Runs CopyParty in the background
4. Attaches you to the session to see the output
**Quick tmux Commands**
* To detach from tmux and leave CopyParty running: Press `Ctrl+B` then `D`
* To reattach to the session later: `tmux attach -t copyparty`
* To stop CopyParty: Reattach and press `Ctrl+C`
### Step 4: Access the CopyParty interface
Once CopyParty is running, click on the port number in your Runpod dashboard:
## Using CopyParty
### File navigation
The interface displays your file system on the left side:
### Uploading files
Simply drag and drop files into the interface to upload them:
### Downloading files
To download files:
1. Click on files to select them (they'll be highlighted in pink)
2. Use the buttons in the bottom right:
* **"dl"** - Download individual files
* **"zip"** - Download multiple files as a zip archive
## Tips and best practices
1. **Data Persistence**: Always store important files in `/workspace` to survive pod restarts
2. **Port Selection**: Choose a port that doesn't conflict with other services (avoid 8888 if using Jupyter)
3. **Large Files**: CopyParty handles large file transfers well, making it ideal for model weights and datasets
## Troubleshooting
### Session ends when terminal closes
Use the tmux option (Option 2) to keep CopyParty running in the background
## Alternative file transfer methods
While CopyParty provides an excellent web-based solution, Runpod also supports:
* Direct SSH/SCP transfers (if SSH is enabled)
* JupyterLab file browser
* [Runpod CLI](/runpodctl/overview) tool
* [Cloud storage integration](/pods/storage/cloud-sync) (S3, Google Drive, etc.)
Choose the method that best fits your workflow and security requirements.
# OhMyRunpod
Source: https://docs.runpod.io/community-solutions/ohmyrunpod/overview
Community solution for easy (SFTP) setup on Runpod. Review installation, configuration, and usage guidance for this community solution on Runpod.
## What is OhMyRunpod?
OhMyRunpod is a Python package that solves one of the most common Runpod user challenges: **easily transferring files to and from your Pod**.
While OhMyRunpod includes various features, **the SFTP setup functionality** is the one we want to highlight to help get you started with file transfer software such as FileZilla or similar tools.
Check the repository for additional features, updates, and documentation: [github.com/kodxana/OhMyRunpod-python](https://github.com/kodxana/OhMyRunpod-python)
## Key features
Simplified file transfer between your local machine and Runpod instances using SFTP or Croc
Automatically configures SSH access with secure key generation and password management
## Requirements
Before getting started with OhMyRunpod, make sure you have:
* An active Runpod account with a running Pod.
* Python 3.6 or higher installed on your local machine.
* Port 22 exposed in your Pod's TCP ports configuration (for SFTP transfers).
## Installation
Install OhMyRunpod using pip:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
pip install OhMyRunpod
```
## Quick start: file transfer
The most common use case for OhMyRunpod is transferring files between your local machine and Runpod instances.
Before starting, make sure port 22 is exposed in your Pod. This is required for SFTP transfers.
To expose port 22:
1. Go to your Pod in the Runpod console
2. Click **Edit Pod**
3. Under the **TCP Ports** section, add port `22`
4. Save the changes
**Data loss warning**: Changing TCP ports will restart your pod and **erase all data outside of `/workspace`**. Ensure all important files are in `/workspace` (the default persistent directory) before making port changes.
Execute the file transfer command:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
OhMyRunpod --file-transfer
```
You'll be presented with transfer options:
* **Croc**: Easy peer-to-peer file transfer (recommended for quick transfers)
* **SFTP**: Traditional file transfer via SSH/SFTP clients (recommended for persistent connections)
Use arrow keys to navigate and Enter to select.
If you choose SFTP, OhMyRunpod will automatically:
* Install and configure SSH server
* Set up SSH keys
* Generate a secure password
* Create connection scripts for your operating system
The tool provides:
* Complete connection information (server address, port, username, password)
* Platform-specific connection instructions for FileZilla, WinSCP, Command Line, and VS Code
* Password saved to `/workspace/root_password.txt` for reference
## What can you do after setup?
Once SFTP is configured, you can:
* **Upload datasets**: Transfer training data directly to `/workspace/`
* **Download models**: Pull trained checkpoints back to your local machine
* **Sync code**: Use VS Code Remote-SSH for live editing
* **Backup files**: Regular backups of important work
* **Use any SFTP client**: FileZilla, WinSCP, or command line tools
## Troubleshooting
**Problem:** Cannot connect via SFTP, getting "Connection refused" error
**Solution:**
1. Ensure port 22 is exposed in your Pod (Pod > Edit Pod > TCP Ports > Add 22)
2. Restart your Pod after adding the port
3. Re-run `OhMyRunpod --file-transfer` to reconfigure SSH
**Problem:** Getting "Permission denied" when trying to connect via SFTP
**Solution:**
1. Check that you're using the correct password from `/workspace/root_password.txt`
2. Ensure you're using username `root`
3. Verify the port number matches what's shown in the connection details
**Problem:** SFTP client cannot resolve the host address
**Solution:**
1. Verify your Pod is running and not stopped
2. Check that you're using the correct IP address and port from the connection details
3. Ensure your network allows outbound connections to the Runpod IP range
## Tips
* The password is automatically saved to `/workspace/root_password.txt` for your reference.
* For large file transfers, use a dedicated SFTP client like FileZilla or WinSCP instead of the command line.
## More community solutions
Have a tool that solves common Runpod problems? Community contributions are welcome! Share your solutions in the [Runpod Discord](https://discord.com/invite/runpod) community.
## Related Runpod documentation
* [SSH Configuration Guide](/pods/configuration/use-ssh)
* [Managing Pods](/pods/manage-pods)
* [Connecting to Pods](/pods/connect-to-a-pod)
# Community solutions
Source: https://docs.runpod.io/community-solutions/overview
Community-created tools and solutions by and for Runpod users. Review setup and usage guidance for this Runpod community solution.
This section showcases innovative tools and solutions created by the Runpod community. These are tools and workflows developed by users who encountered common challenges and built creative solutions to share with others.
While these tools and solutions are not officially maintained by Runpod, we highlight them here because they demonstrate the ingenuity of our community and can be valuable resources for users facing similar challenges.
## Using community solutions
Community tools and solutions are provided as-is and maintained by their creators. While we showcase them for their utility, please note that you should test in a development environment first, review the source code when available, report issues to the tool creators, and ultimately use them at your own discretion.
## Featured tools & solutions
Explore these community-created tools that can enhance your Runpod workflow:
Web-based tool that analyzes ComfyUI workflows and automatically generates Dockerfiles and GitHub repositories for deployment on Runpod Serverless.
Python package for easy SFTP setup and file transfers. Simplifies connecting to your Pods with file transfer software like FileZilla.
Bash and Python scripts for migrating data between Runpod instances when you need to move Pods or backup data.
Web-based GUI for easy file browsing, uploading, downloading, and media viewing on Runpod without complex command-line operations.
Browse through our community tools section in the sidebar for detailed documentation on each tool's purpose, installation, and usage.
## Contributing
Built something useful on Runpod? The community would love to learn about it! Share your tools and get feedback in the [Runpod Discord](https://discord.com/invite/runpod).
# Network volume storage tool
Source: https://docs.runpod.io/community-solutions/runpod-network-volume-storage-tool
A command-line tool for managing Runpod network storage volumes and files. Review setup and usage guidance for this Runpod community solution.
GitHub repository: [github.com/justinwlin/Runpod-Network-Volume-Storage-Tool](https://github.com/justinwlin/Runpod-Network-Volume-Storage-Tool)
Runpod provides an [S3-compatible layer](/storage/s3-api) for network volumes, enabling object storage operations on your network storage. This community tool makes it easy to interact with that S3 layer through three interfaces: a command-line interface, a Python SDK for programmatic access, and a self-hosted REST API server for integration with other applications.
## Requirements
* Python 3.8 or higher.
* Runpod API key from [console settings](https://console.runpod.io/user/settings).
* S3 API keys (access key and secret key) for file operations.
## Installation
Clone the repository and install dependencies:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
git clone https://github.com/justinwlin/Runpod-Network-Volume-Storage-Tool.git
cd Runpod-Network-Volume-Storage-Tool
# Install dependencies with uv
uv sync
```
## Configuration
Set your API credentials as environment variables:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
export RUNPOD_API_KEY="YOUR_RUNPOD_API_KEY"
export RUNPOD_S3_ACCESS_KEY="YOUR_S3_ACCESS_KEY"
export RUNPOD_S3_SECRET_KEY="YOUR_S3_SECRET_KEY"
```
## Interactive mode
The interactive mode provides a menu-driven interface:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
uv run runpod-storage interactive
```
Features include volume management, file upload/download, and an interactive file browser with navigation and selection modes.
## Command line usage
Manage volumes directly from the command line:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
# List volumes
uv run runpod-storage list-volumes
# Create a volume
uv run runpod-storage create-volume --name "my-storage" --size 50 --datacenter EU-RO-1
# Upload files
uv run runpod-storage upload /path/to/file.txt volume-id
uv run runpod-storage upload /path/to/directory volume-id
# Download files
uv run runpod-storage download volume-id remote/file.txt
```
## Python SDK
The SDK provides programmatic access to all features:
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
from runpod_storage import RunpodStorageAPI
api = RunpodStorageAPI()
# List volumes
volumes = api.list_volumes()
# Create volume
volume = api.create_volume(
name="ml-datasets",
size=100,
datacenter="EU-RO-1"
)
# Upload with automatic chunk size optimization
api.upload_file("data.csv", volume_id, "datasets/data.csv")
# Upload directory with progress tracking
def progress_callback(current, total, filename):
percent = (current / total) * 100
print(f"[{current}/{total}] {percent:.1f}% - Uploading: {filename}")
api.upload_directory(
"my_project/",
volume_id,
"projects/my_project/",
progress_callback=progress_callback
)
```
## API server
Run a REST API server that proxies to Runpod's API:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
uv run runpod-storage-server --host 0.0.0.0 --port 8000
```
# SSH password setup & migration tools
Source: https://docs.runpod.io/community-solutions/ssh-password-migration/overview
Simple tools for migrating data between Runpod instances. Review installation, configuration, and usage guidance for this community solution on Runpod.
## What are these tools?
These are simple bash and Python scripts that solve a critical problem: **migrating data between Runpod instances when you need to move Pods** (e.g., when your Pod gets stuck with zero GPUs or you need to switch to a different instance).
Check the repository for additional features, updates, and documentation: [github.com/justinwlin/Runpod-SSH-Password](https://github.com/justinwlin/Runpod-SSH-Password)
## The problem it solves
When Runpod users encounter issues like:
* Pod stuck with **zero GPUs allocated**.
* Need to **migrate to a different GPU type**.
* Want to **transfer data between Pods**.
* Need to **backup data before terminating a Pod**.
This tool provides a simple solution for transferring your files to a new Pod.
## Requirements
Before using these tools, make sure you have:
1. **Terminal access** on your Runpod instances:
* Web Terminal in Runpod's interface, or
* Terminal in Jupyter Notebook/Lab
2. **Port 22 exposed** on destination Pod / source Pod:
* Go to: **Pod > Edit Pod > TCP Ports**
* Add port `22` if not present
**Important**: Changing TCP ports will restart your Pod and **erase all data outside of `/workspace`**. Make sure all important files are stored in `/workspace` (the default persistent directory) before making port changes.
3. **Linux-based template**: Works on Ubuntu/Debian images (most Runpod templates)
## Quick migration workflow
On your **destination Pod** (where you're migrating to):
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
wget https://raw.githubusercontent.com/justinwlin/Runpod-SSH-Password/main/passwordrunpod.sh && chmod +x passwordrunpod.sh && ./passwordrunpod.sh
```
The script will prompt you to set a password. After setup, it displays your connection details:
```
You can now connect using: ssh root@213.173.105.86 -p 17958
Password: [the password you just set]
```
**Save these details** - you'll need them in Step 4.
On your **source Pod** (where your files currently are):
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
wget https://raw.githubusercontent.com/justinwlin/Runpod-SSH-Password/refs/heads/main/SCPMigration -O scp_migration.py && python3 scp_migration.py
```
The tool provides an interactive interface:
**Navigation Mode** - Browse directories:
```
Current directory: /workspace
Navigate by entering a number:
1. models/
2. data/
3. ComfyUI/
Navigate to: 1 (enters models folder)
Navigate to: s (switch to selection mode)
```
**Selection Mode** - Choose what to transfer:
```
Select item: 1 (selects folder/file)
```
Enter the SSH details from Step 1:
```
Enter SSH command: ssh root@213.173.105.86 -p 17958
Enter password: [the password you set in Step 1]
```
The tool will:
* **Files**: Transfer directly via SCP
* **Folders**: Compress to ZIP, transfer, then extract automatically
## Key features
* **Interactive navigation**: Browse files with simple number selection
* **Smart compression**: Automatically zips folders for reliable transfer
* **Mode switching**: Press `s` for selection, `n` for navigation
* **Clean transfer**: Auto-extracts and cleans up on destination
## Related resources
* [Managing Pods](/pods/manage-pods)
* [Pod storage overview](/pods/storage/types)
* [Network volumes](/storage/network-volumes)
# Containers
Source: https://docs.runpod.io/containers
Explore Runpod container tutorials for Dockerfiles, persistence, common Docker commands, application packaging, and Serverless deployment.
## 📄️ Overview
Learn how to build and deploy applications on the Runpod platform with this set of tutorials, covering tools, technologies, and deployment methods, including Containers, Docker, and Serverless implementation.
## 📄️ Intro to containers
Discover the world of containerization with Docker, a platform for isolated environments that package applications, frameworks, and libraries into self-contained containers for consistent and reliable deployment across diverse computing environments.
## 📄️ Dockerfile
Learn how to create a Dockerfile to customize a Docker image and use an entrypoint script to run a command when the container starts, making it a reusable and executable unit for deploying and sharing applications.
## 📄️ Persist data outside of containers
Learn how to persist data outside of containers by creating named volumes, mounting volumes to data directories, and accessing persisted data from multiple container runs and removals in Docker.
## 📄️ Docker commands
Runpod enables BYOC development with Docker, providing a reference sheet for commonly used Docker commands, including login, images, containers, Dockerfile, volumes, network, and execute.
# Fine-tune a model
Source: https://docs.runpod.io/fine-tune
Learn how to fine-tune a large language model on Runpod using Axolotl. Review setup and usage guidance in the Runpod documentation.
Fine-tuning is the process of taking a pre-trained large language model (LLM) and further training it on a smaller, specific dataset. This process adapts the model to a particular task or domain, improving its performance and accuracy for your use case.
This guide explains how to use Runpod's fine-tuning feature, powered by [Axolotl](https://github.com/axolotl-ai-cloud/axolotl), to customize an LLM. You'll learn how to select a base model, choose a dataset, configure your training environment, and deploy your fine-tuned model.
For more information about fine-tuning with Axolotl, see the [Axolotl Documentation](https://github.com/axolotl-ai-cloud/axolotl).
## Requirements
Before you begin, you'll need:
* A Runpod account.
* (Optional) A [Hugging Face](https://huggingface.co/) account and an access token if you plan to use gated models or upload your fine-tuned model.
## Select a base model and dataset
The base model is the starting point for your fine-tuning process, while the dataset provides the specific knowledge needed to adapt the base model to your task.
You can choose from thousands of models and datasets on [Hugging Face](https://huggingface.co/models).
## Deploy a fine-tuning Pod
Navigate to the [Fine-Tuning](https://console.runpod.io/fine-tuning) section in the Runpod console.
In the **Base Model** field, enter the Hugging Face model ID. In the **Dataset** field, enter the Hugging Face dataset ID.
If this is your first time fine-tuning and you're just experimenting, try:
```sh theme={"theme":{"light":"github-light","dark":"github-dark"}}
# Base model
TinyLlama/TinyLlama_v1.1
# Dataset (alpaca)
mhenrichsen/alpaca_2k_test
```
If you're using a gated model that requires special access, generate a Hugging Face token with the necessary permissions and add it to the **Hugging Face Access Token** field.
Click **Deploy the Fine-Tuning Pod** to start configuring your fine-tuning Pod.
Select a GPU instance based on your model's requirements. Larger models and datasets require GPUs with more memory.
Finishing configuring the Pod, then click **Deploy on-demand**. This should open the detail pane for your Pod automatically.
Click **Logs** to monitor the system logs for deployment progress. Wait for the success message: `"You've successfully configured your training environment!"` Depending on the size of your model and dataset, this may take some time.
Once your training environment is ready, you can connect to it to configure and start the fine-tuning process.
Click **Connect** and choose your preferred connection method:
* **Jupyter Notebook**: A browser-based notebook interface.
* **Web Terminal**: A browser-based terminal.
* **SSH**: A secure connection from your local machine.
To use SSH, add your public SSH key in your account settings. The system automatically adds your key to the Pod's `authorized_keys` file. For more information, see [Connect to a Pod with SSH](/pods/configuration/use-ssh).
## Configure your environment
For a list of working configuration examples, check out the [Axolotl examples repository](https://github.com/axolotl-ai-cloud/axolotl/tree/main/examples) (also available in your training environment at `/workspace/fine-tuning/examples/`).
Your training environment is located in the `/workspace/fine-tuning/` directory and has the following structure:
`/examples/` contains sample configurations and scripts, `/outputs/` contains your training results and model outputs, and `/config.yaml/` is the main configuration file for your training parameters.
The system generates an initial `config.yaml` based on your selected base model and dataset. This is where you define all the hyperparameters for your fine-tuning job. You may need to experiment with these settings to achieve the best results.
Navigate to the fine-tuning directory (`/workspace/fine-tuning/`) and open the configuration file (`config.yaml`) in JupyterLab or your preferred text editor to review and adjust the fine-tuning parameters.
If you're using the web terminal, the fine-tuning directory should open automatically. Use `nano` to edit the `config.yaml` file:
```sh theme={"theme":{"light":"github-light","dark":"github-dark"}}
nano config.yaml
```
The `config.yaml` file will look something like this (`base_model` and `datasets` will be replaced with the model and dataset you selected in Step 2):
```yaml theme={"theme":{"light":"github-light","dark":"github-dark"}}
adapter: lora
base_model: TinyLlama/TinyLlama_v1.1
bf16: auto
datasets:
- path: mhenrichsen/alpaca_2k_test
type: null
gradient_accumulation_steps: 1
learning_rate: 0.0002
load_in_8bit: true
lora_alpha: 16
lora_dropout: 0.05
lora_r: 8
lora_target_modules:
- q_proj
- v_proj
- k_proj
- o_proj
- gate_proj
- down_proj
- up_proj
micro_batch_size: 16
num_epochs: 1
optimizer: adamw_bnb_8bit
output_dir: ./outputs/mymodel
sequence_len: 4096
train_on_inputs: false
```
Here's a breakdown of the `config.yaml` file:
Model and precision:
* **`base_model`**: The base model you want to fine-tune.
* **`bf16: auto`**: This tells the GPU to use **Bfloat16** precision if it can. It’s more stable than standard FP16 and helps prevent the model's math from "overflowing" (exploding) during training.
* **`load_in_8bit: true`**: This is a memory-saving trick. It squashes the base model weights into 8 bits so it takes up less VRAM, allowing you to train on smaller GPUs.
LoRA settings:
* **`lora_r: 8`**: The rank of the LoRA adapter. 8 is a standard starting point; higher numbers (like 16 or 32) let the model learn more complex patterns but use more VRAM.
* **`lora_alpha: 16`**: This scales the learned weights.
* **`lora_target_modules`**: This list tells Axolotl exactly which parts of the Transformer architecture to attach the adapters to.
Dataset logic
* **`path`**: Where the data is coming from (Hugging Face).
* **`type: null`**: This tells Axolotl how to format the text into prompts.
You'll need to change this value depending on the dataset you selected—see the next step for details.
* **`train_on_inputs: false`**: This is a smart setting. It tells the model: *"Don't try to predict the user's question; only learn how to predict the assistant's answer."* This focuses the "learning energy" on the responses.
* **`sequence_len: 4096`**: The maximum length of text the model can "read" at once.
Training mechanics
* **`micro_batch_size: 16`**: How many examples the GPU processes at a single time.
* **`gradient_accumulation_steps: 1`**: How many batches to "save up" before actually updating the model's weights.
* **`learning_rate: 0.0002`**: How fast the model changes. Too high and it "forgets" everything; too low and it never learns.
* **`optimizer: adamw_bnb_8bit`**: A special version of the AdamW optimizer that uses 8-bit math to save even more VRAM.
The dataset type is set to `null` by default. You'll need to change this value depending on the dataset you selected. For example, if you selected the `mhenrichsen/alpaca_2k_test` dataset, you'll need to change `type: null` to `type: alpaca` to load the dataset correctly.
Once you've changed the dataset type, save the file (`config.yaml`) and continue to the next step.
If you're not sure what dataset type to use, you can find an overview of common dataset types below:
`chat_template` for chat-based datasets:
```json theme={"theme":{"light":"github-light","dark":"github-dark"}}
{
"messages" : [
{"role": "user", "content": "What is the capital of France?"},
{"role": "assistant", "content": "The capital of France is Paris."}
]
}
```
You'll also need to add the `field_messages` key to `datasets` to specify the field that contains the messages:
```yaml theme={"theme":{"light":"github-light","dark":"github-dark"}}
datasets:
- path: your/dataset
type: chat_template
field_messages: messages
```
`completion` for raw text datasets:
```json theme={"theme":{"light":"github-light","dark":"github-dark"}}
{
"text": "The quick brown fox jumps over the lazy dog."
}
```
`input_output` for template-free datasets:
```json theme={"theme":{"light":"github-light","dark":"github-dark"}}
{
"input": "User: What is the capital of France?\nAssistant: ",
"output": "The capital is Paris."
}
```
`alpaca` for instruction-following datasets:
```json theme={"theme":{"light":"github-light","dark":"github-dark"}}
{
"instruction": "Summarize the following text.",
"input": "The sun is a star at the center of the Solar System.",
"output": "The sun is the central star of our solar system."
}
```
`sharegpt` for conversational datasets:
```json theme={"theme":{"light":"github-light","dark":"github-dark"}}
{
"conversations": [
{
"from": "human",
"value": "What are the three laws of thermodynamics?"
},
{
"from": "gpt",
"value": "1. Energy cannot be created or destroyed. 2. Entropy always increases. 3. Absolute zero cannot be reached."
}
]
}
```
You'll also need to add the `conversation` key to `datasets` to specify the name of the list field that contains the messages:
```yaml theme={"theme":{"light":"github-light","dark":"github-dark"}}
datasets:
- path: your/dataset
type: sharegpt
conversation: conversations
```
## Start the fine-tuning process
Once you're satisfied with your configuration, you can start the training.
Run the following command in your terminal:
```sh theme={"theme":{"light":"github-light","dark":"github-dark"}}
axolotl train config.yaml
```
Monitor the training progress in your terminal. The output will show the training loss, validation loss, and other metrics.
## Test your model with vLLM
Once the fine-tuning process is complete, you can test the capabilities of your fine-tuned model with vLLM.
For example, to serve the fine-tuned TinyLlama model used in the examples above, you would follow these steps:
To serve your fine-tuned model, run the following command:
```sh theme={"theme":{"light":"github-light","dark":"github-dark"}}
vllm serve TinyLlama/TinyLlama_v1.1 --enable-lora --lora-modules my-adapter=/workspace/fine-tuning/outputs/mymodel --port 8000
```
To test your model, first you'll need to start a new terminal window, tab, or pane.
If you're using the [web terminal](/pods/connect-to-a-pod#web-terminal-connection), `tmux` is already installed, and you can create a new horizontal pane by running:
```sh theme={"theme":{"light":"github-light","dark":"github-dark"}}
tmux split-window -h
```
In the new window/tab/pane, you can send a test request to the vLLM server using `curl`:
```sh theme={"theme":{"light":"github-light","dark":"github-dark"}}
curl http://localhost:8000/v1/completions \
-H "Content-Type: application/json" \
-d '{
"model": "my-adapter",
"prompt": "### Instruction:\nExplain gravity in one sentence.\n\n### Response:\n",
"max_tokens": 50
}'
```
You should see the response from your model in the terminal.
## Push your model to Hugging Face
After the fine-tuning process is complete, you can upload your model to the Hugging Face Hub to share it with the community or use it in your applications.
Run this command to log in to your Hugging Face account:
```sh theme={"theme":{"light":"github-light","dark":"github-dark"}}
huggingface-cli login
```
To upload your model files to the Hugging Face Hub, run this command:
```sh theme={"theme":{"light":"github-light","dark":"github-dark"}}
huggingface-cli upload YOUR_USERNAME/MODEL_NAME ./outputs/mymodel
```
Replace `YOUR_USERNAME` with your Hugging Face username and `MODEL_NAME` with your desired model name.
## Next steps
Now that you've successfully fine-tuned a model, you can deploy it for inference using [Runpod Serverless](/serverless/overview). If you've uploaded your model to Hugging Face, you can deploy it as a [cached model](/serverless/endpoints/model-caching) to reduce cost and cold start times.
# Apps and environments
Source: https://docs.runpod.io/flash/apps/apps-and-environments
Understanding Flash's two-level deployment structure for organizing projects and managing deployments.
Flash uses a two-level organizational structure for deployments: **apps** and **environments**. Understanding this structure helps you organize projects, manage multiple deployment stages, and isolate resources effectively.
## Flash apps
A **Flash app** is a namespace on Runpod's backend that groups all resources for a single project. The app itself is just metadata—actual cloud resources (endpoints, volumes) are created when you deploy to an environment.
An app consists of:
* **App registry entry**: Metadata in Runpod's system (namespace, project identifier).
* **Environments**: Different deployment stages (dev, staging, production).
* **Builds**: Versioned tarball artifacts containing your code and dependencies.
* **Serverless endpoints**: Running infrastructure created per environment.
Apps are created automatically when you first run `flash deploy`, or explicitly with `flash app create `.
### App hierarchy
## Environments
An **environment** is an isolated deployment stage within a Flash app (e.g., `dev`, `staging`, `production`). Each environment has its own endpoints, build version, volumes, and deployment state. Environments are completely independent—deploying to `dev` has no effect on `production`.
An environment contains:
* **Deployed endpoints**: Serverless workers for your `@Endpoint` functions.
* **Build version**: The specific code version running in this environment.
* **Volumes**: Persistent storage attached to workers.
* **State**: Current deployment status (deploying, deployed, failed).
Environments are created automatically when you deploy with `--env ` or explicitly with `flash env create `.
### Environment states
Environments can be in several states:
| State | Description |
| ----------- | -------------------------------------------------------------------------------------------- |
| `deploying` | Deployment in progress (building artifacts, provisioning endpoints) |
| `deployed` | Successfully deployed and running |
| `failed` | Deployment failed (check logs in the [Runpod console](https://console.runpod.io/serverless)) |
| `updating` | Configuration update in progress |
## Builds and deployments
When you run `flash deploy`, Flash creates and uploads a build artifact, then provisions endpoints:
### Build process
1. **Create tarball**: Flash packages your code into `.flash/artifact.tar.gz` containing:
* Worker Python files (`lb_worker.py`, `gpu_worker.py`, `cpu_worker.py`).
* Pre-installed dependencies (bundled during build).
* Deployment manifest (`flash_manifest.json`).
* Auto-generated handler code.
2. **Upload artifact**: The tarball is uploaded to Runpod's storage and associated with your app as a versioned "build".
3. **Provision endpoints**: For each resource configuration, Flash creates a Serverless endpoint that:
* Runs on pre-built Flash Docker images (`runpod/flash:latest` or `runpod/flash-cpu:latest`).
* Extracts your tarball and executes your code.
* Scales automatically based on load.
4. **Activate environment**: The environment is linked to the build and endpoints.
You're **not** building custom Docker images. Flash uses pre-built images that extract your tarball and run your code. This is why deployments are fast (no image build step) and limited to 1.5 GB (code and dependencies only).
## Common environment patterns
### Single environment (simple projects)
For small projects or solo development, use a single environment:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
flash deploy --env production
# If using uv:
uv run flash deploy --env production
```
All deployments go to `production`. Simple, but no testing isolation.
### Multiple environments (team projects)
For team projects, use separate environments for development, testing, and production:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
flash deploy --env dev # Development and testing
flash deploy --env staging # QA and pre-production validation
flash deploy --env production # Live user-facing deployment
# If using uv:
uv run flash deploy --env dev
uv run flash deploy --env staging
uv run flash deploy --env production
```
Each environment is completely isolated. Deploy to `dev` for testing, `staging` for QA approval, then `production` for users.
## Managing apps and environments
Use the CLI to manage your apps and environments:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
# Apps
flash app list # List all apps
flash app get # View app details
flash app delete # Delete app and all environments
# Environments
flash env list # List environments in current app
flash env get # View environment details
flash env delete # Delete specific environment
# If using uv:
uv run flash app list
uv run flash env list
```
Deleting an app or environment is irreversible. All endpoints and configuration are permanently removed.
# Build a Flash app
Source: https://docs.runpod.io/flash/apps/build-app
Create a Flash app, test it locally, and deploy it to production. Review setup, configuration, deployment, and usage guidance for Runpod Flash.
Flash apps let you build APIs to serve AI/ML workloads on Runpod Serverless. This guide walks you through the process of building a Flash app from scratch, from project initialization and local testing to production deployment.
If you haven't already, we recommend starting with the [Quickstart](/flash/quickstart) guide to get a feel for how Flash `@Endpoint` functions work.
## Requirements:
* You've [created a Runpod account](/accounts-billing/manage-accounts).
* You've [created a Runpod API key](/get-started/api-keys).
* You've installed [Python 3.10, 3.11, 3.12, or 3.13](https://www.python.org/downloads/).
## Step 1: Initialize a new project
Create a new directory and install Flash using [uv](https://docs.astral.sh/uv/):
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
# Create the project directory and navigate into it:
mkdir flash_app
cd flash_app
# Install Flash:
uv venv
source .venv/bin/activate
uv pip install runpod-flash
```
Use the `flash init` command to generate a structured project template with a preconfigured application entry point:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
uv run flash init .
```
Authenticate with Runpod:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
uv run flash login
```
This opens your browser to authorize Flash. After you approve, your credentials are saved for all Flash CLI commands.
## Step 2: Explore the project template
This is the structure of the project template created by `flash init`:
This template includes:
* Example worker files with `@Endpoint` decorated functions for load-balanced and queue-based endpoints.
* Templates for `requirements.txt`, `.env.example`, `.gitignore`, etc.
* Pre-configured endpoint configurations for GPU and CPU workers.
When you start the server, it creates API endpoints at `/gpu/hello` and `/cpu/hello`, which call the endpoint functions described in their respective worker files.
## Step 3: Install Python dependencies
Install required dependencies:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
uv pip install -r requirements.txt
```
## Step 4: Start the local API server
Use `flash dev` to start the API server:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
uv run flash dev
```
Open a new terminal tab or window and test your endpoints using cURL:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
# Test the queue-based GPU endpoint
curl -X POST http://localhost:8888/gpu_worker/runsync \
-H "Content-Type: application/json" \
-d '{"input": {"input_data": {"message": "Hello from the GPU"}}}'
# Test the load-balanced endpoint
curl -X POST http://localhost:8888/lb_worker/process \
-H "Content-Type: application/json" \
-d '{"input_data": {"message": "Hello from Flash"}}'
```
If you switch back to the terminal tab where you used `flash dev`, you'll see the details of the job's progress.
### Faster testing with auto-provisioning
For development with multiple endpoints, use `--auto-provision` to deploy all resources before testing:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
uv run flash dev --auto-provision
```
This eliminates cold-start delays by provisioning all serverless endpoints upfront. Endpoints are cached and reused across server restarts, making subsequent runs faster. Resources are identified by name, so the same endpoint won't be re-deployed if the configuration hasn't changed.
## Step 5: Open the API explorer
Besides starting the API server, `flash dev` also starts an interactive API explorer. Point your web browser at [http://localhost:8888/docs](http://localhost:8888/docs) to explore the API.
To run endpoint functions in the explorer:
1. Expand one of the functions under **GPU Workers** or **CPU Workers**.
2. Click **Try it out** and then **Execute**.
You'll get a response from your workers right in the explorer.
## Step 6: Customize your endpoints
To customize your endpoints:
1. Edit the `@Endpoint` functions in your worker files (`lb_worker.py`, `gpu_worker.py`, `cpu_worker.py`).
2. Add new worker files for new endpoints.
3. Test individual workers by running them as scripts (e.g., `python gpu_worker.py`).
4. Restart the development server to pick up changes.
### Example: Adding a custom GPU endpoint
To add a new GPU endpoint for image generation, create a new worker file or modify an existing one. For deployed apps, each queue-based function needs its own unique endpoint configuration:
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
from runpod_flash import Endpoint, GpuType
@Endpoint(
name="image_generator",
gpu=GpuType.NVIDIA_GEFORCE_RTX_4090,
workers=2,
dependencies=["diffusers", "torch", "transformers", "pillow"]
)
async def generate_image(prompt: str, width: int = 512, height: int = 512) -> dict:
import torch
from diffusers import StableDiffusionPipeline
import base64
import io
pipeline = StableDiffusionPipeline.from_pretrained(
"runwayml/stable-diffusion-v1-5",
torch_dtype=torch.float16
).to("cuda")
image = pipeline(prompt=prompt, width=width, height=height).images[0]
buffered = io.BytesIO()
image.save(buffered, format="PNG")
img_str = base64.b64encode(buffered.getvalue()).decode()
return {"image": img_str, "prompt": prompt}
```
This creates a new Serverless endpoint specifically for image generation. When deployed, it will be available at its own endpoint URL with its own `/run` or `/runsync` routes.
## Step 7: Deploy to Runpod
When you're ready to deploy your app to Runpod, use `flash deploy`:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
uv run flash deploy
```
This command:
1. Builds your application into a deployment artifact.
2. Uploads it to Runpod's storage.
3. Provisions independent Serverless endpoints for each endpoint configuration.
4. Configures service discovery for inter-endpoint communication.
After deployment, you'll receive URLs for all deployed endpoints, grouped by configuration type:
```text theme={"theme":{"light":"github-light","dark":"github-dark"}}
✓ Deployment Complete
Load-balanced endpoints:
https://abc123xyz.api.runpod.ai (lb_worker)
POST /process
GET /health
Queue-based endpoints:
https://api.runpod.ai/v2/def456xyz (gpu_worker)
https://api.runpod.ai/v2/ghi789xyz (cpu_worker)
```
All requests to deployed endpoints require authentication with your Runpod API key. For example:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
# Call a load-balanced endpoint
curl -X POST https://abc123xyz.api.runpod.ai/process \
-H "Authorization: Bearer $RUNPOD_API_KEY" \
-H "Content-Type: application/json" \
-d '{"input_data": {"message": "Hello from Flash"}}'
# Call a queue-based endpoint
curl -X POST https://api.runpod.ai/v2/def456xyz/runsync \
-H "Authorization: Bearer $RUNPOD_API_KEY" \
-H "Content-Type: application/json" \
-d '{"input": {"input_data": {"message": "Hello from the GPU"}}}'
```
For detailed deployment options including environment management, see [Deploy Flash apps](/flash/apps/deploy-apps).
## Next steps
* [Deploy Flash applications](/flash/apps/deploy-apps) for production use.
* [Configure hardware resources](/flash/configuration/parameters) for your endpoints.
* [Monitor and troubleshoot](/flash/troubleshooting) your endpoints.
# Customize your Flash app
Source: https://docs.runpod.io/flash/apps/customize-app
Modify the Flash project template to build your application. Review setup, configuration, deployment, and usage guidance for Runpod Flash.
After running `flash init`, you have a working project template with example and . This guide shows you how to customize the template to build your application.
## Endpoint types
Flash supports two endpoint types, each suited for different use cases:
| Type | Best for | Functions per endpoint |
| ----------------- | ---------------------- | ---------------------- |
| **Queue-based** | Long-running GPU tasks | One |
| **Load-balanced** | Fast HTTP APIs | Multiple (via routes) |
Each `@Endpoint` function creates a separate Serverless endpoint:
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
@Endpoint(name="preprocess", gpu=GpuType.NVIDIA_A100_80GB_PCIe)
def preprocess(data): ...
@Endpoint(name="inference", gpu=GpuType.NVIDIA_A100_80GB_PCIe)
def run_model(input): ...
```
Call via `/run` or `/runsync`: `https://api.runpod.ai/v2/{endpoint_id}/runsync`
Multiple routes share one endpoint:
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
api = Endpoint(name="api-server", cpu="cpu5c-4-8", workers=(1, 5))
@api.post("/generate")
def generate_text(prompt: str): ...
@api.get("/health")
def health_check(): ...
```
Call via HTTP routes: `https://{endpoint_id}.api.runpod.ai/generate`
## Add load balancing routes
To add routes to an existing load balancing endpoint, use the route decorator pattern:
```python title="lb_worker.py" theme={"theme":{"light":"github-light","dark":"github-dark"}}
from runpod_flash import Endpoint
api = Endpoint(name="lb_worker", cpu="cpu5c-4-8", workers=(1, 5))
# Existing routes
@api.post("/process")
async def process(input_data: dict) -> dict:
# ... existing code ...
pass
# Add a new route
@api.get("/status")
async def get_status() -> dict:
return {"status": "healthy", "version": "1.0"}
```
All routes share the same `lb_worker` Serverless endpoint. Each route is accessible at its defined path.
**Key points:**
* Multiple routes can share one endpoint configuration
* Each route has its own HTTP method and path
* All routes on the same endpoint deploy to one Serverless endpoint
## Add queue-based endpoints
To add a new queue-based endpoint, create a new endpoint with a unique name:
```python title="gpu_worker.py" theme={"theme":{"light":"github-light","dark":"github-dark"}}
from runpod_flash import Endpoint, GpuType
# Existing endpoint
@Endpoint(
name="gpu-inference",
gpu=GpuType.NVIDIA_A100_80GB_PCIe,
workers=3,
dependencies=["torch"]
)
async def run_inference(input: dict) -> dict:
import torch
# Inference logic
return {"result": "processed"}
# New endpoint for a different workload
@Endpoint(
name="gpu-training",
gpu=GpuType.NVIDIA_A100_80GB_PCIe,
workers=1,
dependencies=["torch", "transformers"]
)
async def train_model(config: dict) -> dict:
import torch
from transformers import Trainer
# Training logic
return {"model_path": "/models/trained"}
```
This creates two separate Serverless endpoints, each with its own URL and scaling configuration.
**Do not reuse the same endpoint name for multiple queue-based functions when deploying Flash apps.** Each queue-based `@Endpoint` must have its own unique `name` parameter.
## Modify endpoint configurations
Customize endpoint configurations for each worker function in your app. Each `@Endpoint` function can have its own GPU type, scaling parameters, and timeouts optimized for its specific workload.
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
# Example: Different configs for different workloads
@Endpoint(
name="preprocess",
gpu=GpuType.NVIDIA_GEFORCE_RTX_4090, # Cost-effective for preprocessing
workers=(0, 5)
)
async def preprocess(data): ...
@Endpoint(
name="inference",
gpu=GpuType.NVIDIA_A100_80GB_PCIe, # High VRAM for large models
workers=(1, 10) # Keep one worker ready
)
async def inference(data): ...
```
For details, see:
* [Configuration parameters](/flash/configuration/parameters) for all available options.
* [GPU types](/flash/configuration/gpu-types) for selecting hardware.
* [Best practices](/flash/configuration/best-practices) for optimization guidance.
## Test your customizations
After customizing your app, test locally with `flash dev`:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
flash dev
# If using uv:
uv run flash dev
```
This starts a development server at [http://localhost:8888](http://localhost:8888) with:
* Interactive API documentation at `/docs`
* Auto-reload on code changes
* Real remote execution on Runpod workers
Make sure to test:
* All HTTP routes work as expected
* Endpoint functions execute correctly
* Dependencies install properly
* Error handling works
## Next steps
Use `flash dev` for local development and testing.
Deploy your application to production with `flash deploy`.
Complete reference for configuration options.
Learn more about writing and optimizing endpoint functions.
# Deploy Flash apps to Runpod
Source: https://docs.runpod.io/flash/apps/deploy-apps
Build and deploy your Flash app for production serving. Review setup, configuration, deployment, and usage guidance for Runpod Flash.
When you're satisfied with your endpoint functions and ready to move to production, use `flash deploy` to build and deploy your Flash application:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
flash deploy
# If using uv:
uv run flash deploy
```
The Flash CLI manages endpoint and app lifecycle operations: `deploy`, `undeploy`, `update`, and creating or deleting apps and environments. If you call these methods directly from the SDK, Flash raises a `FlashUsageError` that names the equivalent `flash` command. Routing lifecycle changes through the CLI keeps the build and manifest pipeline aligned with Flash's local state tracking. Direct SDK calls bypass that orchestration and can leave deployments in an inconsistent state.
This command performs the following steps:
1. **Build**: Packages your code, dependencies, and manifest.
2. **Upload**: Sends the artifact to Runpod's storage.
3. **Provision**: Creates or updates Serverless endpoints.
4. **Configure**: Sets up environment variables and service discovery.
When you deploy updates to an existing application, Flash automatically triggers a rolling release if your source code has changed. Flash computes a fingerprint of your source files during build, so code-only changes (without resource configuration changes) still result in updated endpoints.
### Deployment architecture
Flash deploys your application as multiple independent Serverless endpoints. Each endpoint configuration in your worker files becomes a separate endpoint.
**How Flash deployments work:**
* **One Endpoint class = one Serverless endpoint**: Each unique endpoint configuration (defined by its `name` parameter) creates a separate Serverless endpoint with its own URL.
* **Call any endpoint**: After deployment, you can call whichever endpoint you need—`lb_worker` for API requests, `gpu_worker` for GPU tasks, `cpu_worker` for CPU tasks.
* [Load balancing endpoints](/flash/create-endpoints#load-balanced-endpoints): Create HTTP APIs with custom routes using `.get()`, `.post()`, etc. decorators.
* [Queue-based endpoints](/flash/create-endpoints#queue-based-endpoints): Run compute tasks using the `/runsync` or `/run` routes.
* **Inter-endpoint communication**: Endpoints can call each other's functions when needed, using the Runpod GraphQL service for discovery.
### Deploy to a specific environment
Flash organizes deployments using [apps and environments](/flash/apps/apps-and-environments). Deploy to a specific environment using the `--env` flag:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
# Deploy to staging
flash deploy --env staging
# Deploy to production
flash deploy --env production
# If using uv:
uv run flash deploy --env staging
uv run flash deploy --env production
```
If the app doesn't exist, Flash creates it along with the target environment. If only the environment doesn't exist, Flash creates it within the existing app.
### Post-deployment
After a successful deployment, Flash displays all deployed endpoints grouped by type:
```text theme={"theme":{"light":"github-light","dark":"github-dark"}}
✓ Deployment Complete
Load-balanced endpoints:
https://abc123xyz.api.runpod.ai (lb_worker)
POST /process
GET /health
Try it:
curl -X POST https://abc123xyz.api.runpod.ai/process \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $RUNPOD_API_KEY" \
-d '{"input_data": {"message": "Hello from Flash"}}'
Queue-based endpoints:
https://api.runpod.ai/v2/def456xyz (gpu_worker)
https://api.runpod.ai/v2/ghi789xyz (cpu_worker)
Try it:
curl -X POST https://api.runpod.ai/v2/def456xyz/runsync \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $RUNPOD_API_KEY" \
-d '{"input": {"input_data": {"message": "Hello from the GPU"}}}'
```
Each endpoint is independent with its own URL and authentication.
The relationship between endpoint configurations and deployed endpoints differs between load-balanced and queue-based endpoints:
### Queue-based endpoints (one function per endpoint)
For queue-based endpoints, each `@Endpoint` function must have its own unique name:
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
from runpod_flash import Endpoint, GpuType
# Each function needs its own endpoint name
@Endpoint(
name="run-model",
gpu=GpuType.NVIDIA_A100_80GB_PCIe,
dependencies=["torch"]
)
def run_model(input: dict): ...
@Endpoint(
name="preprocess",
gpu=GpuType.NVIDIA_A100_80GB_PCIe,
dependencies=["transformers"]
)
def preprocess(data: dict): ...
```
This creates two separate Serverless endpoints:
* `https://api.runpod.ai/v2/abc123xyz` (run-model)
* `https://api.runpod.ai/v2/def456xyz` (preprocess)
**Calling queue-based endpoints:**
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
# Call run_model endpoint (synchronous):
curl -X POST https://api.runpod.ai/v2/abc123xyz/runsync \
-H "Authorization: Bearer $RUNPOD_API_KEY" \
-H "Content-Type: application/json" \
-d '{"input": {"your": "data"}}'
# Or call asynchronously with /run:
curl -X POST https://api.runpod.ai/v2/abc123xyz/run \
-H "Authorization: Bearer $RUNPOD_API_KEY" \
-H "Content-Type: application/json" \
-d '{"input": {"your": "data"}}'
```
**Important:** For deployed queue-based endpoints, you must use **one function per endpoint name**. Each function creates its own Serverless endpoint. Do not create multiple `@Endpoint` functions with the same `name` when building Flash apps.
### Load-balanced endpoints (multiple routes per endpoint)
For load-balanced endpoints, you can define multiple HTTP routes on a single endpoint:
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
from runpod_flash import Endpoint
api = Endpoint(name="api", cpu="cpu5c-4-8", workers=(1, 5))
# Multiple routes on a single Serverless endpoint:
@api.post("/generate")
def generate_text(prompt: str): ...
@api.post("/translate")
def translate_text(text: str): ...
@api.get("/health")
def health_check(): ...
```
This creates:
* **One Serverless endpoint**: `https://abc123xyz.api.runpod.ai` (named "api")
* **Three HTTP routes**: `POST /generate`, `POST /translate`, `GET /health`
**Calling load-balanced endpoints:**
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
# Call the /generate route:
curl -X POST https://abc123xyz.api.runpod.ai/generate \
-H "Authorization: Bearer $RUNPOD_API_KEY" \
-H "Content-Type: application/json" \
-d '{"prompt": "hello"}'
# Call the /health route (same endpoint URL):
curl -X GET https://abc123xyz.api.runpod.ai/health \
-H "Authorization: Bearer $RUNPOD_API_KEY"
```
## Preview before deploying
You can test your deployment locally using Docker before pushing to production using the `--preview` flag:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
flash deploy --preview
# If using uv:
uv run flash deploy --preview
```
This command:
1. Builds your project (creates the deployment artifact and manifest).
2. Creates a Docker network for inter-container communication.
3. Starts one container per endpoint configuration (`lb_worker`, `gpu_worker`, `cpu_worker`, etc.).
4. Exposes all endpoints for local testing.
Press `Ctrl+C` to stop the preview environment.
## Managing deployment size
Runpod Serverless has a **1.5GB deployment limit**. Flash automatically excludes packages that are pre-installed in the base image:
* `torch`, `torchvision`, `torchaudio`
* `numpy`, `triton`
If your deployment still exceeds the limit, use the `--exclude` flag to skip additional packages:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
flash deploy --exclude scipy,pandas
# If using uv:
uv run flash deploy --exclude scipy,pandas
```
### Base image packages
| Configuration type | Base image | Auto-excluded packages |
| ------------------ | --------------- | ------------------------------------------------------- |
| GPU (`gpu=`) | PyTorch base | `torch`, `torchvision`, `torchaudio`, `numpy`, `triton` |
| CPU (`cpu=`) | Python slim | `torch`, `torchvision`, `torchaudio`, `numpy`, `triton` |
| Load-balanced | Same as GPU/CPU | Same as GPU/CPU |
Check the [Flash worker repository](https://github.com/runpod-workers/flash) for current base images and pre-installed packages.
## Build process
When you run `flash deploy` (or `flash build`), Flash:
1. **Discovers** all `@Endpoint` decorated functions.
2. **Groups** functions by their endpoint name.
3. **Generates** handler files for each endpoint.
4. **Creates** a `flash_manifest.json` file for service discovery.
5. **Installs** dependencies with Linux x86\_64 compatibility.
6. **Packages** everything into `.flash/artifact.tar.gz`.
### Build artifacts
After building, these artifacts are created in the `.flash/` directory:
| Artifact | Description |
| ---------------------------- | ---------------------------------------------- |
| `.flash/artifact.tar.gz` | Deployment package |
| `.flash/flash_manifest.json` | Service discovery configuration |
| `.flash/.build/` | Temporary build directory (removed by default) |
## What gets deployed
When you deploy a Flash app, you're deploying a **build artifact** (tarball) onto pre-built Flash Docker images. This architecture is similar to AWS Lambda layers: the base runtime is pre-built, and your code and dependencies are layered on top.
### The build artifact
The `.flash/artifact.tar.gz` file (max 1.5 GB) contains:
Dependencies are installed locally during the build process and bundled into the tarball. They are **not** installed at runtime on endpoints.
### The deployment manifest
The `flash_manifest.json` file is the brain of your deployment. It tells each endpoint:
* Which functions to execute.
* What Docker image to use.
* How to configure resources (GPUs, workers, scaling).
* Environment variables for workers.
* How to route HTTP requests (for load balancer endpoints).
```json theme={"theme":{"light":"github-light","dark":"github-dark"}}
{
"resources": {
"lb_worker": {
"is_load_balanced": true,
"imageName": "runpod/flash-lb-cpu:latest",
"workersMin": 1,
"functions": [
{"name": "process", "module": "lb_worker"},
{"name": "health", "module": "lb_worker"}
]
},
"gpu_worker": {
"imageName": "runpod/flash:latest",
"gpuIds": "AMPERE_16",
"workersMax": 3,
"env": {
"HF_TOKEN": "your_token",
"MODEL_ID": "gpt2"
},
"functions": [
{"name": "gpu_hello", "module": "gpu_worker"}
]
},
"cpu_worker": {
"imageName": "runpod/flash-cpu:latest",
"workersMax": 2,
"functions": [
{"name": "cpu_hello", "module": "cpu_worker"}
]
}
},
"routes": {
"lb_worker": {
"POST /process": "process",
"GET /health": "health"
}
}
}
```
### What gets created on Runpod
For each endpoint configuration in the manifest, Flash creates an independent Serverless endpoint, identified by its `name` parameter.
### Cross-endpoint communication
When one endpoint needs to call a function on another endpoint:
1. **Manifest lookup**: The calling endpoint checks `flash_manifest.json` for function-to-resource mapping.
2. **Service discovery**: It queries the state manager (Runpod GraphQL API) for target endpoint URL.
3. **Direct call**: It makes an HTTP request directly to the target endpoint.
4. **Response**: The target endpoint executes the function and returns the result.
Each endpoint maintains its own connection to the state manager, querying for peer endpoint URLs as needed and caching results for 300 seconds to minimize API calls.
#### Calling another endpoint from your code
To call one endpoint from another, import the target endpoint function **inside** your function body. Flash automatically detects these imports and generates the necessary dispatch stubs.
For example, if you have a GPU worker for inference:
```python gpu_worker.py theme={"theme":{"light":"github-light","dark":"github-dark"}}
from runpod_flash import Endpoint, GpuType
@Endpoint(
name="gpu-inference",
gpu=GpuType.NVIDIA_GEFORCE_RTX_4090,
dependencies=["torch"]
)
async def gpu_inference(payload: dict) -> dict:
import torch
# GPU inference logic
return {"result": "processed"}
```
You can call it from a CPU-based pipeline endpoint:
```python cpu_worker.py theme={"theme":{"light":"github-light","dark":"github-dark"}}
from runpod_flash import Endpoint
@Endpoint(name="pipeline", cpu="cpu5c-4-8")
async def classify(text: str) -> dict:
# Import the GPU endpoint inside the function body
from gpu_worker import gpu_inference
# Flash routes this call to the gpu-inference endpoint
result = await gpu_inference({"text": text})
return {"classification": result}
```
## Call deployed endpoints from scripts
After deploying your Flash app, you can call your `@Endpoint` functions directly from Python scripts. Flash automatically resolves the app context from your project structure, so in most cases you can run scripts without any additional configuration.
### How it works
When you run a script that calls an `@Endpoint` function, Flash:
1. Detects the app context from the project directory structure.
2. Looks up the deployed endpoint by name within the resolved app and environment.
3. Routes the request to that endpoint using Flash's sentinel service.
4. Returns the result to your script.
This lets you reuse the same `@Endpoint` function definitions to interact with deployed endpoints without modifying your code.
### Example: calling within the same script
The simplest approach is to call the endpoint directly in the same file where it's defined:
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
# gpu_worker.py
import asyncio
from runpod_flash import Endpoint, GpuType
@Endpoint(
name="inference",
gpu=GpuType.NVIDIA_GEFORCE_RTX_4090,
dependencies=["torch"]
)
async def run_inference(data: dict) -> dict:
import torch
# Inference logic
return {"result": "processed"}
async def main():
result = await run_inference({"input": "data"})
print(result)
if __name__ == "__main__":
asyncio.run(main())
```
Run the script:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
python gpu_worker.py
```
### Example: importing from another script
You can also import and call endpoints from a separate script:
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
# call_inference.py
import asyncio
from gpu_worker import run_inference
async def main():
# Flash resolves the app context automatically
result = await run_inference({"input": "data"})
print(result)
if __name__ == "__main__":
asyncio.run(main())
```
Run the script:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
python call_inference.py
```
### Override the resolved context
Flash resolves the app name from your project's directory structure. Use `FLASH_APP` and `FLASH_ENV` environment variables to override this automatic resolution when needed.
A common use case is when you move a script to a different directory. Since the resolved app name depends on the directory location, moving the script changes the resolved context. To continue targeting the original app, set `FLASH_APP` explicitly:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
FLASH_APP=my-app python call_inference.py
```
You can also override the environment:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
FLASH_APP=my-app FLASH_ENV=production python call_inference.py
```
### Error without context
If Flash cannot resolve the app context and you haven't set the environment variables, it raises an error:
```text theme={"theme":{"light":"github-light","dark":"github-dark"}}
RuntimeError: no flash context for endpoint 'inference'. either:
- use 'flash dev' for local development
- set FLASH_APP and FLASH_ENV to target a deployed environment
```
### Automatic context in deployed workers
When Flash deploys your app, it automatically sets `FLASH_APP` and `FLASH_ENV` environment variables on each worker. This enables cross-endpoint communication within your deployed application without additional configuration.
## Troubleshooting
### No @Endpoint functions found
If the build process can't find your endpoint functions:
* Ensure functions are decorated with `@Endpoint(...)`.
* Check that Python files aren't excluded by `.gitignore` or Flash's [built-in ignore patterns](/flash/cli/build#built-in-ignore-patterns).
* Verify decorator syntax is correct.
### Deployment size limit exceeded
Base image packages are auto-excluded. If your deployment still exceeds 1.5GB, use `--exclude` to skip additional packages:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
flash deploy --exclude scipy,pandas
```
### Authentication errors
Verify your API key is set correctly:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
echo $RUNPOD_API_KEY
```
If not set, add it to your `.env` file or export it:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
export RUNPOD_API_KEY=your_api_key_here
```
### Import errors in endpoint functions
Import packages inside the endpoint function, not at the top of the file:
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
@Endpoint(name="fetch-data", gpu=GpuGroup.ANY, dependencies=["requests"])
def fetch_data(url):
import requests # Import here
return requests.get(url).json()
```
## Next steps
* [Learn about apps and environments](/flash/apps/apps-and-environments) for managing deployments.
* [View the CLI reference](/flash/cli/overview) for all available commands.
* [Configure hardware resources](/flash/configuration/parameters) for your endpoints.
* [Monitor and troubleshoot](/flash/troubleshooting) your deployments.
# Initialize a Flash app project
Source: https://docs.runpod.io/flash/apps/initialize-project
Use flash init to create a new Flash project with a ready-to-use structure. Review configuration and usage details for Runpod Flash.
The `flash init` command creates a new Flash project with a complete project structure, including example and , and configuration files. This gives you a working starting point for building Flash applications.
Use `flash init` whenever you want to start a new Flash project, fully configured for you to run `flash dev` and `flash deploy`.
## Create a new project
Create a new project in a new directory:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
flash init PROJECT_NAME
cd PROJECT_NAME
# If using uv:
uv run flash init PROJECT_NAME
```
Or initialize in your current directory:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
flash init .
# If using uv:
uv run flash init .
```
## Project structure
`flash init` creates the following structure:
### Key files
**lb\_worker.py**: An example load-balanced worker with HTTP routes. Contains `@Endpoint` functions with custom HTTP methods and paths (e.g., `POST /process`, `GET /health`). Creates one endpoint when deployed.
**gpu\_worker.py**: An example GPU queue-based worker. Contains `@Endpoint` functions that run on GPU hardware. Provides `/runsync` route for job submission. Creates one endpoint when deployed.
**cpu\_worker.py**: An example CPU queue-based worker. Contains `@Endpoint` functions that run on CPU-only instances. Provides `/runsync` route for job submission. Creates one endpoint when deployed.
Each worker file defines a resource configuration and its associated functions. When you deploy, Flash creates one Serverless endpoint per unique resource configuration.
## Set up the project
After initialization, complete the setup:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
# Install dependencies
pip install -r requirements.txt
# Copy environment template
cp .env.example .env
# Add your API key to .env
# RUNPOD_API_KEY=your_api_key_here
```
## Handle existing files
If you run `flash init` in a directory with existing files, Flash detects conflicts and prompts for confirmation:
```text theme={"theme":{"light":"github-light","dark":"github-dark"}}
File Conflicts Detected
Warning: The following files will be overwritten:
• requirements.txt
• gpu_worker.py
• README.md
• lb_worker.py
• cpu_worker.py
Continue and overwrite these files? [y/N]:
```
Use `--force` to skip the prompt and overwrite files:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
flash init . --force
# If using uv:
uv run flash init . --force
```
## Start developing
Once your project is set up:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
# Start the development server
flash dev
# Open the API explorer
# http://localhost:8888/docs
# If using uv:
uv run flash dev
```
Make changes to your worker files, and the server reloads automatically. When you're ready, deploy with:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
flash deploy
# If using uv:
uv run flash deploy
```
## Next steps
* [Customize your app](/flash/apps/customize-app) to add endpoints and modify configurations.
* [Test locally](/flash/apps/local-testing) with `flash dev`.
* [Deploy to production](/flash/apps/deploy-apps) with `flash deploy`.
* [View the flash init reference](/flash/cli/init) for all options.
# Test Flash apps locally
Source: https://docs.runpod.io/flash/apps/local-testing
Use flash dev to test your Flash application locally before deploying. Review setup, configuration, deployment, and usage guidance for Runpod Flash.
The `flash dev` command starts a local development server that lets you test your Flash application before deploying to production. The development server runs locally and updates automatically as you edit files.
When you call a `@Endpoint` function, Flash sends the latest function code to Serverless workers on Runpod, so your changes are reflected immediately.
## Start the development server
From inside your [project directory](/flash/apps/initialize-project), run:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
flash dev
# If using uv:
uv run flash dev
```
The server starts at `http://localhost:8888` by default. Your endpoints are available immediately for testing, and `@Endpoint` functions provision Serverless endpoints on first call.
### Using a custom host and port
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
# Change port
flash dev --port 3000
# Make accessible on network
flash dev --host 0.0.0.0
# If using uv:
uv run flash dev --port 3000
uv run flash dev --host 0.0.0.0
```
## Test your endpoints
### Using curl
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
# Call a queue-based endpoint (gpu_worker.py)
curl -X POST http://localhost:8888/gpu_worker/runsync \
-H "Content-Type: application/json" \
-d '{"input": {"input_data": {"message": "Hello from the GPU"}}}'
# Call a load-balanced endpoint (lb_worker.py)
curl -X POST http://localhost:8888/lb_worker/process \
-H "Content-Type: application/json" \
-d '{"input_data": {"message": "Hello from Flash"}}'
```
Queue-based endpoints require the `{"input": {...}}` wrapper to match the deployed endpoint behavior. The inner payload structure maps to your function's parameter names—the skeleton template uses `input_data: dict`, so the payload is `{"input_data": {...}}`. Load-balanced endpoints accept the payload directly without the `input` wrapper.
### Using the API explorer
Open [http://localhost:8888/docs](http://localhost:8888/docs) in your browser to access the interactive Swagger UI. You can test all endpoints directly from the browser.
Flash extracts the first line of each function's docstring and displays it as the endpoint description in the API explorer. Add docstrings to your `@Endpoint` functions to make your API self-documenting:
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
@Endpoint(name="gpu-worker", gpu=GpuGroup.ANY)
def process_data(input_data: dict) -> dict:
"""Process input data and return computed results."""
# Function implementation
return {"result": "processed"}
```
The docstring "Process input data and return computed results" appears in the Swagger UI, making it easier to understand what each endpoint does.
### Using Python
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
import requests
# Call queue-based endpoint
response = requests.post(
"http://localhost:8888/gpu_worker/runsync",
json={"input": {"input_data": {"message": "Hello from the GPU"}}}
)
print(response.json())
# Call load-balanced endpoint
response = requests.post(
"http://localhost:8888/lb_worker/process",
json={"input_data": {"message": "Hello from Flash"}}
)
print(response.json())
```
## Reduce cold-start delays
The first call to a `@Endpoint` function provisions a Serverless endpoint, which takes 30-60 seconds. Use `--auto-provision` to provision all endpoints at startup:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
flash dev --auto-provision
# If using uv:
uv run flash dev --auto-provision
```
This scans your project for `@Endpoint` functions and deploys them before the server starts accepting requests. Endpoints are cached in `.flash/resources.pkl` and reused across server restarts.
## How it works
With `flash dev`, Flash starts a local development server alongside remote Serverless endpoints:
```mermaid theme={"theme":{"light":"github-light","dark":"github-dark"}}
%%{init: {'theme':'base', 'themeVariables': { 'primaryColor':'#9289FE','primaryTextColor':'#fff','primaryBorderColor':'#9289FE','lineColor':'#5F4CFE','secondaryColor':'#AE6DFF','tertiaryColor':'#FCB1FF','edgeLabelBackground':'#5F4CFE', 'fontSize':'14px','fontFamily':'font-inter'}}}%%
flowchart TB
Browser(["BROWSER/CURL"])
subgraph Local ["YOUR MACHINE (localhost:8888)"]
DevServer["Development Server • Auto-reload on changes • API explorer at /docs • Routes requests"]
end
subgraph Runpod ["RUNPOD SERVERLESS"]
LB["live-lb_worker"]
GPU["live-gpu_worker"]
CPU["live-cpu_worker"]
end
Browser -->|"HTTP"| DevServer
DevServer -->|"HTTPS"| LB
DevServer -->|"HTTPS"| GPU
DevServer -->|"HTTPS"| CPU
style Local fill:#1a1a2e,stroke:#5F4CFE,stroke-width:2px,color:#fff
style Runpod fill:#1a1a2e,stroke:#5F4CFE,stroke-width:2px,color:#fff
style Browser fill:#4D38F5,stroke:#4D38F5,color:#fff
style DevServer fill:#5F4CFE,stroke:#5F4CFE,color:#fff
style LB fill:#22C55E,stroke:#22C55E,color:#000
style GPU fill:#22C55E,stroke:#22C55E,color:#000
style CPU fill:#22C55E,stroke:#22C55E,color:#000
```
**What runs where:**
| Component | Location |
| ------------------------- | ----------------------------- |
| Development server | Your machine (localhost:8888) |
| `@Endpoint` function code | Runpod Serverless |
| Endpoint storage | Runpod Serverless |
Your code updates automatically as you edit files. Endpoints created by `flash dev` are prefixed with `live-` to distinguish them from production endpoints.
## Clean up after testing
Endpoints created by `flash dev` persist until you delete them. To clean up:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
# List all endpoints
flash undeploy list
# Remove a specific endpoint
flash undeploy ENDPOINT_NAME
# Remove all endpoints
flash undeploy --all
# If using uv:
uv run flash undeploy list
uv run flash undeploy ENDPOINT_NAME
uv run flash undeploy --all
```
## Troubleshooting
**Port already in use**
Flash automatically selects the next available port if your specified port is in use. You'll see a message like `Port 8888 is in use, using 8889 instead.` If you need a specific port, stop the process using it or specify a different starting port with `--port`.
**Slow first request**
Use `--auto-provision` to eliminate cold-start delays:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
flash dev --auto-provision
# If using uv:
uv run flash dev --auto-provision
```
**Authentication errors**
Run `flash login` to authenticate:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
flash login
# If using uv:
uv run flash login
```
Alternatively, set `RUNPOD_API_KEY` as an environment variable or in your `.env` file:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
export RUNPOD_API_KEY="your_api_key_here"
```
Values in your `.env` file are only available locally for CLI commands. They are not passed to deployed endpoints.
## Next steps
* [Deploy to production](/flash/apps/deploy-apps) when your app is ready.
* [Clean up endpoints](/flash/cli/undeploy) after testing.
* [View the flash dev reference](/flash/cli/dev) for all options.
# Overview
Source: https://docs.runpod.io/flash/apps/overview
Understand the Flash app development lifecycle. Review setup, configuration, deployment, and usage guidance for Runpod Flash.
A Flash app is a collection of endpoints deployed to Runpod.
Create a Flash app, test it locally, and deploy it to production.
Create boilerplate code for a new Flash project with `flash init`.
## App development workflow
Building a Flash application follows a clear progression from initialization to production deployment:
Use `flash init` to create a new project with example workers:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
flash init PROJECT_NAME
cd PROJECT_NAME
pip install -r requirements.txt
```
This gives you a working project structure with GPU and CPU worker examples. [Learn more about project initialization](/flash/apps/initialize-project).
Write your application code by defining `Endpoint` functions that execute on Runpod workers:
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
from runpod_flash import Endpoint, GpuType
@Endpoint(
name="inference-worker",
gpu=GpuType.NVIDIA_GEFORCE_RTX_4090,
workers=3,
dependencies=["torch"]
)
def run_inference(prompt: str) -> dict:
import torch
# Your inference logic here
return {"result": "..."}
```
[Learn more about customizing your app](/flash/apps/customize-app).
Start a local development server to test your application:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
flash dev
```
Your app runs locally and updates automatically. When you call an `@Endpoint` function, Flash sends the latest code to Runpod workers. [Learn more about local testing](/flash/apps/local-testing).
When ready for production, deploy your application to Runpod Serverless:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
flash deploy
```
When you deploy an app, Runpod:
1. Packages your code, dependencies, and deployment manifest into a tarball (max 1.5 GB).
2. Uploads the tarball to Runpod.
3. Provisions independent Serverless endpoints based on your [endpoint configurations](/flash/create-endpoints).
Your entire application—including all worker functions—runs on Runpod infrastructure. [Learn more about deployment](/flash/apps/deploy-apps).
Use apps and environments to organize and manage your deployments across different stages (dev, staging, production). [Learn more about apps and environments](/flash/apps/apps-and-environments).
## Apps and environments
Flash uses a two-level organizational structure: **apps** (project containers) and **environments** (deployment stages like dev, staging, production). See [Apps and environments](/flash/apps/apps-and-environments) for complete details.
## Next steps
Create a Flash app, test it locally, and deploy it to production.
Create boilerplate code for a new Flash project with `flash init`.
Use `flash dev` for local development and testing.
Deploy your application to production with `flash deploy`.
# Send API requests to deployed endpoints
Source: https://docs.runpod.io/flash/apps/requests
Call your deployed Flash endpoints using HTTP requests for queue-based and load-balanced configurations.
After deploying your Flash app with `flash deploy`, you can call your endpoints directly via HTTP. The request format depends on whether you're using queue-based or load-balanced configurations.
## Authentication
All deployed endpoints require authentication with your Runpod API key:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
export RUNPOD_API_KEY="your_key_here"
curl -X POST https://YOUR_ENDPOINT_URL/path \
-H "Authorization: Bearer $RUNPOD_API_KEY" \
-H "Content-Type: application/json" \
-d '{"param": "value"}'
```
Your endpoint URLs are displayed after running `flash deploy`. You can also view them with `flash env get `.
## Queue-based endpoints
Queue-based endpoints (using `@Endpoint(name=..., gpu=...)` decorator) provide two routes for job submission: `/run` (asynchronous) and `/runsync` (synchronous).
### Asynchronous calls (`/run`)
Submit a job and receive a job ID for later status checking:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
curl -X POST https://api.runpod.ai/v2/abc123xyz/run \
-H "Authorization: Bearer $RUNPOD_API_KEY" \
-H "Content-Type: application/json" \
-d '{"input": {"prompt": "Hello world"}}'
```
**Response:**
```json theme={"theme":{"light":"github-light","dark":"github-dark"}}
{
"id": "job-abc-123",
"status": "IN_QUEUE"
}
```
**Check job status and retrieve results:**
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
curl https://api.runpod.ai/v2/abc123xyz/status/job-abc-123 \
-H "Authorization: Bearer $RUNPOD_API_KEY"
```
**When the job completes:**
```json theme={"theme":{"light":"github-light","dark":"github-dark"}}
{
"id": "job-abc-123",
"status": "COMPLETED",
"output": {
"generated_text": "Hello world from GPU!"
}
}
```
### Synchronous calls (`/runsync`)
Wait for job completion and receive results directly (with timeout):
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
curl -X POST https://api.runpod.ai/v2/abc123xyz/runsync \
-H "Authorization: Bearer $RUNPOD_API_KEY" \
-H "Content-Type: application/json" \
-d '{"input": {"prompt": "Hello world"}}'
```
**Response (after job completes):**
```json theme={"theme":{"light":"github-light","dark":"github-dark"}}
{
"id": "job-abc-123",
"status": "COMPLETED",
"output": {
"generated_text": "Hello world from GPU!"
}
}
```
The `/runsync` endpoint has a 60-second client-side timeout by default. If you've configured `execution_timeout_ms` on your endpoint, the client timeout uses that value instead. For jobs that take longer than 60 seconds, set `execution_timeout_ms` to prevent `/runsync` requests from timing out.
Use `/run` for long-running jobs that you'll check later. Use `/runsync` for quick jobs where you want immediate results (with timeout protection).
### Queue-based request format
Queue-based endpoints expect input wrapped in an `{"input": {...}}` object:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
curl -X POST https://api.runpod.ai/v2/abc123xyz/runsync \
-H "Authorization: Bearer $RUNPOD_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"input": {
"param1": "value1",
"param2": "value2"
}
}'
```
The structure inside `"input"` depends on your `@Endpoint` function signature.
### Job status states
| Status | Description |
| ------------- | --------------------------------- |
| `IN_QUEUE` | Waiting for an available worker |
| `IN_PROGRESS` | Worker is executing your function |
| `COMPLETED` | Function finished successfully |
| `FAILED` | Execution encountered an error |
## Load-balanced endpoints
Load-balanced endpoints (using `api = Endpoint(...); @api.post("/path")` pattern) provide custom HTTP routes with direct request/response patterns.
### Calling load-balanced routes
All routes share the same base URL. Append the route path to call specific functions:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
# POST route
curl -X POST https://abc123xyz.api.runpod.ai/analyze \
-H "Authorization: Bearer $RUNPOD_API_KEY" \
-H "Content-Type: application/json" \
-d '{"text": "Hello world from Flash"}'
# GET route
curl -X GET https://abc123xyz.api.runpod.ai/info \
-H "Authorization: Bearer $RUNPOD_API_KEY"
# Another POST route (same endpoint URL)
curl -X POST https://abc123xyz.api.runpod.ai/validate \
-H "Authorization: Bearer $RUNPOD_API_KEY" \
-H "Content-Type: application/json" \
-d '{"name": "Alice", "email": "alice@example.com"}'
```
### Load-balanced request format
Load-balanced endpoints accept direct JSON payloads (no `{"input": {...}}` wrapper):
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
curl -X POST https://abc123xyz.api.runpod.ai/process \
-H "Authorization: Bearer $RUNPOD_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"param1": "value1",
"param2": "value2"
}'
```
The payload structure depends on your function signature. Each route can accept different parameters.
### Multiple routes, single endpoint
A single load-balanced endpoint can serve multiple routes:
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
from runpod_flash import Endpoint
api = Endpoint(name="api-server", cpu="cpu5c-4-8", workers=(1, 5))
# All these routes share one endpoint URL
@api.post("/generate")
async def generate_text(prompt: str): ...
@api.post("/translate")
async def translate_text(text: str): ...
@api.get("/health")
async def health_check(): ...
```
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
# All use the same base URL with different paths
curl -X POST https://abc123xyz.api.runpod.ai/generate -H "..." -d '{...}'
curl -X POST https://abc123xyz.api.runpod.ai/translate -H "..." -d '{...}'
curl -X GET https://abc123xyz.api.runpod.ai/health -H "..."
```
## Quick reference
| Endpoint Type | Routes | Request Format | Response |
| ------------- | ---------------------------------- | ------------------- | ------------------------------- |
| Queue-based | `/run`, `/runsync`, `/status/{id}` | `{"input": {...}}` | Job ID (async) or result (sync) |
| Load-balanced | Custom paths (e.g., `/process`) | Direct JSON payload | Direct response |
## Response status codes
| Code | Meaning |
| ----- | ----------------------------------------------------- |
| `200` | Success (load-balanced) or job accepted (queue-based) |
| `400` | Bad request (invalid input format) |
| `401` | Unauthorized (invalid or missing API key) |
| `404` | Route not found |
| `500` | Internal server error |
## Error handling
Queue-based errors appear in the job output:
```json theme={"theme":{"light":"github-light","dark":"github-dark"}}
{
"id": "job-abc-123",
"status": "FAILED",
"error": "Error message from your function"
}
```
Load-balanced errors return HTTP error codes with JSON body:
```json theme={"theme":{"light":"github-light","dark":"github-dark"}}
{
"error": "Error message from your function",
"detail": "Additional error context"
}
```
## Using SDKs
For programmatic access, use the Runpod Python SDK:
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
import runpod
# Set API key
runpod.api_key = "your_api_key"
# Connect to endpoint
endpoint = runpod.Endpoint("YOUR_ENDPOINT_ID")
# Async call (returns job object immediately)
run_request = endpoint.run({"prompt": "Hello world"})
status = run_request.status() # Check status
output = run_request.output() # Get result once complete
# Sync call (blocks until complete)
result = endpoint.run_sync({"prompt": "Hello world"})
```
See the [Runpod SDK documentation](/serverless/endpoints/send-requests) for complete SDK usage.
## Next steps
Deploy your Flash app to get endpoint URLs.
View all endpoint configuration parameters.
Use the Python SDK for programmatic access.
# app
Source: https://docs.runpod.io/flash/cli/app
Create, inspect, list, and delete Flash applications that organize deployment environments, build artifacts, and configuration.
Manage Flash applications. An app is the top-level container that groups your deployment environments, build artifacts, and configuration.
```bash Command theme={"theme":{"light":"github-light","dark":"github-dark"}}
flash app [OPTIONS]
```
## Subcommands
| Subcommand | Description |
| ---------- | ----------------------------------- |
| `list` | Show all apps in your account |
| `create` | Create a new app |
| `get` | Show details of an app |
| `delete` | Delete an app and all its resources |
***
## app list
Show all Flash apps under your account.
```bash Command theme={"theme":{"light":"github-light","dark":"github-dark"}}
flash app list
```
### Output
```text theme={"theme":{"light":"github-light","dark":"github-dark"}}
┏━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━┓
┃ Name ┃ ID ┃ Environments ┃ Builds ┃
┡━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━┩
│ my-project │ app_abc123 │ dev, staging, prod │ build_1, build_2 │
│ demo-api │ app_def456 │ production │ build_3 │
│ ml-inference │ app_ghi789 │ dev, production │ build_4, build_5 │
└────────────────┴──────────────────────┴─────────────────────────┴──────────────────┘
```
***
## app create
Register a new Flash app on Runpod's backend.
```bash Command theme={"theme":{"light":"github-light","dark":"github-dark"}}
flash app create
```
### Arguments
Name for the new Flash app. Must be unique within your account.
### What it creates
This command registers a Flash app in Runpod's backend—essentially creating a namespace for your environments and builds. It does not:
* Create local files (use `flash init` for that).
* Provision cloud resources (endpoints, volumes, etc.).
* Deploy any code.
The app is just a container that groups environments and builds together.
### When to use
Most users don't need to run `flash app create` explicitly. Apps are created automatically when you first run `flash deploy`. This command is primarily for CI/CD pipelines that need to pre-register apps before deployment.
***
## app get
Get detailed information about a Flash app.
```bash Command theme={"theme":{"light":"github-light","dark":"github-dark"}}
flash app get
```
### Arguments
Name of the Flash app to inspect.
### Output
```text theme={"theme":{"light":"github-light","dark":"github-dark"}}
╭─────────────────────────────────╮
│ Flash App: my-project │
├─────────────────────────────────┤
│ Name: my-project │
│ ID: app_abc123 │
│ Environments: 3 │
│ Builds: 5 │
╰─────────────────────────────────╯
Environments
┏━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━┓
┃ Name ┃ ID ┃ State ┃ Active Build ┃ Created ┃
┡━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━┩
│ dev │ env_dev123 │ DEPLOYED│ build_xyz789 │ 2024-01-15 10:30 │
│ staging │ env_stg456 │ DEPLOYED│ build_xyz789 │ 2024-01-16 14:20 │
│ production │ env_prd789 │ DEPLOYED│ build_abc123 │ 2024-01-20 09:15 │
└────────────┴────────────────────┴─────────┴──────────────────┴──────────────────┘
Builds
┏━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━┓
┃ ID ┃ Status ┃ Created ┃
┡━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━┩
│ build_abc123 │ COMPLETED │ 2024-01-20 09:00 │
│ build_xyz789 │ COMPLETED │ 2024-01-18 15:45 │
│ build_def456 │ COMPLETED │ 2024-01-15 11:20 │
└────────────────────┴──────────────────────────┴──────────────────┘
```
***
## app delete
Delete a Flash app and all its associated resources.
```bash Command theme={"theme":{"light":"github-light","dark":"github-dark"}}
flash app delete
```
### Arguments
Name of the Flash app to delete.
### Process
1. Shows app details and resources to be deleted.
2. Prompts for confirmation (required).
3. Deletes all environments and their resources.
4. Deletes all builds.
5. Tears down the app's Serverless endpoints. Flash discovers them per Flash environment on Runpod's backend, so teardown works even when no local resource tracking exists (for example, in CI).
6. If every endpoint was removed, deletes the app record.
If any endpoint cannot be removed, the app record is not deleted and the command exits with a non-zero status; see [app delete fails to remove an endpoint](#app-delete-fails-to-remove-an-endpoint).
### Output
Each removed endpoint prints a confirmation line, where `` and `` are the endpoint's name and ID:
```text theme={"theme":{"light":"github-light","dark":"github-dark"}}
✓ deleted endpoint
```
This operation is irreversible. All environments, builds, endpoints, volumes, and configuration will be permanently deleted.
***
## App hierarchy
See [Apps and environments](/flash/apps/apps-and-environments#app-hierarchy) for the complete app organization structure.
## Auto-detection
Flash CLI automatically detects the app name from your current directory:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
cd /path/to/APP_NAME
flash deploy # Deploys to 'APP_NAME' app
flash env list # Lists 'APP_NAME' environments
```
Override with the `--app` flag:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
flash deploy --app other-project
flash env list --app other-project
```
## Troubleshooting
### app delete fails to remove an endpoint
When any endpoint cannot be removed, `flash app delete` does not delete the app record and exits with a non-zero status (exit code 1). It names each endpoint that survived teardown. When the endpoint still has an ID, the message gives the command to remove it:
```text theme={"theme":{"light":"github-light","dark":"github-dark"}}
endpoint not removed; delete it with runpodctl serverless delete
```
Remove the surviving endpoint with [`runpodctl serverless delete`](/runpodctl/reference/runpodctl-serverless#delete-an-endpoint):
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
runpodctl serverless delete
```
If an endpoint is reported without an ID, it cannot be removed this way; delete it from the Runpod console instead.
Re-running `flash app delete ` after a partial teardown is safe: endpoints that were already removed count as removed, so the command retries only the survivors.
## Related commands
* [`flash env`](/flash/cli/env) - Manage environments within an app
* [`flash deploy`](/flash/cli/deploy) - Deploy to an app's environment
* [`flash init`](/flash/cli/init) - Create a new project
# build
Source: https://docs.runpod.io/flash/cli/build
Build a deployment-ready Flash application artifact without deploying, with controls for dependencies, exclusions, and output files.
Build a deployment-ready artifact for your Flash application without deploying. Use this for more control over the build process or to inspect the artifact before deploying.
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
flash build [OPTIONS]
```
## Examples
Build with all dependencies:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
flash build
```
Build with additional excluded packages:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
flash build --exclude scipy,pandas
```
Build with custom output name:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
flash build -o my-app.tar.gz
```
## Flags
Skip transitive dependencies during pip install. Only installs direct dependencies specified in `@Endpoint` decorators. Useful when the base image already includes dependencies.
Custom name for the output archive file.
Comma-separated list of packages to exclude from the build (e.g., `torch,torchvision`). Use this to skip packages already in the base image.
Target Python version for worker images (3.10, 3.11, 3.12, or 3.13). Overrides per-resource `python_version` declarations and local interpreter detection.
## What happens during build
1. **Python version resolution**: Resolves the target Python version from CLI flag, resource configs, or your local interpreter.
2. **Function discovery**: Finds all `@Endpoint` decorated functions.
3. **Grouping**: Groups functions by their endpoint configuration.
4. **Manifest generation**: Creates `.flash/flash_manifest.json` with endpoint definitions.
5. **Source fingerprinting**: Computes a SHA-256 fingerprint of your source files to detect code changes between deployments.
6. **Dependency installation**: Installs Python packages for Linux x86\_64.
7. **Packaging**: Bundles everything into `.flash/artifact.tar.gz`.
## Built-in ignore patterns
Flash automatically excludes certain files and directories from deployment artifacts. These patterns cover common development files that shouldn't be deployed to production.
| Category | Patterns |
| -------------------- | ---------------------------------------------------------------------------------------------------------------------------------- |
| Build artifacts | `.build/`, `.flash/`, `.runpod/`, `*.tar.gz`, `.git/`, `__pycache__/`, `*.pyc`, `*.pyo`, `*.pyd`, `*.egg-info/`, `dist/`, `build/` |
| Virtual environments | `.venv/`, `venv/`, `env/` |
| IDE files | `.vscode/`, `.idea/` |
| Environment files | `.env`, `.env.local` |
| Tests | `tests/`, `test_*.py`, `*_test.py` |
| Documentation | `docs/`, `*.md` (except `README.md`) |
Flash also respects your `.gitignore` file and excludes any files matching those patterns.
If you use other environment file variants like `.env.dev` or `.env.staging`, add them to your `.gitignore` to exclude them from deployment artifacts.
### Local modules and the ignore filter
Local (non-pip) modules that your endpoints import are bundled automatically, provided they pass the ignore filter: your `.gitignore` plus the built-in patterns listed above. During the build, Flash resolves the transitive local-import closure of every shipped Python file and checks it against the files the ignore filter already selected.
If shipped code imports a local module that an ignore rule excludes (for example, a `test_*.py` sibling or a file under `tests/`), the build fails with a `LocalModuleResolutionError` instead of silently overriding your ignore rules or shipping a broken artifact. Flash names each excluded file and the file that imports it:
```
Shipped code imports local modules that the build ignore rules (.gitignore or built-in defaults) exclude:
utils/helpers.py (imported by endpoint.py)
Shipping them would silently override a deliberate exclusion, and omitting them would break the worker with ModuleNotFoundError. Remove the matching ignore pattern or stop importing these modules from shipped code.
```
If an `@Endpoint` file has a local import Flash can't resolve at all (a broken relative import, or a file outside your project root), the build also fails with a clear error. A file that fails resolution but doesn't define an endpoint is skipped with a warning and the build continues. For details on how local imports are resolved and bundled, see [Import local modules](/flash/create-endpoints#import-local-modules).
## Build artifacts
After running `flash build`:
| File/Directory | Description |
| ---------------------------- | ------------------------------------- |
| `.flash/artifact.tar.gz` | Deployment package ready for Runpod |
| `.flash/flash_manifest.json` | Service discovery configuration |
| `.flash/.build/` | Build directory (kept for inspection) |
## Cross-platform builds
Flash automatically handles cross-platform builds:
* **Automatic platform targeting**: Dependencies are installed for Linux x86\_64, regardless of your build platform.
* **Binary wheel enforcement**: Only pre-built wheels are used, preventing compilation issues.
### Python version in deployed workers
Flash workers support Python 3.10, 3.11, 3.12, and 3.13. The target version is determined by:
1. **CLI flag:** The `--python-version` flag takes precedence.
2. **Resource config:** The `python_version` parameter on your endpoint configs.
3. **Local interpreter:** Your local Python version (from `sys.version_info`) when neither is specified.
All resources in a Flash app must use the same Python version because Flash ships a single tarball for the entire app. If resources declare conflicting versions, the build fails.
**Breaking change in Flash 0.x:** Flash now matches your local Python version by default instead of always defaulting to Python 3.12. If you upgrade Flash and your local Python differs from 3.12, your first deploy will trigger a rolling release. To maintain consistent behavior across team members, declare `python_version` explicitly on your resource configs or use `--python-version` on the CLI.
| Version | Cold start | Notes |
| ------- | -------------- | ------------------------------------------- |
| 3.12 | No overhead | PyTorch pre-installed in base image |
| 3.13 | +\~7 GB on GPU | Alternative Python install required |
| 3.11 | +\~7 GB on GPU | Alternative Python install required |
| 3.10 | +\~7 GB on GPU | EOL 2026-10-31; consider migrating to 3.11+ |
If your local Python version is not supported (for example, 3.9 or 3.14), the build fails with an actionable error message listing the supported versions.
Image tags follow the pattern `py{version}-{tag}` (for example, `runpod/flash:py3.12-latest`).
## Managing deployment size
Runpod Serverless has a **1.5GB deployment limit**. Flash automatically excludes packages that are pre-installed in the base image:
* `torch`, `torchvision`, `torchaudio`, `triton`
These packages are excluded at archive time, so you don't need to specify them manually.
### Manual exclusions
Use `--exclude` to skip additional packages that are already in a custom base image or not needed:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
flash build --exclude scipy,pandas
```
### Base image reference
| Resource type | Base image | Auto-excluded packages |
| ------------- | ------------ | ---------------------------------------------- |
| GPU | PyTorch base | `torch`, `torchvision`, `torchaudio`, `triton` |
| CPU | Python slim | `torch`, `torchvision`, `torchaudio`, `triton` |
Check the [Flash worker repository](https://github.com/runpod-workers/flash) for current base images and pre-installed packages.
## Troubleshooting
### Build fails with "functions not found"
Ensure your project has `@Endpoint` decorated functions:
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
from runpod_flash import Endpoint, GpuGroup
@Endpoint(name="my-worker", gpu=GpuGroup.ANY)
def my_function(data):
return {"result": data}
```
### Archive is too large
Some CUDA packages (`torch`, `torchvision`, `torchaudio`, `triton`) are auto-excluded. If the archive is still too large, use `--exclude` to skip additional packages or `--no-deps` to skip transitive dependencies:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
flash build --exclude scipy,pandas
```
### Dependency installation fails
If a package doesn't have Linux x86\_64 wheels:
1. Ensure standard pip is installed: `python -m ensurepip --upgrade`
2. Check PyPI for Linux wheel availability.
### Need to examine generated files
The build directory is kept after building. Inspect it with:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
ls .flash/.build/
```
## Related commands
* [`flash deploy`](/flash/cli/deploy) - Build and deploy in one step (includes `--preview` option for local testing)
* [`flash dev`](/flash/cli/dev) - Start development server
* [`flash env`](/flash/cli/env) - Manage environments
Most users should use `flash deploy` instead, which runs build and deploy in one step. Use `flash build` when you need more control or want to inspect the artifact.
# deploy
Source: https://docs.runpod.io/flash/cli/deploy
Build and deploy a Flash application to Runpod Serverless, select an environment, preview the build, and control packaged dependencies.
Build and deploy your Flash application to Runpod Serverless endpoints in one step. This is the primary command for getting your application running in the cloud.
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
flash deploy [OPTIONS]
```
## Examples
Build and deploy a Flash app from the current directory (auto-selects environment if only one exists):
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
flash deploy
```
Deploy to a specific environment:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
flash deploy --env production
```
Deploy with additional excluded packages:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
flash deploy --exclude scipy,pandas
```
Build and test locally before deploying:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
flash deploy --preview
```
## Flags
Target environment name (e.g., `dev`, `staging`, `production`). Auto-selected if only one exists. Creates the environment if it doesn't exist.
Flash app name. Auto-detected from the current directory if not specified.
Skip transitive dependencies during pip install. Useful when the base image already includes dependencies.
Comma-separated packages to exclude (e.g., `torch,torchvision`). Use this to stay under the 1.5GB deployment limit.
Custom archive name for the build artifact.
Build and launch a local Docker-based preview environment instead of deploying to Runpod.
Target Python version for worker images (3.10, 3.11, 3.12, or 3.13). Overrides per-resource `python_version` declarations and local interpreter detection.
## What happens during deployment
1. **Build phase**: Creates the deployment artifact (same as `flash build`).
2. **Environment resolution**: Detects or creates the target environment.
3. **Upload**: Sends the artifact to Runpod storage.
4. **Provisioning**: Creates or updates Serverless endpoints.
5. **Configuration**: Sets up environment variables and service discovery.
## Rolling releases for code changes
When you run `flash deploy` on an already-deployed application, Flash compares your current build against the previous deployment to determine what needs updating.
Flash triggers a rolling release when your source code changes, even if your resource configuration stays the same. During the build phase, Flash computes a fingerprint of your source files. If this fingerprint differs from the previous deployment, Flash treats it as a configuration change and initiates a rolling update to your endpoints.
This means you can iterate on your code without modifying resource configurations like GPU types or worker counts. Run `flash deploy` after making code changes, and Flash rolls out the updated code to your endpoints.
## Architecture
After deployment, your Flash app runs as independent Serverless endpoints on Runpod:
```mermaid theme={"theme":{"light":"github-light","dark":"github-dark"}}
%%{init: {'theme':'base', 'themeVariables': { 'primaryColor':'#9289FE','primaryTextColor':'#fff','primaryBorderColor':'#9289FE','lineColor':'#5F4CFE','secondaryColor':'#AE6DFF','tertiaryColor':'#FCB1FF','edgeLabelBackground':'#5F4CFE', 'fontSize':'14px','fontFamily':'font-inter'}}}%%
flowchart TB
Users(["USERS"])
StateManager["Runpod GraphQL API • Service discovery • Manifest registry"]
subgraph Runpod ["RUNPOD SERVERLESS"]
LB["lb_worker ENDPOINT (load-balanced) • POST /process • GET /health"]
GPU["gpu_worker ENDPOINT (queue-based) • POST /runsync"]
CPU["cpu_worker ENDPOINT (queue-based) • POST /runsync"]
LB <-.->|"inter-endpoint calls"| GPU
LB <-.->|"inter-endpoint calls"| CPU
LB -.->|"service discovery"| StateManager
GPU -.->|"service discovery"| StateManager
CPU -.->|"service discovery"| StateManager
end
Users -->|"call directly"| LB
Users -->|"call directly"| GPU
Users -->|"call directly"| CPU
style Runpod fill:#1a1a2e,stroke:#5F4CFE,stroke-width:2px,color:#fff
style Users fill:#4D38F5,stroke:#4D38F5,color:#fff
style LB fill:#5F4CFE,stroke:#5F4CFE,color:#fff
style GPU fill:#22C55E,stroke:#22C55E,color:#000
style CPU fill:#22C55E,stroke:#22C55E,color:#000
style StateManager fill:#AE6DFF,stroke:#AE6DFF,color:#fff
```
Each resource configuration in your code creates an independent endpoint. You can call any endpoint directly based on your needs.
## App and environment management
### Automatic creation
Flash automatically creates apps and environments as needed during deployment:
* If the app doesn't exist, Flash creates it along with the target environment.
* If only the environment doesn't exist, Flash creates it within the existing app.
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
# Creates the app and 'staging' environment if they don't exist
flash deploy --env staging
```
### Auto-selection
When you have only one environment, it's selected automatically:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
# Auto-selects the only available environment
flash deploy
```
When multiple environments exist, you must specify one:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
# Required when multiple environments exist
flash deploy --env staging
```
### Default environment
If no app or environment exists and none is specified, Flash creates the app with a `production` environment by default.
## Post-deployment
After successful deployment, Flash displays all deployed endpoints:
```text theme={"theme":{"light":"github-light","dark":"github-dark"}}
✓ Deployment Complete
Load-balanced endpoints:
https://abc123xyz.api.runpod.ai (lb_worker)
POST /process
GET /health
Try it:
curl -X POST https://abc123xyz.api.runpod.ai/process \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $RUNPOD_API_KEY" \
-d '{"input_data": {"message": "Hello from Flash"}}'
Queue-based endpoints:
https://api.runpod.ai/v2/def456xyz (gpu_worker)
https://api.runpod.ai/v2/ghi789xyz (cpu_worker)
Try it:
curl -X POST https://api.runpod.ai/v2/def456xyz/runsync \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $RUNPOD_API_KEY" \
-d '{"input": {"input_data": {"message": "Hello from the GPU"}}}'
```
Each endpoint is independent with its own URL and can be called directly.
### Authentication
All deployed endpoints require authentication with your Runpod API key:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
export RUNPOD_API_KEY="your_key_here"
curl -X POST https://YOUR_ENDPOINT_URL/path \
-H "Authorization: Bearer $RUNPOD_API_KEY" \
-H "Content-Type: application/json" \
-d '{"param": "value"}'
```
## Preview mode
Test locally before deploying:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
flash deploy --preview
```
This builds your project and runs it in Docker containers locally:
* Each endpoint runs in its own container.
* All containers communicate via Docker network.
* Endpoints exposed on local ports for testing.
* Press `Ctrl+C` to stop.
## Managing deployment size
Runpod Serverless has a **1.5GB limit**. Flash automatically excludes packages that are pre-installed in the base image (`torch`, `torchvision`, `torchaudio`, `numpy`, `triton`).
If the deployment is still too large, use `--exclude` to skip additional packages:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
flash deploy --exclude scipy,pandas
```
See [`flash build` - Managing deployment size](/flash/cli/build#managing-deployment-size) for more details.
## flash dev vs flash deploy
See [`flash dev`](/flash/cli/dev#flash-dev-vs-flash-deploy) for a detailed comparison of local development vs production deployment.
## Troubleshooting
### Multiple environments error
```text theme={"theme":{"light":"github-light","dark":"github-dark"}}
Error: Multiple environments found: dev, staging, production
```
Specify the target environment:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
flash deploy --env staging
```
### Deployment size limit
Base image packages are auto-excluded. If the deployment is still too large, use `--exclude` to skip additional packages:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
flash deploy --exclude scipy,pandas
```
### Authentication fails
Ensure your API key is set:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
echo $RUNPOD_API_KEY
export RUNPOD_API_KEY="your_key_here"
```
## Related commands
* [`flash build`](/flash/cli/build) - Build without deploying
* [`flash dev`](/flash/cli/dev) - Local development server
* [`flash env`](/flash/cli/env) - Manage environments
* [`flash app`](/flash/cli/app) - Manage applications
* [`flash undeploy`](/flash/cli/undeploy) - Remove endpoints
# dev
Source: https://docs.runpod.io/flash/cli/dev
Run a local Flash development server, test Endpoint functions on Runpod Serverless, and configure provisioning, ports, and updates.
Start the Flash development server for local testing with automatic updates. A local development server provides a unified interface for testing while `@Endpoint` functions execute on Runpod Serverless.
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
flash dev [OPTIONS]
```
`flash run` is a hidden alias for `flash dev` and works identically. New projects should use `flash dev`.
## Example
Start the development server with defaults:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
flash dev
```
Start with auto-provisioning to eliminate cold-start delays:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
flash dev --auto-provision
```
Start on a custom port:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
flash dev --port 3000
```
## Flags
Host address to bind the server to.
Port number for the server. If the port is already in use, Flash automatically tries the next available port.
Enable or disable auto-reload on code changes. Enabled by default.
Auto-provision all Serverless endpoints on startup instead of lazily on first call. Eliminates cold-start delays during development.
## Endpoint descriptions from docstrings
Flash extracts the first line of each function's docstring and uses it in two places:
* **Startup table**: The "Description" column shows the docstring when the server starts.
* **Swagger UI**: The endpoint summary in the API explorer at `/docs`.
Add docstrings to your `@Endpoint` functions to make your API self-documenting:
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
@Endpoint(name="text-processor", gpu=GpuGroup.ANY)
def analyze_text(text: str) -> dict:
"""Analyze text and return sentiment scores."""
# Implementation here
return {"sentiment": "positive"}
```
When you run `flash dev`, the startup table displays "Analyze text and return sentiment scores" as the description for this endpoint, and the same text appears in the Swagger UI summary.
## Architecture
With `flash dev`, Flash starts a local development server alongside remote Serverless endpoints:
```mermaid theme={"theme":{"light":"github-light","dark":"github-dark"}}
%%{init: {'theme':'base', 'themeVariables': { 'primaryColor':'#9289FE','primaryTextColor':'#fff','primaryBorderColor':'#9289FE','lineColor':'#5F4CFE','secondaryColor':'#AE6DFF','tertiaryColor':'#FCB1FF','edgeLabelBackground':'#5F4CFE', 'fontSize':'14px','fontFamily':'font-inter'}}}%%
flowchart TB
Browser(["BROWSER/CURL"])
subgraph Local ["YOUR MACHINE (localhost:8888)"]
DevServer["Development Server • Auto-reload on changes • API explorer at /docs • Routes requests"]
end
subgraph Runpod ["RUNPOD SERVERLESS"]
LB["live-lb_worker"]
GPU["live-gpu_worker"]
CPU["live-cpu_worker"]
end
Browser -->|"HTTP"| DevServer
DevServer -->|"HTTPS"| LB
DevServer -->|"HTTPS"| GPU
DevServer -->|"HTTPS"| CPU
style Local fill:#1a1a2e,stroke:#5F4CFE,stroke-width:2px,color:#fff
style Runpod fill:#1a1a2e,stroke:#5F4CFE,stroke-width:2px,color:#fff
style Browser fill:#4D38F5,stroke:#4D38F5,color:#fff
style DevServer fill:#5F4CFE,stroke:#5F4CFE,color:#fff
style LB fill:#22C55E,stroke:#22C55E,color:#000
style GPU fill:#22C55E,stroke:#22C55E,color:#000
style CPU fill:#22C55E,stroke:#22C55E,color:#000
```
**Key points:**
* A local development server provides a convenient testing interface at `localhost:8888`.
* `@Endpoint` functions deploy to Runpod Serverless with `live-` prefix to distinguish from production.
* Code changes are picked up automatically without restarting the server.
* The development server routes requests to appropriate remote endpoints.
This differs from `flash deploy`, where all endpoints run on Runpod without a local server.
## Auto-provisioning
By default, endpoints are provisioned lazily on first `@Endpoint` function call. Use `--auto-provision` to provision all endpoints at server startup:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
flash dev --auto-provision
```
### How it works
1. **Discovery**: Scans your app for `@Endpoint` decorated functions.
2. **Deployment**: Deploys resources concurrently (up to 3 at a time).
3. **Confirmation**: Asks for confirmation if deploying more than 5 endpoints.
4. **Caching**: Stores deployed resources in `.flash/resources.pkl` for reuse.
5. **Updates**: Recognizes existing endpoints and updates if configuration changed.
### Benefits
* **Zero cold start**: All endpoints ready before you test them.
* **Faster development**: No waiting for deployment on first HTTP call.
* **Resource reuse**: Cached endpoints are reused across server restarts.
### When to use
* Local development with multiple endpoints.
* Testing workflows that call multiple remote functions.
* Debugging where you want deployment separated from handler logic.
## Provisioning modes
| Mode | When endpoints are deployed |
| ------------------ | ---------------------------------- |
| Default (lazy) | On first `@Endpoint` function call |
| `--auto-provision` | At server startup |
## Testing your API
Once the server is running, test your endpoints:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
# Health check
curl http://localhost:8888/
# Call a queue-based GPU endpoint (gpu_worker.py)
curl -X POST http://localhost:8888/gpu_worker/runsync \
-H "Content-Type: application/json" \
-d '{"input": {"input_data": {"message": "Hello from the GPU"}}}'
# Call a load-balanced endpoint (lb_worker.py)
curl -X POST http://localhost:8888/lb_worker/process \
-H "Content-Type: application/json" \
-d '{"input_data": {"message": "Hello from Flash"}}'
```
Queue-based endpoints require the `{"input": {...}}` wrapper format to match deployed endpoint behavior. Load-balanced endpoints accept direct JSON payloads.
Open [http://localhost:8888/docs](http://localhost:8888/docs) for the interactive API explorer.
## Requirements
* `RUNPOD_API_KEY` must be set in your `.env` file or environment.
* A valid Flash project structure (created by `flash init` or manually).
## flash dev vs flash deploy
| Aspect | `flash dev` | `flash deploy` |
| ---------------------------- | ---------------------------------------------------- | ----------------- |
| Local development server | Yes ([http://localhost:8888](http://localhost:8888)) | No |
| `@Endpoint` functions run on | Runpod Serverless | Runpod Serverless |
| Endpoint persistence | Temporary (`live-` prefix) | Persistent |
| Code updates | Automatic reload | Manual redeploy |
| Use case | Development | Production |
## Related commands
* [`flash init`](/flash/cli/init) - Create a new project
* [`flash deploy`](/flash/cli/deploy) - Deploy to production
* [`flash undeploy`](/flash/cli/undeploy) - Remove endpoints
# env
Source: https://docs.runpod.io/flash/cli/env
Create, inspect, list, and delete isolated Flash deployment environments for development, staging, and production applications.
Manage deployment environments for Flash applications. Environments are isolated deployment contexts (like `dev`, `staging`, `production`) within a Flash app.
```bash Command theme={"theme":{"light":"github-light","dark":"github-dark"}}
flash env [OPTIONS]
```
## Subcommands
| Subcommand | Description |
| ---------- | --------------------------------------- |
| `list` | Show all environments for an app |
| `create` | Create a new environment |
| `get` | Show details of an environment |
| `delete` | Delete an environment and its resources |
***
## env list
Show all available environments for an app.
```bash Command theme={"theme":{"light":"github-light","dark":"github-dark"}}
flash env list [OPTIONS]
```
### Example
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
# List environments for current app
flash env list
# List environments for specific app
flash env list --app APP_NAME
```
### Flags
Flash app name. Auto-detected from current directory if not specified.
### Output
```text theme={"theme":{"light":"github-light","dark":"github-dark"}}
┏━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━┓
┃ Name ┃ ID ┃ Active Build ┃ Created At ┃
┡━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━┩
│ dev │ env_abc123 │ build_xyz789 │ 2024-01-15 10:30 │
│ staging │ env_def456 │ build_uvw456 │ 2024-01-16 14:20 │
│ production │ env_ghi789 │ build_rst123 │ 2024-01-20 09:15 │
└────────────┴─────────────────────┴───────────────────┴──────────────────┘
```
***
## env create
Create a new deployment environment.
```bash Command theme={"theme":{"light":"github-light","dark":"github-dark"}}
flash env create [OPTIONS]
```
### Example
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
# Create staging environment
flash env create staging
# Create environment in specific app
flash env create production --app APP_NAME
```
### Arguments
Name for the new environment (e.g., `dev`, `staging`, `production`).
### Flags
Flash app name. Auto-detected from current directory if not specified.
### Notes
* If the app doesn't exist, it's created automatically.
* Environment names must be unique within an app.
* Newly created environments have no active build until first deployment.
You don't always need to create environments explicitly. Running `flash deploy --env ` creates the environment automatically if it doesn't exist.
***
## env get
Show detailed information about a deployment environment.
```bash Command theme={"theme":{"light":"github-light","dark":"github-dark"}}
flash env get [OPTIONS]
```
### Example
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
# Get details for production environment
flash env get production
# Get details for specific app's environment
flash env get staging --app APP_NAME
```
### Arguments
Name of the environment to inspect.
### Flags
Flash app name. Auto-detected from current directory if not specified.
### Output
```text theme={"theme":{"light":"github-light","dark":"github-dark"}}
╭────────────────────────────────────╮
│ Environment: production │
├────────────────────────────────────┤
│ ID: env_ghi789 │
│ State: DEPLOYED │
│ Active Build: build_rst123 │
│ Created: 2024-01-20 09:15:00 │
╰────────────────────────────────────╯
Associated Endpoints
┏━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━┓
┃ Name ┃ ID ┃
┡━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━┩
│ my-gpu │ ep_abc123 │
│ my-cpu │ ep_def456 │
└────────────────┴────────────────────┘
Associated Network Volumes
┏━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━┓
┃ Name ┃ ID ┃
┡━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━┩
│ model-cache │ nv_xyz789 │
└────────────────┴────────────────────┘
```
***
## env delete
Delete a deployment environment and all its associated resources.
```bash Command theme={"theme":{"light":"github-light","dark":"github-dark"}}
flash env delete [OPTIONS]
```
### Examples
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
# Delete development environment
flash env delete dev
# Delete environment in specific app
flash env delete staging --app APP_NAME
```
### Arguments
Name of the environment to delete.
### Flags
Flash app name. Auto-detected from current directory if not specified.
### Process
1. Shows environment details and resources to be deleted.
2. Prompts for confirmation (required).
3. Undeploys all associated endpoints.
4. Removes all associated network volumes.
5. Deletes the environment from the app.
This operation is irreversible. All endpoints, volumes, and configuration associated with the environment will be permanently deleted.
***
## Environment states
| State | Description |
| --------- | ------------------------------------ |
| PENDING | Environment created but not deployed |
| DEPLOYING | Deployment in progress |
| DEPLOYED | Successfully deployed and running |
| FAILED | Deployment or health check failed |
| DELETING | Deletion in progress |
## Common workflows
### Three-tier deployment
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
# Create environments
flash env create dev
flash env create staging
flash env create production
# Deploy to each
flash deploy --env dev
flash deploy --env staging
flash deploy --env production
```
### Feature branch testing
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
# Create feature environment
flash env create FEATURE_NAME
# Deploy feature branch
git checkout FEATURE_NAME
flash deploy --env FEATURE_NAME
# Clean up after merge
flash env delete FEATURE_NAME
```
## Related commands
* [`flash deploy`](/flash/cli/deploy) - Deploy to an environment
* [`flash app`](/flash/cli/app) - Manage applications
* [`flash undeploy`](/flash/cli/undeploy) - Remove specific endpoints
# init
Source: https://docs.runpod.io/flash/cli/init
Initialize a Flash project with a FastAPI server, example GPU and CPU workers, configuration files, and a ready-to-run directory structure.
Create a new Flash project with a ready-to-use template structure including a FastAPI server, example GPU and CPU workers, and configuration files.
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
flash init PROJECT_NAME [OPTIONS]
```
## Example
Create a new project directory:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
flash init PROJECT_NAME
cd PROJECT_NAME
pip install -r requirements.txt
flash dev
```
Initialize in the current directory:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
flash init .
```
## Arguments
Name of the project directory to create. Use `.` to initialize in the current directory.
## Flags
Overwrite existing files if they already exist in the target directory.
## What it creates
The command creates the following project structure:
### Template contents
* **lb\_worker.py**: load-balanced endpoint with HTTP routes. Contains `@Endpoint` functions with custom HTTP methods and paths (e.g., `POST /process`, `GET /health`). Multiple routes can share the same endpoint.
* **gpu\_worker.py**: GPU queue-based endpoint. Contains an `@Endpoint` function that runs on GPU hardware. Provides `/run` or `/runsync` routes for job submission. Creates one Serverless endpoint when deployed.
* **cpu\_worker.py**: CPU queue-based endpoint. Contains an `@Endpoint` function that runs on CPU-only instances. Provides `/run` or `/runsync` routes for job submission. Creates one Serverless endpoint when deployed.
* **.env**: Template for environment variables including `RUNPOD_API_KEY`.
### AI coding agent files
Flash generates context files that help AI coding assistants (Claude Code, Cursor, GitHub Copilot, Codex, Aider, and others) use Flash correctly. These files tell agents to use Flash CLI commands instead of raw Runpod API calls.
| File | Purpose |
| ----------- | ---------------------------------------------------------------------------------- |
| `AGENTS.md` | CLI-first rules for AI coding tools (Cursor, Codex, Aider, Amp, Jules, and others) |
| `CLAUDE.md` | Symlink to `AGENTS.md` so Claude Code picks up the same rules |
Flash writes these files only when they don't already exist. If you have your own `AGENTS.md` or `CLAUDE.md`, Flash leaves them alone.
**Tools using other conventions:** GitHub Copilot reads `.github/copilot-instructions.md` and Cursor (legacy) reads `.cursorrules`. If you use those, symlink or copy `AGENTS.md`:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
ln -s ../AGENTS.md .github/copilot-instructions.md
ln -s AGENTS.md .cursorrules
```
### Add agent files to existing projects
If you've already run `flash init`, add the agent files with:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
python -c "from runpod_flash.rules import install_agent_files; from pathlib import Path; install_agent_files(Path.cwd())"
```
### Opt out
Delete `AGENTS.md`. Flash won't re-create it.
## Next steps
After initialization:
1. Copy `.env.example` to `.env` (if needed) and add your `RUNPOD_API_KEY`.
2. Install dependencies: `pip install -r requirements.txt`
3. Start the development server: `flash dev`
4. Open [http://localhost:8888/docs](http://localhost:8888/docs) to explore the API.
5. Customize the workers for your use case.
6. Deploy with `flash deploy` when ready.
This command only creates local files. It doesn't interact with Runpod or create any cloud resources. Cloud resources are created when you run `flash dev` or `flash deploy`.
## Related commands
* [`flash dev`](/flash/cli/dev) - Start the development server
* [`flash deploy`](/flash/cli/deploy) - Build and deploy to Runpod
# login
Source: https://docs.runpod.io/flash/cli/login
Authenticate the Flash CLI with Runpod, save your API key locally, and configure browser-based authorization options and timeouts.
Authenticate with Runpod and save your API key for all Flash operations, including CLI commands and standalone `@Endpoint` functions.
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
flash login [OPTIONS]
```
## Example
Authenticate with Runpod (opens browser automatically):
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
flash login
```
The command opens your default browser to the Runpod authorization page. After you approve the request, your API key is saved locally for future CLI operations.
## How it works
1. Flash generates an authorization request.
2. Your browser opens to the Runpod console authorization page.
3. You approve the request in your browser.
4. Flash saves your API key to `~/.runpod/config.toml`.
## Flags
Don't automatically open the browser. Instead, manually copy the authorization URL and open it yourself.
Maximum time in seconds to wait for authorization. Default is 600 seconds (10 minutes).
Force re-authentication even if credentials already exist. By default, `flash login` skips the browser flow when it finds existing credentials.
## Credential storage
After successful login, your API key is saved to `~/.runpod/config.toml`. This file is used by:
* All Flash CLI commands (`flash dev`, `flash deploy`, etc.)
* Standalone Python scripts using `@Endpoint` functions
* Any code using the Flash SDK
Keep your API key secure. Never commit it to version control. The credentials file is stored in your home directory, outside of project directories.
## Alternative: Environment variable authentication
Instead of using `flash login`, you can set your API key directly as an environment variable:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
export RUNPOD_API_KEY=your_api_key_here
```
Or add it to your project's `.env` file for local CLI use:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
RUNPOD_API_KEY=your_api_key_here
```
Generate an API key from [Settings > API Keys](https://console.runpod.io/user/settings) in the Runpod console.
## Credential resolution priority
Flash checks for credentials in this order:
1. `RUNPOD_API_KEY` environment variable
2. `RUNPOD_API_KEY` in a `.env` file (loaded automatically at startup)
3. Credentials file (`~/.runpod/config.toml`)
The first valid credential found is used. Environment variables take precedence over the credentials file, so you can override per-project or per-session without modifying stored credentials.
Values in your `.env` file are only used for local CLI commands and development. They are **not** passed to deployed endpoints. To set environment variables on deployed endpoints, use the `env` parameter in your endpoint configuration. See [Endpoint parameters](/flash/configuration/parameters#env) for details.
Your Runpod API key needs **All** access permissions.
## Related commands
* [`flash init`](/flash/cli/init) - Create a new Flash project
* [`flash dev`](/flash/cli/dev) - Start the development server
* [`flash deploy`](/flash/cli/deploy) - Build and deploy to Runpod
# CLI overview
Source: https://docs.runpod.io/flash/cli/overview
Learn how to use the Flash CLI for local development and deployment. Review setup, configuration, deployment, and usage guidance for Runpod Flash.
The Flash CLI provides commands for initializing projects, running local development servers, building deployment artifacts, and managing your applications on Runpod Serverless.
Before using the CLI, make sure you've [installed Flash](/flash/overview#install-flash).
## Available commands
| Command | Description |
| --------------------------------------- | --------------------------------------------------------- |
| [`flash init`](/flash/cli/init) | Create a new Flash project with a template structure |
| [`flash login`](/flash/cli/login) | Authenticate with Runpod using your API key |
| [`flash dev`](/flash/cli/dev) | Start the local development server with automatic updates |
| [`flash build`](/flash/cli/build) | Build a deployment artifact without deploying |
| [`flash deploy`](/flash/cli/deploy) | Build and deploy your application to Runpod |
| [`flash env`](/flash/cli/env) | Manage deployment environments |
| [`flash app`](/flash/cli/app) | Manage Flash applications |
| [`flash undeploy`](/flash/cli/undeploy) | Remove deployed endpoints |
| [`flash update`](/flash/cli/update) | Update Flash to the latest or a specific version |
## Getting help
View help for any command by adding `--help`:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
flash --help
flash deploy --help
flash env --help
```
## Authentication
Authenticate with Runpod:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
flash login
```
This command opens your browser to authenticate with Runpod. After you approve the request, your API key is saved locally for future CLI operations. See [`flash login`](/flash/cli/login) for details.
Alternatively, set the `RUNPOD_API_KEY` environment variable:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
export RUNPOD_API_KEY=your_api_key_here
```
## Using uv
If you installed Flash with [uv](https://docs.astral.sh/uv/), prefix all Flash commands with `uv run`:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
uv run flash login
uv run flash dev
uv run flash deploy
```
## Common workflows
### Local development
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
# Create a new project
flash init PROJECT_NAME
cd PROJECT_NAME
# Install dependencies
pip install -r requirements.txt
# Add your API key to .env
# Start the development server
flash dev
```
### Deploy to production
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
# Build and deploy
flash deploy
# Deploy to a specific environment
flash deploy --env ENVIRONMENT_NAME
```
### Manage deployments
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
# List environments
flash env list
# Check environment status
flash env get ENVIRONMENT_NAME
# Remove an environment
flash env delete ENVIRONMENT_NAME
```
### Clean up endpoints
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
# List deployed endpoints
flash undeploy list
# Remove specific endpoint
flash undeploy ENDPOINT_NAME
# Remove all endpoints
flash undeploy --all
```
# undeploy
Source: https://docs.runpod.io/flash/cli/undeploy
List or delete Runpod Serverless endpoints created by Flash, including individual deployments and local development resources.
Manage and delete Runpod Serverless endpoints deployed via Flash. Use this command to clean up endpoints created during local development with `flash dev`.
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
flash undeploy [NAME|list] [OPTIONS]
```
## Example
List all tracked endpoints:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
flash undeploy list
```
Remove a specific endpoint:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
flash undeploy ENDPOINT_NAME
```
Remove all endpoints:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
flash undeploy --all
```
## Usage modes
### List endpoints
Display all tracked endpoints with their current status:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
flash undeploy list
```
Output includes:
* **Name**: Endpoint name
* **Endpoint ID**: Runpod endpoint identifier
* **Status**: Current health status (Active/Inactive/Unknown)
* **Type**: Resource type (Live Serverless, Cpu Live Serverless, etc.)
**Status indicators:**
| Status | Meaning |
| -------- | ----------------------------------------------- |
| Active | Endpoint is running and responding |
| Inactive | Tracking exists but endpoint deleted externally |
| Unknown | Error during health check |
### Undeploy by name
Delete a specific endpoint:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
flash undeploy ENDPOINT_NAME
```
This:
1. Searches for endpoints matching the name.
2. Shows endpoint details.
3. Prompts for confirmation.
4. Deletes the endpoint from Runpod.
5. Removes from local tracking.
### Undeploy all
Delete all tracked endpoints (requires double confirmation):
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
flash undeploy --all
```
Safety features:
1. Shows total count of endpoints.
2. First confirmation: Yes/No prompt.
3. Second confirmation: Type "DELETE ALL" exactly.
4. Deletes all endpoints from Runpod.
5. Removes all from tracking.
### Interactive selection
Select endpoints to undeploy using checkboxes:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
flash undeploy --interactive
```
Use arrow keys to navigate, space bar to select/deselect, and Enter to confirm.
### Clean up stale tracking
Remove inactive endpoints from tracking without API deletion:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
flash undeploy --cleanup-stale
```
Use this when endpoints were deleted via the Runpod console or API (not through Flash). The local tracking file (`.flash/resources.pkl`) becomes stale, and this command cleans it up.
## Flags
Undeploy all tracked endpoints. Requires double confirmation for safety.
Interactive checkbox selection mode. Select multiple endpoints to undeploy.
Remove inactive endpoints from local tracking without attempting API deletion. Use when endpoints were deleted externally.
## Arguments
Name of the endpoint to undeploy. Use `list` to show all endpoints.
## undeploy vs env delete
| Command | Scope | When to use |
| ------------------ | ---------------------------------------- | ------------------------------------- |
| `flash undeploy` | Individual endpoints from local tracking | Development cleanup, granular control |
| `flash env delete` | Entire environment + all resources | Production cleanup, full teardown |
For production deployments, use `flash env delete` to remove entire environments and all associated resources.
## How tracking works
Flash tracks deployed endpoints in `.flash/resources.pkl`. Endpoints are added when you:
* Run `flash dev --auto-provision`
* Run `flash dev` and call `@Endpoint` functions
* Run `flash deploy`
The tracking file is in `.gitignore` and should never be committed. It contains local deployment state.
## Common workflows
### Basic cleanup
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
# Check what's deployed
flash undeploy list
# Remove a specific endpoint
flash undeploy ENDPOINT_NAME
# Clean up stale tracking
flash undeploy --cleanup-stale
```
### Bulk operations
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
# Undeploy all endpoints
flash undeploy --all
# Interactive selection
flash undeploy --interactive
```
### Managing external deletions
If you delete endpoints via the Runpod console:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
# Check status - will show as "Inactive"
flash undeploy list
# Remove stale tracking entries
flash undeploy --cleanup-stale
```
## Troubleshooting
### Endpoint shows as "Inactive"
The endpoint was deleted via Runpod console or API. Clean up:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
flash undeploy --cleanup-stale
```
### Can't find endpoint by name
Check the exact name:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
flash undeploy list
```
### Undeploy fails with API error
1. Check `RUNPOD_API_KEY` in `.env`.
2. Verify network connectivity.
3. Check if the endpoint still exists on Runpod.
## Related commands
* [`flash dev`](/flash/cli/dev) - Development server (creates endpoints)
* [`flash deploy`](/flash/cli/deploy) - Deploy to Runpod
* [`flash env delete`](/flash/cli/env) - Delete entire environment
# update
Source: https://docs.runpod.io/flash/cli/update
Update the Flash CLI to the latest or a selected version, understand automatic update checks, and configure when those checks run.
Update the Flash CLI to the latest version or a specific version. The command fetches version information from PyPI and installs using uv (when available) or pip.
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
flash update [OPTIONS]
```
## Examples
Update to the latest version:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
flash update
```
Update to a specific version:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
flash update --version 1.4.0
```
## Flags
Target version to install. If not specified, updates to the latest version available on PyPI.
## Automatic update checks
Flash checks for newer versions in the background when you run most commands. If an update is available, a notice appears after the command completes:
```
A new version of runpod-flash is available: 2.0.0
Run 'flash update' to upgrade.
```
This check runs at most once every 24 hours and is cached locally to `~/.config/runpod/update_check.json`.
### Excluded commands
The background check does not run for:
* `flash dev` - Long-running development server where the notice would appear at an unpredictable time.
* `flash update` - Already managing versions directly.
### Disabling update checks
Set the `FLASH_NO_UPDATE_CHECK` environment variable to skip automatic update checks:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
export FLASH_NO_UPDATE_CHECK=1
```
Update checks are also skipped automatically in CI environments (when the `CI` environment variable is set) and in non-interactive sessions (when neither stdout nor stderr is a TTY).
## Related commands
* [`flash --version`](/flash/cli/overview#getting-help) - Check your current Flash version
# Configuration best practices
Source: https://docs.runpod.io/flash/configuration/best-practices
Recommended configurations for production, development, and cost optimization. Review configuration and usage details for Runpod Flash.
This guide provides best practices for configuring Flash endpoints based on your use case. Recommendations are organized by workload type and optimization goal.
## Production workloads
Here are some best practices for production deployments requiring reliability and consistent performance:
### General recommendations
* **Pin specific GPU types** instead of using `GpuGroup.ANY` for predictable performance and costs.
* **Use network volumes** for large models to avoid downloading on each worker startup.
* **Set appropriate `execution_timeout_ms`** to prevent runaway jobs and control costs.
* **Use environment variables** for configuration and secrets, not hardcoded values.
### Queue-based endpoints
Queue-based endpoints handle asynchronous batch processing where jobs can wait in queue:
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
from runpod_flash import Endpoint, GpuType, NetworkVolume
@Endpoint(
name="production-batch",
gpu=GpuType.NVIDIA_A100_80GB_PCIe, # Specific GPU for predictable performance
workers=(1, 10), # At least 1 worker, scale up to 10
idle_timeout=1200, # 20 minutes - keep workers longer for variable traffic
execution_timeout_ms=600000, # 10 minute timeout
volume=NetworkVolume(name="my-volume"),
env={"MODEL_PATH": "/runpod-volume/models"}
)
def process_batch(data): ...
```
**Key settings**:
* `workers=(1, n)`: Set min to 1 to avoid cold starts for first job in queue.
* `workers=(n, max)`: Set max based on expected peak concurrent jobs.
* `idle_timeout`: 900-1800 seconds (15-30 minutes) for production workloads.
### Load-balanced endpoints
Load-balanced endpoints handle synchronous HTTP requests where immediate response is critical:
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
from runpod_flash import Endpoint, GpuType, NetworkVolume
api = Endpoint(
name="production-api",
gpu=GpuType.NVIDIA_GEFORCE_RTX_4090, # Specific GPU for consistent performance
workers=(3, 20), # Always keep 3 workers ready, scale to 20
idle_timeout=1800, # 30 minutes - keep workers active longer
execution_timeout_ms=60000, # 60 second timeout per request
volume=NetworkVolume(name="my-volume")
)
@api.post("/process")
async def process_request(data: dict) -> dict:
return {"result": "processed"}
@api.get("/health")
async def health_check() -> dict:
return {"status": "healthy"}
```
**Key settings**:
* `workers=(n, max)`: Set min ≥ 1 for production APIs to avoid cold starts. Unlike queue-based endpoints where jobs can wait, API clients expect immediate responses.
* `workers=(min, n)`: Set max based on expected peak concurrent requests.
* `idle_timeout`: 1200-1800 seconds (20-30 minutes) to keep workers ready.
* Include health check routes (e.g., `GET /health`) for monitoring.
## Development
Here are some best practices for development and testing environments prioritizing fast iteration:
### General recommendations
* **Use `GpuGroup.ANY`** for fastest GPU provisioning during development.
* **Set `workers=(0, n)`** to minimize costs when not actively testing.
* **Keep max workers low** (1-3) to control development expenses.
* **Use short `idle_timeout`** (300 seconds / 5 minutes) to scale down quickly between test runs.
* **Test locally** with `flash dev` before deploying to production.
### Example configuration
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
from runpod_flash import Endpoint, GpuGroup
@Endpoint(
name="dev-testing",
gpu=GpuGroup.ANY, # Fast provisioning
workers=(0, 2), # Scale to zero, limit to 2 concurrent
idle_timeout=300 # 5 minutes - quick scale-down
)
def test_function(data): ...
```
## Cost optimization
Here are some best practices for minimizing costs on infrequent or batch workloads:
### General recommendations
* **Set `workers=(0, n)`** to scale to zero when idle (no usage = no cost).
* **Use smaller GPU types** when workload allows (e.g., `GpuType.NVIDIA_GEFORCE_RTX_4090` instead of `GpuType.NVIDIA_A100_80GB_PCIe`).
* **Use CPU endpoints** when GPU acceleration isn't needed.
* **Reduce `idle_timeout`** for sporadic workloads (300-600 seconds / 5-10 minutes).
* **Batch operations** into fewer job submissions when possible.
### Cost-optimized queue-based endpoint
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
from runpod_flash import Endpoint, GpuType, NetworkVolume
@Endpoint(
name="batch-job",
gpu=GpuType.NVIDIA_GEFORCE_RTX_4090, # Cost-effective GPU
workers=(0, 5), # Scale to zero, controlled max
idle_timeout=300, # 5 minutes - fast scale-down
volume=NetworkVolume(name="my-volume") # Avoid re-downloading models
)
def batch_process(data): ...
```
### Cost-optimized CPU endpoint
For workloads that don't require GPU acceleration:
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
from runpod_flash import Endpoint
@Endpoint(
name="cpu-batch",
cpu="cpu5c-4-8", # 4 vCPU, 8GB RAM
workers=(0, 3), # Scale to zero, limit to 3
idle_timeout=300 # 5 minutes - fast scale-down
)
def cpu_process(data): ...
```
## Configuration trade-offs
Understanding the trade-offs helps you balance cost, latency, and performance:
| Configuration | Cost | Cold Start Latency | Best For |
| ---------------- | ------- | -------------------------- | --------------------------------------------- |
| `workers=(0, n)` | Lowest | 20-90 seconds first run | Batch jobs, development, infrequent workloads |
| `workers=(1, n)` | Medium | \<1 second for queued jobs | Production batch, variable traffic |
| `workers=(3, n)` | Highest | Always ready | Production APIs, high-traffic endpoints |
| GPU Choice | Cost | Availability | Best For |
| ------------------------------------------------------- | ----------- | ------------ | -------------------------------------- |
| `GpuGroup.ANY` | Variable | Highest | Development, fastest provisioning |
| Specific type (e.g., `GpuType.NVIDIA_GEFORCE_RTX_4090`) | Predictable | Medium | Production with specific hardware |
| Specific type (e.g., `GpuType.NVIDIA_A100_80GB_PCIe`) | Predictable | Lower | Production requiring specific hardware |
## Configuration checklist
Before deploying to production, verify:
* **GPU selection**: Using specific GPU types (not `GpuGroup.ANY`) for predictable performance
* **Worker scaling**: `workers=(1, n)` or higher min for load balancers and latency-sensitive workloads
* **Timeouts**: `execution_timeout_ms` set appropriately for your workload
* **Storage**: Network volume attached if using large models or datasets
* **Environment variables**: All configuration and secrets passed via `env` parameter
* **Monitoring**: Health check routes implemented (load balancers)
* **Testing**: Tested locally with `flash dev` before production deployment
# CPU types
Source: https://docs.runpod.io/flash/configuration/cpu-types
Available CPU instance types for Flash endpoints. Review setup, configuration, deployment, and usage guidance for Runpod Flash.
Flash provides access to CPU-only compute instances for workloads that don't require GPU acceleration. This reference lists all available CPU instance types.
## Using CPU instances
Specify a CPU instance using the `cpu` parameter. You can use either a string shorthand or the `CpuInstanceType` enum:
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
from runpod_flash import Endpoint, CpuInstanceType
# String shorthand
@Endpoint(name="data-processor", cpu="cpu5c-4-8")
async def process(data: dict) -> dict:
...
# Using enum
@Endpoint(name="data-processor", cpu=CpuInstanceType.CPU5C_4_8)
async def process(data: dict) -> dict:
...
```
## Available CPU instance types
CPU instances are organized by generation and optimization profile.
### 5th generation compute-optimized
Latest generation, optimized for compute-intensive workloads:
| CpuInstanceType | ID | vCPU | RAM | Best For |
| --------------- | ---------- | ---- | ---- | -------------------------------- |
| `CPU5C_1_2` | cpu5c-1-2 | 1 | 2GB | Lightweight APIs, simple tasks |
| `CPU5C_2_4` | cpu5c-2-4 | 2 | 4GB | Small APIs, data validation |
| `CPU5C_4_8` | cpu5c-4-8 | 4 | 8GB | General APIs, data processing |
| `CPU5C_8_16` | cpu5c-8-16 | 8 | 16GB | Heavy processing, parallel tasks |
### 3rd generation compute-optimized
Balanced compute focus:
| CpuInstanceType | ID | vCPU | RAM | Best For |
| --------------- | ---------- | ---- | ---- | ------------------------- |
| `CPU3C_1_2` | cpu3c-1-2 | 1 | 2GB | Basic endpoints, webhooks |
| `CPU3C_2_4` | cpu3c-2-4 | 2 | 4GB | Simple data processing |
| `CPU3C_4_8` | cpu3c-4-8 | 4 | 8GB | Moderate workloads |
| `CPU3C_8_16` | cpu3c-8-16 | 8 | 16GB | CPU-intensive tasks |
### 3rd generation general purpose
Balanced CPU and memory:
| CpuInstanceType | ID | vCPU | RAM | Best For |
| --------------- | ---------- | ---- | ---- | --------------------------- |
| `CPU3G_1_4` | cpu3g-1-4 | 1 | 4GB | Memory-light tasks |
| `CPU3G_2_8` | cpu3g-2-8 | 2 | 8GB | General workloads |
| `CPU3G_4_16` | cpu3g-4-16 | 4 | 16GB | Memory-intensive processing |
| `CPU3G_8_32` | cpu3g-8-32 | 8 | 32GB | High-memory workloads |
## Common configurations
### APIs and webhooks
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
# Lightweight API
@Endpoint(name="webhook", cpu="cpu5c-2-4")
async def handle_webhook(data: dict) -> dict:
...
# Production API
@Endpoint(name="api", cpu="cpu5c-4-8", workers=(1, 10))
async def handle_request(data: dict) -> dict:
...
```
### Data processing
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
# Light processing
@Endpoint(name="processor", cpu="cpu3g-2-8") # More RAM per vCPU
async def process(data: dict) -> dict:
...
# Heavy processing
@Endpoint(name="heavy-processor", cpu="cpu5c-8-16")
async def heavy_process(data: dict) -> dict:
...
```
### Memory-intensive tasks
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
# High memory requirement
@Endpoint(name="memory-worker", cpu="cpu3g-8-32") # 8 vCPU, 32GB RAM
async def process_large_data(data: dict) -> dict:
...
```
### Load-balanced CPU API
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
from runpod_flash import Endpoint
api = Endpoint(
name="cpu-api",
cpu="cpu5c-4-8",
workers=(1, 10)
)
@api.post("/process")
async def process(data: dict) -> dict:
return {"result": "processed"}
@api.get("/health")
async def health():
return {"status": "ok"}
```
## Container disk sizing
CPU endpoints automatically adjust container disk size based on instance limits:
* `CPU3G` and `CPU3C` instances: vCPU count × 10GB (e.g., 2 vCPU = 20GB)
* `CPU5C` instances: vCPU count × 15GB (e.g., 4 vCPU = 60GB)
If you specify a custom size via `PodTemplate` that exceeds the instance limit, deployment will fail with a validation error.
# GPU types
Source: https://docs.runpod.io/flash/configuration/gpu-types
Available GPU pools and specific GPU types for Flash endpoints. Review setup, configuration, deployment, and usage guidance for Runpod Flash.
Flash provides access to a wide range of NVIDIA GPUs through both pool-based and specific GPU selection. This page lists all available GPU types and explains how to use them.
## GPU selection methods
Flash offers two ways to specify GPU hardware:
1. [GPU pools](/flash/configuration/gpu-types#gpu-pools) (`GpuGroup`): Select from predefined pools of similar GPUs grouped by architecture and VRAM.
2. [Specific GPU types](/flash/configuration/gpu-types#specific-gpu-types) (`GpuType`): Target exact GPU models when you need precise hardware characteristics.
You can use either method or mix both for [advanced fallback strategies](/flash/configuration/gpu-types#advanced-fallback-strategies).
## GPU pools
The `GpuGroup` enum provides access to GPU pools. Each pool contains specific GPU models grouped by architecture and VRAM capacity.
### Available GPU pools
| GpuGroup | GPUs Included | VRAM | Best For |
| ------------------------ | --------------------------------------------------- | ------- | -------------------------------- |
| `GpuGroup.ANY` | Any available GPU | Varies | Fast provisioning, prototyping |
| `GpuGroup.AMPERE_16` | RTX A4000, RTX 4000 Ada, RTX 2000 Ada | 16GB | Small models, basic inference |
| `GpuGroup.AMPERE_24` | RTX A4500, RTX A5000, RTX 3090 | 20-24GB | General ML, mid-size models |
| `GpuGroup.ADA_24` | L4, RTX 4090 | 24GB | Cost-effective inference |
| `GpuGroup.ADA_32_PRO` | RTX 5090 | 32GB | Latest consumer flagship |
| `GpuGroup.AMPERE_48` | A40, RTX A6000 | 48GB | Large models, fine-tuning |
| `GpuGroup.ADA_48_PRO` | L40S, L40, RTX 6000 Ada | 48GB | Professional inference |
| `GpuGroup.AMPERE_80` | A100 80GB PCIe, A100-SXM4-80GB | 80GB | XL models, intensive training |
| `GpuGroup.ADA_80_PRO` | H100 80GB HBM3 | 80GB | Cutting-edge inference |
| `GpuGroup.BLACKWELL_96` | RTX PRO 6000 Blackwell (Server, Workstation, Max-Q) | 96GB | Professional Blackwell workloads |
| `GpuGroup.HOPPER_141` | H200 | 141GB | Largest models, maximum VRAM |
| `GpuGroup.BLACKWELL_180` | B200 | 180GB | Maximum VRAM, next-gen training |
### Using GPU pools
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
from runpod_flash import Endpoint, GpuGroup
# Single GPU pool
@Endpoint(name="inference", gpu=GpuGroup.AMPERE_80)
async def infer(data: dict) -> dict:
...
# Multiple pools for fallback
@Endpoint(
name="flexible",
gpu=[GpuGroup.AMPERE_80, GpuGroup.AMPERE_48, GpuGroup.ADA_24]
)
async def flexible_infer(data: dict) -> dict:
...
# Any available GPU (fastest provisioning)
@Endpoint(name="development", gpu=GpuGroup.ANY)
async def dev_infer(data: dict) -> dict:
...
```
## Specific GPU types
The `GpuType` enum provides access to specific GPU models. Use these when you need exact hardware characteristics.
### Available GPU types
| GpuType | GPU Model | VRAM | Architecture |
| ----------------------------------------------------------------- | ------------------------------------------------------- | ------ | ------------ |
| `GpuType.ANY` | Any available GPU | Varies | Any |
| `GpuType.NVIDIA_RTX_A4000` | NVIDIA RTX A4000 | 16GB | Ampere |
| `GpuType.NVIDIA_RTX_A4500` | NVIDIA RTX A4500 | 20GB | Ampere |
| `GpuType.NVIDIA_RTX_4000_ADA_GENERATION` | NVIDIA RTX 4000 Ada | 16GB | Ada Lovelace |
| `GpuType.NVIDIA_RTX_2000_ADA_GENERATION` | NVIDIA RTX 2000 Ada | 16GB | Ada Lovelace |
| `GpuType.NVIDIA_RTX_A5000` | NVIDIA RTX A5000 | 24GB | Ampere |
| `GpuType.NVIDIA_L4` | NVIDIA L4 | 24GB | Ada Lovelace |
| `GpuType.NVIDIA_GEFORCE_RTX_3090` | NVIDIA GeForce RTX 3090 | 24GB | Ampere |
| `GpuType.NVIDIA_GEFORCE_RTX_4090` | NVIDIA GeForce RTX 4090 | 24GB | Ada Lovelace |
| `GpuType.NVIDIA_GEFORCE_RTX_5090` | NVIDIA GeForce RTX 5090 | 32GB | Blackwell |
| `GpuType.NVIDIA_A40` | NVIDIA A40 | 48GB | Ampere |
| `GpuType.NVIDIA_RTX_A6000` | NVIDIA RTX A6000 | 48GB | Ampere |
| `GpuType.NVIDIA_RTX_6000_ADA_GENERATION` | NVIDIA RTX 6000 Ada | 48GB | Ada Lovelace |
| `GpuType.NVIDIA_A100_80GB_PCIe` | NVIDIA A100 80GB PCIe | 80GB | Ampere |
| `GpuType.NVIDIA_A100_SXM4_80GB` | NVIDIA A100-SXM4-80GB | 80GB | Ampere |
| `GpuType.NVIDIA_H100_80GB_HBM3` | NVIDIA H100 80GB HBM3 | 80GB | Hopper |
| `GpuType.NVIDIA_RTX_PRO_6000_BLACKWELL_SERVER_EDITION` | NVIDIA RTX PRO 6000 Blackwell Server Edition | 96GB | Blackwell |
| `GpuType.NVIDIA_RTX_PRO_6000_BLACKWELL_WORKSTATION_EDITION` | NVIDIA RTX PRO 6000 Blackwell Workstation Edition | 96GB | Blackwell |
| `GpuType.NVIDIA_RTX_PRO_6000_BLACKWELL_MAX_Q_WORKSTATION_EDITION` | NVIDIA RTX PRO 6000 Blackwell Max-Q Workstation Edition | 96GB | Blackwell |
| `GpuType.NVIDIA_H200` | NVIDIA H200 | 141GB | Hopper |
| `GpuType.NVIDIA_B200` | NVIDIA B200 | 180GB | Blackwell |
### Using specific GPU types
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
from runpod_flash import Endpoint, GpuType
# Single specific GPU
@Endpoint(name="inference", gpu=GpuType.NVIDIA_A100_80GB_PCIe)
async def infer(data: dict) -> dict:
...
# Multiple specific GPUs (fallback strategy)
@Endpoint(
name="flexible",
gpu=[
GpuType.NVIDIA_A100_80GB_PCIe, # Try A100 PCIe first
GpuType.NVIDIA_A100_SXM4_80GB, # Fall back to A100 SXM4
GpuType.NVIDIA_A40 # Final fallback to A40
]
)
async def flexible_infer(data: dict) -> dict:
...
```
## Advanced fallback strategies
Combine `GpuGroup` and `GpuType` for robust availability:
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
from runpod_flash import Endpoint, GpuGroup, GpuType
@Endpoint(
name="hybrid-selection",
gpu=[
GpuType.NVIDIA_A100_80GB_PCIe, # Specific GPU first
GpuGroup.AMPERE_48, # Pool fallback
GpuGroup.ANY # Ultimate fallback
]
)
async def infer(data: dict) -> dict:
...
```
## GPU selection behavior
**Single GPU type:**
Flash waits for this specific GPU to become available. Jobs stay in queue until capacity is available.
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
gpu=GpuGroup.AMPERE_80 # Only A100 80GB
```
**Multiple GPU types (fallback):**
Flash attempts to provision in the order specified.
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
gpu=[GpuGroup.AMPERE_80, GpuGroup.AMPERE_48, GpuGroup.ADA_24]
# Tries: A100 → A40/A6000 → RTX 4090
```
**GpuGroup.ANY:**
Flash selects the first available GPU based on current capacity.
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
gpu=GpuGroup.ANY # Fastest provisioning, unpredictable GPU type
```
**For production**: Use specific GPU types for predictable cost and performance.
**For development**: Use `GpuGroup.ANY` for fastest iteration.
## Multi-GPU workers
Request multiple GPUs per worker using `gpu_count`:
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
@Endpoint(
name="multi-gpu-training",
gpu=GpuGroup.AMPERE_80,
gpu_count=4, # Each worker gets 4 GPUs
workers=2 # Maximum 2 workers = 8 GPUs total
)
async def train(data: dict) -> dict:
...
```
## Handling unavailability
If requested GPUs are unavailable, jobs stay in queue:
```text theme={"theme":{"light":"github-light","dark":"github-dark"}}
Initial job status: IN_QUEUE
[Waiting for capacity...]
```
**Solutions:**
1. **Add fallback options**: Use multiple GPU types.
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
gpu=[GpuGroup.AMPERE_80, GpuGroup.AMPERE_48, GpuGroup.ADA_24]
```
2. **Use broader selection**: Switch to `GpuGroup.ANY`.
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
gpu=GpuGroup.ANY
```
3. **Contact support**: For capacity guarantees, contact [Runpod support](https://www.runpod.io/contact).
# Endpoint parameters
Source: https://docs.runpod.io/flash/configuration/parameters
Complete reference for all Endpoint class parameters. Review setup, configuration, deployment, and usage guidance for Runpod Flash.
This page provides a complete reference for all parameters available on the `Endpoint` class.
## Parameter overview
| Parameter | Type | Description | Default |
| ---------------------- | ------------------------------ | ------------------------------------------------- | ----------------------------- |
| `name` | `str` | Endpoint name (required unless `id=` is used) | - |
| `id` | `str` | Connect to existing endpoint by ID | `None` |
| `gpu` | `GpuGroup`, `GpuType`, or list | GPU type(s) for the endpoint | `GpuGroup.ANY` |
| `cpu` | `str` or `CpuInstanceType` | CPU instance type (mutually exclusive with `gpu`) | `None` |
| `workers` | `int` or `(min, max)` | Worker scaling configuration | `(0, 1)` |
| `idle_timeout` | `int` | Seconds before scaling down idle workers | `60` |
| `dependencies` | `list[str]` | Python packages to install | `None` |
| `system_dependencies` | `list[str]` | System packages to install (apt) | `None` |
| `accelerate_downloads` | `bool` | Enable download acceleration | `True` |
| `volume` | `NetworkVolume` or list | Network volume(s) for persistent storage | `None` |
| `datacenter` | `DataCenter`, list, or `None` | Datacenter(s) for deployment | `None` (all DCs) |
| `env` | `dict[str, str]` | Environment variables | `None` |
| `gpu_count` | `int` | GPUs per worker | `1` |
| `execution_timeout_ms` | `int` | Max execution time in milliseconds | `0` (no limit) |
| `flashboot` | `bool` | Enable Flashboot fast startup | `True` |
| `image` | `str` | Custom Docker image to deploy | `None` |
| `scaler_type` | `ServerlessScalerType` | Scaling strategy | auto |
| `scaler_value` | `int` | Scaling threshold | `4` |
| `template` | `PodTemplate` | Pod template overrides | `None` |
| `min_cuda_version` | `str` or `CudaVersion` | Minimum CUDA version for GPU host selection | `"12.8"` (GPU) / `None` (CPU) |
| `python_version` | `str` | Python version for the worker image | Local Python |
## Parameter details
### name
**Type**: `str`
**Required**: Yes (unless `id=` is specified)
The endpoint name visible in the [Runpod console](https://console.runpod.io/serverless). Use descriptive names to easily identify endpoints.
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
@Endpoint(name="ml-inference-prod", gpu=GpuGroup.ANY)
async def infer(data): ...
```
Use naming conventions like `image-generation-prod` or `batch-processor-dev` to organize your endpoints.
### id
**Type**: `str`
**Default**: `None`
Connect to an existing deployed endpoint by its ID. When `id` is specified, `name` is not required.
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
# Connect to existing endpoint
ep = Endpoint(id="abc123xyz")
# Make requests
job = await ep.run({"prompt": "hello"})
result = await ep.post("/inference", {"data": "..."})
```
### gpu
**Type**: `GpuGroup`, `GpuType`, or `list[GpuGroup | GpuType]`
**Default**: `GpuGroup.ANY` (if neither `gpu` nor `cpu` is specified)
Specifies GPU hardware for the endpoint. Accepts a single GPU type/group or a list for fallback strategies.
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
from runpod_flash import Endpoint, GpuType, GpuGroup
# Specific GPU type
@Endpoint(name="inference", gpu=GpuType.NVIDIA_A100_80GB_PCIe)
async def infer(data): ...
# Another specific GPU type
@Endpoint(name="rtx-worker", gpu=GpuType.NVIDIA_GEFORCE_RTX_4090)
async def process(data): ...
# Multiple types for fallback
@Endpoint(name="flexible", gpu=[GpuType.NVIDIA_A100_80GB_PCIe, GpuType.NVIDIA_RTX_A6000, GpuType.NVIDIA_GEFORCE_RTX_4090])
async def flexible_infer(data): ...
```
See [GPU types](/flash/configuration/gpu-types) for all available options.
### cpu
**Type**: `str` or `CpuInstanceType`
**Default**: `None`
Specifies a CPU instance type. Mutually exclusive with `gpu`.
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
from runpod_flash import Endpoint, CpuInstanceType
# String shorthand
@Endpoint(name="data-processor", cpu="cpu5c-4-8")
async def process(data): ...
# Using enum
@Endpoint(name="data-processor", cpu=CpuInstanceType.CPU5C_4_8)
async def process(data): ...
```
See [CPU types](/flash/configuration/cpu-types) for all available options.
### workers
**Type**: `int` or `tuple[int, int]`
**Default**: `(0, 1)`
Controls worker scaling. Accepts either a single integer (max workers with min=0) or a tuple of (min, max).
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
# Just max: scales from 0 to 5
@Endpoint(name="elastic", gpu=GpuGroup.ANY, workers=5)
# Min and max: always keep 2 warm, scale up to 10
@Endpoint(name="always-on", gpu=GpuGroup.ANY, workers=(2, 10))
# Default: (0, 1)
@Endpoint(name="default", gpu=GpuGroup.ANY)
```
**Recommendations**:
* `workers=N` or `workers=(0, N)`: Cost-optimized, allows scale to zero
* `workers=(1, N)`: Avoid cold starts by keeping at least one worker warm
* `workers=(N, N)`: Fixed worker count for consistent performance
### idle\_timeout
**Type**: `int`
**Default**: `60`
**Valid range**: 1-3600 seconds
Number of seconds workers will stay active (running) after completing a request, waiting for additional requests before scaling down (to minimum workers).
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
# Quick scale-down for cost savings
@Endpoint(name="batch", gpu=GpuGroup.ANY, idle_timeout=30)
# Keep workers longer for variable traffic
@Endpoint(name="api", gpu=GpuGroup.ANY, idle_timeout=120)
```
**Recommendations**:
* `30-60 seconds`: Cost-optimized, infrequent traffic
* `60-120 seconds`: Balanced, variable traffic patterns
* `120-300 seconds`: Latency-optimized, consistent traffic
### dependencies
**Type**: `list[str]`
**Default**: `None`
Python packages to install on the remote worker before executing your function. Supports standard pip syntax.
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
@Endpoint(
name="ml-worker",
gpu=GpuGroup.ANY,
dependencies=["torch>=2.0.0", "transformers==4.36.0", "pillow"]
)
async def process(data): ...
```
Packages must be imported **inside** the function body, not at the top of your file.
### system\_dependencies
**Type**: `list[str]`
**Default**: `None`
System-level packages to install via apt before your function runs.
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
@Endpoint(
name="video-processor",
gpu=GpuGroup.ANY,
dependencies=["opencv-python"],
system_dependencies=["libgl1-mesa-glx", "libglib2.0-0"]
)
async def process_video(data): ...
```
### accelerate\_downloads
**Type**: `bool`
**Default**: `True`
Enables faster downloads for dependencies, models, and large files. Disable if you encounter compatibility issues.
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
@Endpoint(
name="standard-downloads",
gpu=GpuGroup.ANY,
accelerate_downloads=False
)
async def process(data): ...
```
### volume
**Type**: `NetworkVolume` or `list[NetworkVolume]`
**Default**: `None`
Attaches network volume(s) for persistent storage. Volumes are mounted at `/runpod-volume/`. Flash uses the volume `name` to find an existing volume or create a new one. Each volume is tied to a specific datacenter.
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
from runpod_flash import Endpoint, GpuGroup, DataCenter, NetworkVolume
# Single volume in a specific datacenter
vol = NetworkVolume(name="model-cache", size=100, datacenter=DataCenter.US_GA_2)
@Endpoint(
name="model-server",
gpu=GpuGroup.ANY,
datacenter=DataCenter.US_GA_2,
volume=vol
)
async def serve(data):
# Access files at /runpod-volume/
model = load_model("/runpod-volume/models/bert")
...
```
For multi-datacenter deployments, pass a list of volumes (one per datacenter):
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
from runpod_flash import Endpoint, GpuGroup, DataCenter, NetworkVolume
volumes = [
NetworkVolume(name="models-us", size=100, datacenter=DataCenter.US_GA_2),
NetworkVolume(name="models-eu", size=100, datacenter=DataCenter.EU_RO_1),
]
@Endpoint(
name="global-server",
gpu=GpuGroup.ANY,
datacenter=[DataCenter.US_GA_2, DataCenter.EU_RO_1],
volume=volumes
)
async def serve(data):
...
```
Only one network volume is allowed per datacenter. If you specify multiple volumes in the same datacenter, deployment will fail.
**Use cases**:
* Share large models across workers
* Persist data between runs
* Share datasets across endpoints
See [Storage](/flash/configuration/storage) for setup instructions.
### datacenter
**Type**: `DataCenter`, `list[DataCenter]`, `str`, `list[str]`, or `None`
**Default**: `None` (all available datacenters)
Specifies the datacenter(s) for worker deployment. When set to `None`, the endpoint is available in all datacenters.
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
from runpod_flash import Endpoint, GpuGroup, DataCenter
# Deploy to all available datacenters (default)
@Endpoint(name="global", gpu=GpuGroup.ANY)
async def process(data): ...
# Deploy to a single datacenter
@Endpoint(
name="us-workers",
gpu=GpuGroup.ANY,
datacenter=DataCenter.US_GA_2
)
async def process(data): ...
# Deploy to multiple datacenters
@Endpoint(
name="multi-region",
gpu=GpuGroup.ANY,
datacenter=[DataCenter.US_GA_2, DataCenter.EU_RO_1]
)
async def process(data): ...
# String DC IDs also work
@Endpoint(
name="us-workers",
gpu=GpuGroup.ANY,
datacenter="US-GA-2"
)
async def process(data): ...
```
**Available datacenters**:
| Value | Location |
| --------------------- | ----------------------- |
| `DataCenter.US_CA_2` | US - California |
| `DataCenter.US_GA_2` | US - Georgia |
| `DataCenter.US_IL_1` | US - Illinois |
| `DataCenter.US_KS_2` | US - Kansas |
| `DataCenter.US_MD_1` | US - Maryland |
| `DataCenter.US_MO_1` | US - Missouri |
| `DataCenter.US_MO_2` | US - Missouri |
| `DataCenter.US_NC_1` | US - North Carolina |
| `DataCenter.US_NC_2` | US - North Carolina |
| `DataCenter.US_NE_1` | US - Nebraska |
| `DataCenter.US_WA_1` | US - Washington |
| `DataCenter.EU_CZ_1` | Europe - Czech Republic |
| `DataCenter.EU_RO_1` | Europe - Romania |
| `DataCenter.EUR_IS_1` | Europe - Iceland |
| `DataCenter.EUR_NO_1` | Europe - Norway |
CPU endpoints are restricted to `CPU_DATACENTERS`, which currently only includes `EU_RO_1`.
### env
**Type**: `dict[str, str]`
**Default**: `None`
Environment variables passed to all workers. Useful for API keys, configuration, and feature flags.
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
@Endpoint(
name="ml-worker",
gpu=GpuGroup.ANY,
env={
"HF_TOKEN": "your_huggingface_token",
"MODEL_ID": "gpt2",
"LOG_LEVEL": "INFO"
}
)
async def load_model():
import os
token = os.getenv("HF_TOKEN")
model_id = os.getenv("MODEL_ID")
...
```
Values in your project's `.env` file are only available locally for CLI commands and development. They are **not** passed to deployed endpoints. You must declare environment variables explicitly using the `env` parameter.
To pass a local environment variable to your deployed endpoint, read it from `os.environ`:
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
import os
@Endpoint(
name="ml-worker",
gpu=GpuGroup.ANY,
env={"HF_TOKEN": os.environ["HF_TOKEN"]} # Read from local env, pass to workers
)
async def load_model():
...
```
Environment variables are excluded from configuration hashing. Changing environment values won't trigger endpoint recreation, making it easy to rotate API keys.
### gpu\_count
**Type**: `int`
**Default**: `1`
Number of GPUs per worker. Use for multi-GPU workloads.
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
@Endpoint(
name="multi-gpu-training",
gpu=GpuType.NVIDIA_A100_80GB_PCIe,
gpu_count=4, # Each worker gets 4 GPUs
workers=2 # Maximum 2 workers = 8 GPUs total
)
async def train(data): ...
```
### execution\_timeout\_ms
**Type**: `int`
**Default**: `0` (no limit)
Maximum execution time for a single job in milliseconds. Jobs exceeding this timeout are terminated.
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
# 5 minute timeout
@Endpoint(
name="training",
gpu=GpuGroup.ANY,
execution_timeout_ms=300000 # 5 * 60 * 1000
)
async def train(data): ...
# 30 second timeout for quick inference
@Endpoint(
name="quick-inference",
gpu=GpuGroup.ANY,
execution_timeout_ms=30000
)
async def infer(data): ...
```
The Flash SDK's `runsync()` method uses your `execution_timeout_ms` value as the client-side HTTP timeout. If set to a positive value, the SDK waits that duration for the job to complete. If unset or set to `0`, the SDK defaults to a 60-second timeout. For long-running inference jobs, set `execution_timeout_ms` to prevent premature timeouts.
### flashboot
**Type**: `bool`
**Default**: `True`
Enables Flashboot for faster cold starts by pre-loading container images.
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
@Endpoint(
name="fast-startup",
gpu=GpuGroup.ANY,
flashboot=True # Default
)
async def process(data): ...
```
Set to `False` for debugging or compatibility reasons.
### image
**Type**: `str`
**Default**: `None`
Custom Docker image to deploy. When specified, the endpoint runs your Docker image instead of Flash's managed workers.
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
from runpod_flash import Endpoint, GpuType
vllm = Endpoint(
name="vllm-server",
image="runpod/worker-vllm:stable-cuda12.1.0",
gpu=GpuType.NVIDIA_A100_80GB_PCIe,
env={"MODEL_NAME": "meta-llama/Llama-3.2-3B-Instruct"}
)
# Make HTTP calls to the deployed image
result = await vllm.post("/v1/completions", {"prompt": "Hello"})
```
See [Custom Docker images](/flash/custom-docker-images) for complete documentation.
### scaler\_type
**Type**: `ServerlessScalerType`
**Default**: Auto-selected based on endpoint type
Scaling algorithm strategy. Defaults are automatically set:
* Queue-based: `QUEUE_DELAY` (scales based on queue depth)
* Load-balanced: `REQUEST_COUNT` (scales based on active requests)
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
from runpod_flash import Endpoint, ServerlessScalerType
@Endpoint(
name="custom-scaler",
gpu=GpuGroup.ANY,
scaler_type=ServerlessScalerType.QUEUE_DELAY
)
async def process(data): ...
```
### scaler\_value
**Type**: `int`
**Default**: `4`
Parameter value for the scaling algorithm. With `QUEUE_DELAY`, represents target jobs per worker before scaling up.
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
# Scale up when > 2 jobs per worker (more aggressive)
@Endpoint(
name="responsive",
gpu=GpuGroup.ANY,
scaler_value=2
)
async def process(data): ...
```
### template
**Type**: `PodTemplate`
**Default**: `None`
Advanced pod configuration overrides.
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
from runpod_flash import Endpoint, GpuGroup, PodTemplate
@Endpoint(
name="custom-pod",
gpu=GpuGroup.ANY,
template=PodTemplate(
containerDiskInGb=100,
env=[{"key": "PYTHONPATH", "value": "/workspace"}]
)
)
async def process(data): ...
```
## PodTemplate
`PodTemplate` provides advanced pod configuration options:
| Parameter | Type | Description | Default |
| ------------------- | ------------ | ----------------------------------------------------------------- | ------- |
| `containerDiskInGb` | `int` | Container disk size in GB | 64 |
| `env` | `list[dict]` | Environment variables as list of `{"key": "...", "value": "..."}` | `None` |
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
from runpod_flash import PodTemplate
template = PodTemplate(
containerDiskInGb=100,
env=[
{"key": "PYTHONPATH", "value": "/workspace"},
{"key": "CUDA_VISIBLE_DEVICES", "value": "0"}
]
)
```
For simple environment variables, use the `env` parameter on `Endpoint` instead of `PodTemplate.env`.
### min\_cuda\_version
**Type**: `str` or `CudaVersion`
**Default**: `"12.8"` for GPU endpoints, `None` for CPU endpoints
Specifies the minimum CUDA driver version required on the host machine. GPU endpoints default to `"12.8"` to ensure workers run on hosts with recent CUDA drivers.
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
from runpod_flash import Endpoint, GpuType, CudaVersion
# Use the default (12.8)
@Endpoint(name="ml-inference", gpu=GpuType.NVIDIA_A100_80GB_PCIe)
async def infer(data): ...
# Override with string value
@Endpoint(
name="legacy-compatible",
gpu=GpuType.NVIDIA_A100_80GB_PCIe,
min_cuda_version="12.4"
)
async def infer_legacy(data): ...
# Override with CudaVersion enum
@Endpoint(
name="cuda-12",
gpu=GpuType.NVIDIA_A100_80GB_PCIe,
min_cuda_version=CudaVersion.V12_0
)
async def infer_cuda12(data): ...
```
This parameter has no effect on CPU endpoints.
Valid CUDA versions: `CudaVersion.V11_1`, `V11_4`, `V11_7`, `V11_8`, `V12_0`, `V12_1`, `V12_2`, `V12_3`, `V12_4`, `V12_6`, `V12_8` (or equivalent strings like `"12.4"`). Invalid values raise a `ValueError`.
### python\_version
**Type**: `str`
**Default**: Local Python version
Sets the Python version for the worker image. Supported values: `"3.10"`, `"3.11"`, `"3.12"`, and `"3.13"`.
When you don't specify a Python version, Flash matches your local interpreter (the Python version you run Flash from). The resolution order is:
1. `--python-version` CLI flag (highest priority)
2. `python_version` declared on resource configs
3. Your local Python version (`sys.version_info`)
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
from runpod_flash import Endpoint, GpuGroup
# Explicitly set Python 3.11 for this endpoint
@Endpoint(
name="legacy-model",
gpu=GpuGroup.ANY,
python_version="3.11"
)
async def process(data): ...
# Uses your local Python version (e.g., 3.12 if that's what you're running)
@Endpoint(name="modern-model", gpu=GpuGroup.ANY)
async def infer(data): ...
```
All resources in a Flash app must use the same Python version because Flash ships a single tarball for the entire app. If resources declare conflicting versions, the build fails.
**Breaking change:** Flash now matches your local Python version by default instead of always defaulting to Python 3.12. If your local Python differs from 3.12, your first deploy after upgrading Flash will trigger a rolling release. For consistent behavior across team members, declare `python_version` explicitly or use the `--python-version` CLI flag.
Python 3.10, 3.11, and 3.13 workers incur approximately 7 GB of additional cold-start overhead on GPU endpoints because the alternative Python interpreter must be installed alongside the base image's PyTorch environment.
Python 3.10 reaches end-of-life on 2026-10-31. Consider migrating to Python 3.11 or later.
If your local Python version is not supported (for example, 3.9 or 3.14), the build fails with an actionable error message listing the supported versions.
The `--python-version` CLI flag on `flash build` and `flash deploy` overrides both per-resource declarations and local interpreter detection.
## EndpointJob
When using `Endpoint(id=...)` or `Endpoint(image=...)`, the `.run()` method returns an `EndpointJob` object for async operations:
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
ep = Endpoint(id="abc123")
# Submit a job
job = await ep.run({"prompt": "hello"})
# Check status
status = await job.status() # "IN_PROGRESS", "COMPLETED", etc.
# Wait for completion
await job.wait(timeout=60) # Optional timeout in seconds
# Access results
print(job.id) # Job ID
print(job.output) # Result payload
print(job.error) # Error message if failed
print(job.done) # True if completed/failed
# Cancel a job
await job.cancel()
```
## Configuration change behavior
When you change configuration and redeploy, Flash automatically updates your endpoint.
### Changes that recreate workers
These changes restart all workers:
* GPU configuration (`gpu`, `gpu_count`)
* CPU instance type (`cpu`)
* Docker image (`image`)
* Storage (`volume`)
* Datacenter (`datacenter`)
* Flashboot setting (`flashboot`)
* CUDA version requirement (`min_cuda_version`)
* Python version (`python_version`)
Workers are temporarily unavailable during recreation (typically 30-90 seconds).
### Changes that update settings only
These changes apply immediately with no downtime:
* Worker scaling (`workers`)
* Timeouts (`idle_timeout`, `execution_timeout_ms`)
* Scaler settings (`scaler_type`, `scaler_value`)
* Environment variables (`env`)
* Endpoint name (`name`)
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
# First deployment
@Endpoint(
name="inference-api",
gpu=GpuType.NVIDIA_A100_80GB_PCIe,
workers=5,
env={"MODEL": "v1"}
)
async def infer(data): ...
# Update scaling - no worker recreation
@Endpoint(
name="inference-api",
gpu=GpuType.NVIDIA_A100_80GB_PCIe, # Same GPU
workers=10, # Changed - updates settings only
env={"MODEL": "v2"} # Changed - updates settings only
)
async def infer(data): ...
# Change GPU type - workers recreated
@Endpoint(
name="inference-api",
gpu=GpuType.NVIDIA_GEFORCE_RTX_4090, # Changed - triggers recreation
workers=10,
env={"MODEL": "v2"}
)
async def infer(data): ...
```
# Storage
Source: https://docs.runpod.io/flash/configuration/storage
Understand container disk and network volume storage for Flash workloads. Review configuration and usage details for Runpod Flash.
Flash workers have access to two types of storage: for temporary data and for persistent, sharable data.
## Container disk
A container disk provides temporary storage that exists only while a worker is running. Each worker gets its own isolated container disk, with a default size of 64GB for GPU endpoints.
You can read and write temporary files to the container disk using standard filesystem operations from within `@Endpoint` functions.
Any file that is *not* written to a network volume (at `/runpod-volume/`) is written to the container disk, and will be erased when the worker stops.
### Configuring container disk size (GPU-only)
Configure container disk size for GPU endpoints using the `template` parameter (default: 64GB).
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
from runpod_flash import Endpoint, GpuType, PodTemplate
@Endpoint(
name="large-temp-storage",
gpu=GpuType.NVIDIA_A100_80GB_PCIe,
template=PodTemplate(containerDiskInGb=100)
)
async def process(data: dict) -> dict:
# 100GB container disk available
...
```
### CPU auto-sizing
CPU endpoints automatically adjust container disk size based on instance limits:
* `CPU3G` and `CPU3C` instances: vCPU count × 10GB (e.g., 2 vCPU = 20GB)
* `CPU5C` instances: vCPU count × 15GB (e.g., 4 vCPU = 60GB)
If you specify a custom size that exceeds the instance limit, deployment will fail with a validation error.
## Network volumes
Network volumes provide persistent storage that survives worker restarts. Each volume is tied to a specific datacenter. Use volumes to share data between endpoint functions or to persist data between runs.
### Attaching network volumes
Attach a network volume using the `volume` parameter. Flash uses the volume `name` to find an existing volume or create a new one. Specify the `datacenter` parameter to control where the volume is created:
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
from runpod_flash import Endpoint, GpuType, DataCenter, NetworkVolume
vol = NetworkVolume(name="model-cache", size=100, datacenter=DataCenter.US_GA_2)
@Endpoint(
name="persistent-storage",
gpu=GpuType.NVIDIA_A100_80GB_PCIe,
datacenter=DataCenter.US_GA_2,
volume=vol
)
async def process(data: dict) -> dict:
# Access files at /runpod-volume/
...
```
The `size` parameter specifies the volume size in GB. Valid values range from 10 to 4096 GB (4 TB). If not specified, `size` defaults to 100 GB.
You can also reference an existing volume by ID:
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
vol = NetworkVolume(id="vol_abc123")
```
### Multi-datacenter volumes
For endpoints deployed across multiple datacenters, pass a list of volumes (one per datacenter):
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
from runpod_flash import Endpoint, GpuType, DataCenter, NetworkVolume
volumes = [
NetworkVolume(name="models-us", size=100, datacenter=DataCenter.US_GA_2),
NetworkVolume(name="models-eu", size=100, datacenter=DataCenter.EU_RO_1),
]
@Endpoint(
name="global-inference",
gpu=GpuType.NVIDIA_A100_80GB_PCIe,
datacenter=[DataCenter.US_GA_2, DataCenter.EU_RO_1],
volume=volumes
)
async def process(data: dict) -> dict:
# Workers in each region access their local volume at /runpod-volume/
...
```
Only one network volume is allowed per datacenter. If you specify multiple volumes in the same datacenter, deployment will fail.
### Accessing network volume files
Network volumes mount at `/runpod-volume/` and can be accessed like a regular filesystem:
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
from runpod_flash import Endpoint, GpuType, NetworkVolume
vol = NetworkVolume(name="model-storage")
@Endpoint(
name="model-server",
gpu=GpuType.NVIDIA_A100_80GB_PCIe,
volume=vol,
dependencies=["torch", "transformers"]
)
async def run_inference(prompt: str) -> dict:
from transformers import AutoModelForCausalLM, AutoTokenizer
# Load model from network volume
# Persists across worker restarts and shared between workers
model_path = "/runpod-volume/models/llama-7b"
model = AutoModelForCausalLM.from_pretrained(model_path)
tokenizer = AutoTokenizer.from_pretrained(model_path)
# Run inference
inputs = tokenizer(prompt, return_tensors="pt").to("cuda")
outputs = model.generate(**inputs, max_length=100)
text = tokenizer.decode(outputs[0])
return {"generated_text": text}
```
### Load-balanced endpoints with storage
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
from runpod_flash import Endpoint, GpuType, NetworkVolume
vol = NetworkVolume(name="model-storage")
api = Endpoint(
name="inference-api",
gpu=GpuType.NVIDIA_GEFORCE_RTX_4090,
volume=vol,
workers=(1, 5)
)
@api.post("/generate")
async def generate(prompt: str) -> dict:
from transformers import AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained("/runpod-volume/models/gpt2")
# Generate text
return {"text": "generated"}
@api.get("/models")
async def list_models() -> dict:
import os
models = os.listdir("/runpod-volume/models")
return {"models": models}
```
### Creating and managing network volumes
Network volumes must be created before attaching them to an Endpoint. See [Network volumes](/storage/network-volumes) for detailed instructions.
# Create endpoints
Source: https://docs.runpod.io/flash/create-endpoints
Learn how to create and configure hardware and scaling behavior with the Flash Endpoint class. Review configuration and usage details for Runpod Flash.
In Flash, endpoints are the bridge between your local Python functions and Runpod's cloud infrastructure. When you decorate a function with `@Endpoint`, you're marking it to run remotely on Runpod instead of your local machine:
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
from runpod_flash import Endpoint, GpuType
@Endpoint(
name="my-inference",
gpu=GpuType.NVIDIA_GEFORCE_RTX_4090,
dependencies=["torch"]
)
def run_model(data):
import torch
# This code runs on a Runpod GPU, not locally
return {"result": "processed"}
```
When you call `run_model(data)`, Flash provisions a GPU on Runpod (or reuses an existing one), sends your function code and input to the worker, executes it, and returns the result to your local environment.
Each unique endpoint `name` creates one Serverless endpoint on Runpod with its own URL, scaling configuration, and hardware allocation. The endpoint manages workers that scale up and down based on demand.
## Endpoint types
The `Endpoint` class supports four distinct patterns.
### Queue-based endpoints
Use `@Endpoint(...)` as a decorator for batch processing and async workloads. Each function gets its own endpoint with dedicated workers.
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
from runpod_flash import Endpoint, GpuType
@Endpoint(
name="image-processor",
gpu=GpuType.NVIDIA_GEFORCE_RTX_4090,
workers=(0, 5),
dependencies=["torch", "pillow"]
)
async def process_image(image_data: dict) -> dict:
import torch
from PIL import Image
# Process image on GPU
return {"processed": True}
```
Queue-based endpoints are ideal for:
* Batch processing jobs
* Long-running computations
* Workloads that don't need immediate responses
### Load-balanced endpoints
Use `Endpoint(...)` as an instance with route decorators for HTTP APIs. Multiple routes share the same workers.
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
from runpod_flash import Endpoint, GpuType
api = Endpoint(
name="inference-api",
gpu=GpuType.NVIDIA_GEFORCE_RTX_4090,
workers=(1, 5),
dependencies=["torch"]
)
@api.post("/predict")
async def predict(data: dict) -> dict:
import torch # Import inside the function body
# Run inference
return {"prediction": "result"}
@api.get("/health")
async def health():
return {"status": "ok"}
```
Load-balanced endpoints are ideal for:
* REST APIs with multiple routes
* Low-latency request/response patterns
* Services requiring custom HTTP methods
### Custom Docker images
Deploy pre-built Docker images (like vLLM or your own workers) and interact with them as a client:
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
from runpod_flash import Endpoint, GpuType
vllm = Endpoint(
name="vllm-server",
image="vllm/vllm-openai:latest",
gpu=GpuType.NVIDIA_A100_80GB_PCIe
)
# Make HTTP calls to the deployed image
result = await vllm.post("/v1/completions", {"prompt": "Hello"})
models = await vllm.get("/v1/models")
```
See [Custom Docker images](/flash/custom-docker-images) for complete documentation, including available images and configuration options.
### Existing endpoints
Connect to an already-deployed Runpod endpoint by ID:
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
from runpod_flash import Endpoint
ep = Endpoint(id="abc123")
# Queue-based calls
job = await ep.run({"prompt": "hello"})
await job.wait()
print(job.output)
# Or load-balanced calls
result = await ep.post("/v1/completions", {"prompt": "hello"})
```
## GPU vs CPU
Specify `gpu=` for GPU endpoints or `cpu=` for CPU endpoints. They are mutually exclusive.
### GPU endpoints
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
from runpod_flash import Endpoint, GpuType, GpuGroup
# Use a specific GPU type
@Endpoint(name="ml-inference", gpu=GpuType.NVIDIA_A100_80GB_PCIe)
async def infer(data: dict) -> dict: ...
# Use another specific GPU type
@Endpoint(name="rtx-worker", gpu=GpuType.NVIDIA_GEFORCE_RTX_4090)
async def render(data: dict) -> dict: ...
# Use multiple GPU types for better availability
@Endpoint(name="flexible", gpu=[GpuType.NVIDIA_GEFORCE_RTX_4090, GpuType.NVIDIA_RTX_A5000])
async def process(data: dict) -> dict: ...
```
If neither `gpu=` nor `cpu=` is specified, GPU defaults to `GpuGroup.ANY`.
### CPU endpoints
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
from runpod_flash import Endpoint, CpuInstanceType
# Use string shorthand
@Endpoint(name="data-processor", cpu="cpu5c-4-8")
async def process(data: dict) -> dict: ...
# Or use the enum
@Endpoint(name="data-processor", cpu=CpuInstanceType.CPU5C_4_8)
async def process(data: dict) -> dict: ...
```
See [GPU types](/flash/configuration/gpu-types) and [CPU types](/flash/configuration/cpu-types) for available options.
## Worker scaling
Control how many workers run for your endpoint with the `workers` parameter:
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
# Just a max: scales from 0 to 5
@Endpoint(name="elastic", gpu=GpuGroup.ANY, workers=5)
# Min and max tuple: always keep 2 warm, scale up to 10
@Endpoint(name="always-on", gpu=GpuGroup.ANY, workers=(2, 10))
# Default is (0, 1) if not specified
@Endpoint(name="default", gpu=GpuGroup.ANY)
```
Setting `workers=(1, N)` keeps at least one worker warm, avoiding cold starts.
## Dependency management
Specify Python packages in the `dependencies` parameter. Flash installs these on the remote worker before executing your function.
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
@Endpoint(
name="text-gen",
gpu=GpuType.NVIDIA_A100_80GB_PCIe,
dependencies=["transformers==4.36.0", "torch", "pillow"]
)
def generate_text(prompt):
from transformers import pipeline
import torch
# Your code here
```
### Version pinning
Use standard pip syntax for version constraints:
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
dependencies=["transformers==4.36.0", "torch>=2.0.0", "numpy<2.0"]
```
### Import packages inside the function body
You must import pip/installed packages **inside the decorated function body**, not at the top of your file. This ensures imports happen on the remote worker. This rule applies to installed packages only; local project modules can be imported at the top of the file because Flash ships their source (see [Import local modules](#import-local-modules)).
**Correct:** imports inside the function.
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
@Endpoint(name="compute", gpu=GpuGroup.ANY, dependencies=["numpy"])
def compute(data):
import numpy as np # Import here
return np.sum(data)
```
**Incorrect:** imports at top of file won't work.
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
import numpy as np # This import happens locally, not on the worker
@Endpoint(name="compute", gpu=GpuGroup.ANY, dependencies=["numpy"])
def compute(data):
return np.sum(data) # numpy not available on the remote worker
```
### System dependencies
Use `system_dependencies` to install system-level packages (via apt):
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
@Endpoint(
name="video-processor",
gpu=GpuGroup.ANY,
dependencies=["opencv-python"],
system_dependencies=["libgl1-mesa-glx", "libglib2.0-0"]
)
async def process_video(video_data):
import cv2
# OpenCV processing
return {"processed": True}
```
## Import local modules
Your endpoint can import local (non-pip) Python modules that live alongside it in your project, such as a sibling `utils.py` file or a `helpers/` package. Flash detects these imports, follows them transitively, and ships the module source to the worker for you, so an import like `import utils` or `from helpers import load` works remotely with no extra configuration.
Flash resolves local imports whether they appear at the top of the file or inside the function body, and it supports absolute imports (`import utils`), relative imports (`from . import helpers`), and dynamic imports with a literal name (`importlib.import_module("plugin")`). It also pulls in the `__init__.py` files for any packages you import. Flash can't resolve dynamic imports whose module name is computed at runtime, so it emits a warning, and you're responsible for making those modules available on the worker.
Flash bundles only local project files. Standard library modules are already present in the worker image, and pip packages must still be declared through the `dependencies` parameter. This applies transitively: if a bundled local module imports a pip package at its top level, that package must still be declared in the `dependencies` of any endpoint that uses the module.
On `flash build` and `flash deploy`, local modules are bundled when they pass the ignore filter, and importing a local module that an ignore rule excludes (or one Flash can't resolve) fails the build. See [Local modules and the ignore filter](/flash/cli/build#local-modules-and-the-ignore-filter) for details.
### Live execution size limit
When you run an endpoint live (calling an `@Endpoint` function directly, or during `flash dev`), Flash ships the resolved module source inline with the request. The combined source is capped at 8 MiB. If your local dependencies exceed this limit, deploy the app with `flash deploy` instead, which bundles local modules into the build artifact rather than the request payload. See [Local module payload too large](/flash/troubleshooting#local-module-payload-too-large) for the corresponding error.
### Local modules in a parent directory
On the live execution path, Flash treats a module imported by absolute name from a parent directory as external and doesn't ship it, which causes a `ModuleNotFoundError` on the worker. For example, this happens with `import shared` when `shared.py` sits above your endpoint file. To avoid it, place your endpoint at or above its local dependencies, or use `flash deploy`, which resolves imports against the whole project directory.
## Parallel execution
Endpoint functions are async. Use Python's `asyncio` to run multiple operations concurrently:
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
import asyncio
async def main():
# Run three functions in parallel
results = await asyncio.gather(
process_item(item1),
process_item(item2),
process_item(item3)
)
return results
```
This is useful for:
* Batch processing multiple inputs
* Running different models on the same data
* Parallelizing independent pipeline stages
## Environment variables
Pass environment variables using the `env` parameter:
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
@Endpoint(
name="api-worker",
gpu=GpuGroup.ANY,
env={
"HF_TOKEN": "your_huggingface_token",
"MODEL_ID": "gpt2"
}
)
async def load_model():
import os
from transformers import AutoModel
hf_token = os.getenv("HF_TOKEN")
model_id = os.getenv("MODEL_ID")
model = AutoModel.from_pretrained(model_id, token=hf_token)
return {"model_loaded": model_id}
```
Environment variables are excluded from configuration hashing. Changing environment values won't trigger endpoint recreation, making it easy to rotate API keys.
## Persistent storage
Attach a network volume for persistent storage across workers. Each volume is tied to a specific datacenter. Flash uses the volume `name` to find an existing volume or create a new one:
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
from runpod_flash import Endpoint, GpuGroup, DataCenter, NetworkVolume
vol = NetworkVolume(name="model-cache", size=100, datacenter=DataCenter.US_GA_2)
@Endpoint(
name="model-server",
gpu=GpuGroup.ANY,
datacenter=DataCenter.US_GA_2,
volume=vol
)
async def serve(data: dict) -> dict:
# Access files at /runpod-volume/
...
```
For multi-datacenter deployments, pass a list of volumes (one per datacenter):
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
from runpod_flash import Endpoint, GpuGroup, DataCenter, NetworkVolume
volumes = [
NetworkVolume(name="models-us", size=100, datacenter=DataCenter.US_GA_2),
NetworkVolume(name="models-eu", size=100, datacenter=DataCenter.EU_RO_1),
]
@Endpoint(
name="global-server",
gpu=GpuGroup.ANY,
datacenter=[DataCenter.US_GA_2, DataCenter.EU_RO_1],
volume=volumes
)
async def serve(data: dict) -> dict:
...
```
See [Flash storage](/flash/configuration/storage) for setup instructions.
## Endpoint parameters
For a complete list of parameters available for the `Endpoint` class, see [Endpoint parameters](/flash/configuration/parameters).
## Working with jobs (client mode)
When using `Endpoint(id=...)` or `Endpoint(image=...)`, you get an `EndpointJob` object for async operations:
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
ep = Endpoint(id="abc123")
# Submit a job
job = await ep.run({"prompt": "hello"})
# Check status
status = await job.status() # "IN_PROGRESS", "COMPLETED", etc.
# Wait for completion
await job.wait(timeout=60) # Optional timeout in seconds
# Access results
print(job.id) # Job ID
print(job.output) # Result payload
print(job.error) # Error message if failed
print(job.done) # True if completed/failed
# Cancel a job
await job.cancel()
```
## Next steps
Deploy pre-built Docker images with Flash.
Create production APIs with Flash apps.
Deploy Flash applications for production.
Remove development endpoints when done testing.
# Use custom containers with Flash
Source: https://docs.runpod.io/flash/custom-docker-images
Deploy pre-built Docker images with Flash using Endpoint. Review setup, configuration, deployment, and usage guidance for Runpod Flash.
The `@Endpoint` decorator handles most use cases, allowing you to execute arbitrary Python code remotely without managing Docker images.
However, for specialized environments that require custom Docker images, you can use `Endpoint(image=...)` to deploy your own Docker images.
## When to use custom Docker images
Use custom Docker images when you need:
* **Pre-built inference servers**: vLLM, TensorRT-LLM, or other specialized serving frameworks.
* **System-level dependencies**: Custom CUDA versions, cuDNN, or system libraries not installable via `pip`.
* **Baked-in models**: Large models pre-downloaded in the image to avoid runtime downloads.
* **Existing Serverless workers**: You already have a working Runpod Serverless Docker image.
For most use cases, use `@Endpoint` with the `dependencies` parameter. It's simpler, faster, and lets you execute arbitrary Python code remotely.
## Available Docker images
### Official Runpod workers
Runpod provides pre-built worker images for common frameworks:
| Framework | Image name | Documentation |
| ------------- | ---------------------------- | ------------------------------------------------------------ |
| vLLM | `runpod/worker-vllm` | [vLLM docs](/serverless/vllm/overview) |
| Automatic1111 | `runpod/worker-a1111:stable` | [Docker Hub](https://hub.docker.com/r/runpod/a1111) |
| ComfyUI | `runpod/worker-comfy` | [Docker Hub](https://hub.docker.com/r/runpod/worker-comfyui) |
### Custom images
To create a custom Docker image:
1. [Build a handler function](/serverless/workers/handler-functions) to process requests.
2. [Create a Dockerfile](/serverless/workers/create-dockerfile) to build the image.
3. [Push the image to a registry](/serverless/workers/deploy).
4. Reference the image with `Endpoint(image=...)`.
## Deploy a custom image
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
from runpod_flash import Endpoint, GpuType
vllm = Endpoint(
name="my-vllm-server",
image="runpod/worker-vllm:stable-cuda12.1.0",
gpu=GpuType.NVIDIA_GEFORCE_RTX_4090,
workers=3,
env={
"MODEL_NAME": "microsoft/Phi-3.5-mini-instruct",
"MAX_MODEL_LEN": "4096"
}
)
```
Use HTTP methods to call your deployed image:
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
import asyncio
async def main():
# POST request
result = await vllm.post("/v1/completions", {
"prompt": "Explain quantum computing:",
"max_tokens": 100
})
print(result)
# GET request
models = await vllm.get("/v1/models")
print(models)
asyncio.run(main())
```
Or use queue-based calls:
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
import asyncio
async def main():
# Submit job to queue
job = await vllm.run({
"input": {
"prompt": "Explain quantum computing:",
"max_tokens": 100
}
})
# Wait for completion
await job.wait()
print(job.output)
asyncio.run(main())
```
## Complete example: vLLM inference
This example deploys vLLM and makes inference requests:
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
import asyncio
from runpod_flash import Endpoint, GpuType
# Configure vLLM endpoint
vllm = Endpoint(
name="vllm-phi",
image="runpod/worker-vllm:stable-cuda12.1.0",
gpu=GpuType.NVIDIA_GEFORCE_RTX_4090,
workers=3,
env={
"MODEL_NAME": "microsoft/Phi-3.5-mini-instruct",
"MAX_MODEL_LEN": "4096",
"GPU_MEMORY_UTILIZATION": "0.9",
"MAX_CONCURRENCY": "30",
}
)
async def main():
# Generate text using queue-based call
job = await vllm.run({
"input": {
"prompt": "Explain quantum computing in simple terms:",
"max_tokens": 100,
"temperature": 0.7
}
})
await job.wait()
# Extract the generated text
text = job.output[0]['choices'][0]['tokens'][0]
print(f"Generated text: {text}")
if __name__ == "__main__":
asyncio.run(main())
```
## Configuration options
All standard `Endpoint` parameters work with custom images:
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
from runpod_flash import Endpoint, GpuType, DataCenter, NetworkVolume, PodTemplate
vol = NetworkVolume(name="model-storage", size=100, datacenter=DataCenter.US_GA_2)
vllm = Endpoint(
name="custom-vllm",
image="your-registry/image:tag",
gpu=GpuType.NVIDIA_A100_80GB_PCIe,
workers=(0, 5),
idle_timeout=600, # 10 minutes
env={
"MODEL_PATH": "/models/llama",
"MAX_BATCH_SIZE": "32"
},
datacenter=DataCenter.US_GA_2,
volume=vol,
execution_timeout_ms=300000, # 5 minutes
template=PodTemplate(containerDiskInGb=100)
)
```
### CPU endpoints
For CPU workloads, use the `cpu` parameter:
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
from runpod_flash import Endpoint
cpu_worker = Endpoint(
name="cpu-worker",
image="your-registry/cpu-worker:latest",
cpu="cpu5c-4-8" # 4 vCPU, 8GB RAM
)
```
## Request/response format
### Queue-based requests
Use `.run()` with a dictionary payload in the format `{"input": {...}}`:
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
job = await endpoint.run({
"input": {
"param1": "value1",
"param2": "value2"
}
})
await job.wait()
print(job.output) # Worker response
print(job.error) # Error message if failed
```
### HTTP requests
Use `.get()`, `.post()`, `.put()`, `.delete()` for direct HTTP calls:
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
# POST request
result = await endpoint.post("/v1/completions", {"prompt": "Hello"})
# GET request
models = await endpoint.get("/v1/models")
# With custom headers
result = await endpoint.post(
"/v1/completions",
{"prompt": "Hello"},
headers={"X-Custom-Header": "value"}
)
```
## EndpointJob reference
The `.run()` method returns an `EndpointJob` for async operations:
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
job = await endpoint.run({"input": {...}})
# Properties
job.id # Job ID
job.output # Result payload (after completion)
job.error # Error message if failed
job.done # True if completed/failed
# Methods
await job.status() # Get current status
await job.wait(timeout=60) # Wait for completion
await job.cancel() # Cancel the job
```
## Limitations
* **Input format**: Queue-based calls require `{"input": {...}}` format.
* **Code execution**: Cannot execute arbitrary Python code remotely. Your Docker image must include all logic.
* **@Endpoint decorator**: The decorator pattern doesn't work with `image=`. Use the instance pattern instead.
* **Handler required**: Your Docker image must implement a Runpod Serverless [handler function](/serverless/workers/handler-functions).
## Troubleshooting
### Endpoint fails to initialize
**Problem**: Workers fail to start or crash immediately.
**Solutions**:
* Verify your Docker image is compatible with [Runpod Serverless](/serverless/overview).
* Check environment variables are correct.
* Ensure the image includes a valid handler function.
* Check worker logs in the [Runpod console](https://console.runpod.io/serverless).
### Out of memory errors
**Problem**: Workers crash with CUDA OOM or RAM errors.
**Solutions**:
* Use a larger GPU: `gpu=GpuType.NVIDIA_A100_80GB_PCIe`
* Reduce `GPU_MEMORY_UTILIZATION` for vLLM.
* Lower `MAX_MODEL_LEN` or batch size.
* Reduce `workers` to limit parallel execution.
### Authentication errors
**Problem**: Cannot download gated models or private images.
**Solutions**:
* Add `HF_TOKEN` to `env` for Hugging Face gated models.
* Configure Docker registry authentication in [Runpod console](https://console.runpod.io/user/settings) for private images.
## Next steps
* [View all Endpoint parameters](/flash/configuration/parameters)
* [Learn about vLLM deployment](/serverless/vllm/overview)
* [Build custom Serverless workers](/serverless/workers/overview)
* [Create Flash apps](/flash/apps/build-app)
# Execution model
Source: https://docs.runpod.io/flash/execution-model
Understand how Flash executes your code on Runpod's infrastructure. Review setup, configuration, deployment, and usage guidance for Runpod Flash.
Flash runs your Python functions on remote GPU/CPU workers while you maintain local control flow. This page explains what happens when you call an `@Endpoint` function.
## What runs where
The `@Endpoint` decorator marks functions for remote execution. Everything else runs locally.
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
import asyncio
from runpod_flash import Endpoint, GpuType
@Endpoint(name="demo", gpu=GpuType.NVIDIA_GEFORCE_RTX_4090)
def process_on_gpu(data):
# This runs on Runpod worker
import torch
return {"result": "processed"}
async def main():
# This runs on your machine
result = await process_on_gpu({"input": "data"})
print(result) # This runs on your machine
if __name__ == "__main__":
asyncio.run(main()) # This runs on your machine
```
| Code | Location |
| ----------------------- | ----------------------------- |
| `@Endpoint` decorator | Your machine (marks function) |
| Inside `process_on_gpu` | Runpod worker |
| Everything else | Your machine |
### Flash apps
When you build a [Flash app](/flash/apps/overview):
**Development (`flash dev`)**:
* FastAPI server runs **locally**.
* `@Endpoint` functions run on **Runpod workers**.
**Production (`flash deploy`)**:
* Each endpoint configuration becomes a **separate Serverless endpoint**.
* All endpoints run on **Runpod**.
## Execution flow
Here's what happens when you call an `@Endpoint` function:
```mermaid theme={"theme":{"light":"github-light","dark":"github-dark"}}
%%{init: {'theme':'base', 'themeVariables': { 'primaryColor':'#9289FE','primaryTextColor':'#fff','primaryBorderColor':'#9289FE','lineColor':'#5F4CFE','secondaryColor':'#AE6DFF','tertiaryColor':'#FCB1FF','edgeLabelBackground':'#5F4CFE', 'fontSize':'14px','fontFamily':'font-inter'}}}%%
sequenceDiagram
participant Local as Your Machine
participant Flash as Flash SDK
participant Runpod as Runpod API
participant Worker as Remote Worker
Local->>Flash: Call remote function
Flash->>Flash: Look up endpoint by name
Flash->>Runpod: Check for existing endpoint
alt Endpoint exists
Runpod-->>Flash: Return endpoint ID
else New endpoint needed
Flash->>Runpod: Create endpoint
Runpod-->>Flash: Return endpoint ID
end
Flash->>Flash: Serialize function + args
Flash->>Runpod: Submit job
Runpod->>Worker: Route to worker
Worker->>Worker: Execute function
Worker->>Runpod: Return result
Runpod-->>Flash: Return result
Flash-->>Local: Return Python object
```
## Endpoint naming
Flash identifies endpoints by their `name` parameter:
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
@Endpoint(
name="inference", # This identifies the endpoint
gpu=GpuType.NVIDIA_A100_80GB_PCIe,
workers=3
)
def run_inference(data): ...
```
* **Same name, same config**: Reuses the existing endpoint.
* **Same name, different config**: Updates the endpoint automatically.
* **New name**: Creates a new endpoint.
This means you can change parameters like `workers` without creating a new endpoint—Flash detects the change and updates it.
## Worker lifecycle
Workers scale up and down based on demand and your configuration.
### Worker states
| State | Description | Billing |
| ---------------- | ----------------------------------------------------------------------------- | ---------------------- |
| **Initializing** | Downloading image, loading code | Yes |
| **Idle** | Scaled down, waiting for requests | No |
| **Running** | Processing requests | Yes |
| **Throttled** | Temporarily unable to run due to host resource constraints | No |
| **Outdated** | Marked for replacement after update | Yes (while processing) |
| **Unhealthy** | Crashed; auto-retries for up to 7 days | No |
### Scaling behavior
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
@Endpoint(
name="demo",
gpu=GpuGroup.ANY,
workers=(0, 5), # (min, max) - Scale to zero when idle, up to 5 workers
idle_timeout=60 # Seconds before running workers scale down
)
def process(data): ...
```
**Example**:
1. First job arrives → Scale to 1 worker (cold start).
2. More jobs arrive while worker busy → Scale up to max workers.
3. Jobs complete → Workers stay running for `idle_timeout` seconds before scaling down to idle.
4. No new jobs → Scale down to min workers.
## Cold starts and warm starts
Understanding cold and warm starts helps you predict latency and set expectations.
### Cold start
A cold start occurs when no workers are available to handle your job, because:
* You're calling an endpoint for the first time.
* All workers have been scaled down after not processing requests for `idle_timeout` seconds.
* All running workers are busy processing requests.
**What happens during a cold start**:
1. Runpod provisions a new worker with your configured GPU/CPU.
2. The worker image starts (dependencies are pre-installed during build).
3. Your function executes.
**Typical timing**: 10-60 seconds total, depending on GPU availability and image size.
When using `flash build` or `flash deploy`, dependencies are pre-installed in the worker image, eliminating pip installation at request time. When running standalone scripts with `@Endpoint` functions outside of a Flash app, dependencies may be installed on the worker at request time.
### Warm start
A warm start occurs when a worker is already running and idle:
* Worker completed a previous job and is waiting for more work.
* Worker is within its `idle_timeout` period.
**What happens during a warm start**:
1. Job is routed immediately to the idle worker.
2. Your function executes.
**Typical timing**: \~1 second + your function's execution time.
### The relationship between configuration and starts
Your `workers` and `idle_timeout` settings directly affect cold start frequency:
* `workers=(0, n)`: Workers scale to zero when not processing. Every request after the `idle_timeout` period triggers a cold start.
* `workers=(1, n)`: At least one worker stays ready. First concurrent request is warm, additional requests may cold start.
* Higher `idle_timeout`: Workers stay running longer before scaling down, reducing cold starts for sporadic traffic.
See [configuration best practices](/flash/configuration/best-practices) for specific recommendations based on your workload.
# Overview
Source: https://docs.runpod.io/flash/overview
Build autoscaling AI/ML apps using local code with Runpod Flash. Review setup, configuration, deployment, and usage guidance for Runpod Flash.
Flash is a Python SDK for developing cloud-native AI apps where you define everything—hardware, remote functions, and dependencies—using local code.
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
import asyncio
from runpod_flash import Endpoint, GpuType
# Mark the function below for remote execution
@Endpoint(name="hello-gpu", gpu=GpuType.NVIDIA_GEFORCE_RTX_4090, dependencies=["torch"])
async def hello(): # This function runs on Runpod
import torch
gpu_name = torch.cuda.get_device_name(0)
print(f"Hello from your GPU! ({gpu_name})")
return {"gpu": gpu_name}
asyncio.run(hello())
print("Done!") # This runs locally
```
Write `@Endpoint` decorated Python functions on your local machine. Run them, and Flash automatically handles GPU/CPU provisioning and worker scaling on [Runpod Serverless](/serverless/overview).
## Get started
Write a Flash script for instant access to Runpod GPUs.
Learn how to create endpoints of various types.
Browse example Flash scripts and apps on GitHub.
## Setup
Flash requires a Runpod account with a verified email address.
### Install Flash
Flash requires [Python 3.10, 3.11, 3.12, or 3.13](https://www.python.org/downloads/) and runs natively on macOS and Linux. Windows users can run Flash through [WSL2](/flash/windows-wsl2).
Install Flash using `pip` or `uv`:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
# Install with pip
pip install runpod-flash
# Or uv
uv tool install runpod-flash
```
### Authentication
Before you can use Flash, you need to authenticate with your Runpod account:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
flash login
# If using uv:
uv run flash login
```
This saves your API key securely and allows you to use the Flash CLI and run `@Endpoint` functions.
### Coding agent integration (optional)
Install the Flash skill package for AI coding agents like Claude Code, Cline, and Cursor:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
npx skills add runpod/runpod-plugins-official
```
You can review the `SKILL.md` file in the [runpod/runpod-plugins-official repository](https://github.com/runpod/runpod-plugins-official/blob/main/plugins/runpod/skills/flash/SKILL.md).
## Flash apps
When you're ready to move beyond scripts and build a production-ready API, you can create a [Flash app](/flash/apps/overview) (a collection of interconnected endpoints with diverse hardware configurations) and deploy it to Runpod.
[Follow this tutorial to build your first Flash app](/flash/apps/build-app).
## Flash CLI
The Flash CLI provides a set of commands for managing your Flash apps and endpoints.
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
flash --help
```
[Learn more about the Flash CLI](/flash/cli/overview).
## Limitations
* Flash runs natively on macOS and Linux. Windows users can run Flash through [WSL2](/flash/windows-wsl2).
* CPU endpoints are restricted to the `EU-RO-1` datacenter. GPU endpoints can deploy to [multiple datacenters](/flash/configuration/parameters#datacenter).
* Flash can rapidly scale workers across multiple endpoints, and you may hit your maximum worker threshold quickly. Contact [Runpod support](https://www.runpod.io/contact) to increase your account's capacity if needed.
## Tutorials
Build a GPU-accelerated image generation service.
Deploy a text generation model on Runpod.
Create HTTP endpoints with load balancing.
# Pricing
Source: https://docs.runpod.io/flash/pricing
Understand Flash pricing and optimize your costs. Review setup, configuration, deployment, and usage guidance for Runpod Flash.
Flash follows the same pricing model as [Runpod Serverless](/serverless/pricing). You pay per second of compute time, with no charges when your code isn't running. Pricing depends on the GPU or CPU type you configure for your endpoints.
## How pricing works
You're billed from when a worker starts until it completes your request, plus any idle time before scaling down. If a worker is already warm, you skip the cold start and only pay for execution time.
### Compute cost breakdown
Flash workers incur charges during these periods:
1. **Start time**: The time required to initialize a worker and load models into GPU memory. This includes starting the container, installing dependencies, and preparing the runtime environment.
2. **Execution time**: The time spent processing your request (running your `@Endpoint` decorated function).
3. **Idle timeout duration**: The time a worker remains active (running) after completing a request, waiting for additional requests before scaling down. Configure using the `idle_timeout` parameter in the [Endpoint class](/flash/configuration/parameters#idle_timeout).
### Pricing by resource type
Flash supports both GPU and CPU workers. Pricing varies based on the hardware type:
* **GPU workers**: Use `@Endpoint(gpu=...)` configuration. Pricing depends on the GPU type (e.g., RTX 4090, A100 80GB).
* **CPU workers**: Use `@Endpoint(cpu=...)` configuration. Pricing depends on the CPU instance type.
See the [Serverless pricing page](/serverless/pricing) for current rates by GPU and CPU type.
## How to estimate and optimize costs
To estimate costs for your Flash workloads, consider:
* How long each function takes to execute.
* How many concurrent workers you need (`workers` setting).
* Which GPU or CPU types you'll use.
* Your idle timeout configuration (`idle_timeout` setting).
### Cost optimization strategies
#### Choose appropriate hardware
Select the smallest GPU or CPU that meets your performance requirements. For example, if your workload fits in 24GB of VRAM, use an RTX 4090 or L4 instead of larger GPUs like the A100.
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
from runpod_flash import Endpoint, GpuType
# Cost-effective configuration for workloads that fit in 24GB VRAM
@Endpoint(
name="cost-optimized",
gpu=[GpuType.NVIDIA_GEFORCE_RTX_4090, GpuType.NVIDIA_L4]
)
def process(data): ...
```
#### Configure idle timeouts
Balance responsiveness and cost by adjusting the `idle_timeout` parameter. Shorter timeouts reduce idle costs but increase cold starts for sporadic traffic.
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
# Lower idle timeout for cost savings (more cold starts)
@Endpoint(
name="low-idle",
gpu=GpuGroup.ANY,
idle_timeout=5 # 5 seconds
)
def process(data): ...
# Higher idle timeout for responsiveness (higher idle costs)
@Endpoint(
name="responsive",
gpu=GpuGroup.ANY,
idle_timeout=30 # 30 seconds
)
def process(data): ...
```
#### Use CPU workers for non-GPU tasks
For data preprocessing, postprocessing, or other tasks that don't require GPU acceleration, use CPU workers instead of GPU workers.
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
from runpod_flash import Endpoint
# CPU configuration for non-GPU tasks
@Endpoint(
name="data-processor",
cpu="cpu5c-2-4" # 2 vCPU, 4GB RAM
)
def process_data(data): ...
```
#### Limit maximum workers
Set `workers` to prevent runaway scaling and unexpected costs:
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
@Endpoint(
name="controlled-scaling",
gpu=GpuGroup.ANY,
workers=3 # Limit to 3 concurrent workers (same as workers=(0, 3))
)
def process(data): ...
```
### Monitoring costs
Monitor your usage in the [Runpod console](https://console.runpod.io/serverless) to track:
* Total compute time across endpoints.
* Worker utilization and idle time.
* Cost breakdown by endpoint.
## Next steps
* [Create endpoint functions](/flash/create-endpoints) with optimized configurations.
* [View Serverless pricing details](/serverless/pricing) for current rates.
* [Configure resources](/flash/configuration/parameters) for your workloads.
# Get started with Flash
Source: https://docs.runpod.io/flash/quickstart
Run your first GPU workload with Flash in less than 5 minutes. Review setup, configuration, deployment, and usage guidance for Runpod Flash.
This quickstart gets you running GPU workloads on Runpod in minutes. You'll execute a function on a remote GPU and see the results immediately.
## Requirements
* [Runpod account](/accounts-billing/manage-accounts) with a verified email address.
* [An API key](/get-started/api-keys) with **All** access permissions to your Runpod account.
* [Python 3.10, 3.11, 3.12, or 3.13](https://www.python.org/downloads/) installed.
* [uv](https://docs.astral.sh/uv/) installed.
## Step 1: Install Flash
Flash runs natively on macOS and Linux. Windows users can run Flash through [WSL2](/flash/windows-wsl2).
Create a virtual environment and install Flash using [uv](https://docs.astral.sh/uv/):
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
uv venv
source .venv/bin/activate
uv pip install runpod-flash
```
When you create a project with `flash init`, Flash generates `AGENTS.md` (and `CLAUDE.md` as a symlink) with CLI-first rules for AI coding assistants. These files tell AI tools like Claude Code, Cursor, and Codex to use Flash commands instead of raw Runpod API calls. See [`flash init`](/flash/cli/init) for details.
### Optional: Install coding agent integration
If you're using an AI coding agent like Claude Code, Cline, or Cursor, you can install the Flash skill package to give your agent detailed context about the Flash SDK:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
npx skills add runpod/runpod-plugins-official
```
This enables your coding agent to provide more accurate Flash code suggestions and troubleshooting help.
## Step 2: Authenticate with Runpod
Log in to your Runpod account:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
flash login
# If using uv:
uv run flash login
```
This opens your browser to authorize Flash. After you approve, your credentials are saved, allowing you to run Flash commands and scripts.
Alternatively, you can set the `RUNPOD_API_KEY` environment variable or add it to a `.env` file. See [`flash login`](/flash/cli/login) for details.
## Step 3: Copy this code
Create a file called `gpu_demo.py` and paste this code into it:
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
import asyncio
from runpod_flash import Endpoint, GpuGroup
@Endpoint(
name="flash-quickstart",
gpu=GpuGroup.ANY, # Use any available GPU
workers=3,
idle_timeout=300, # Keep worker running for 5 minutes
dependencies=["numpy", "torch"]
)
def gpu_matrix_multiply(size):
# IMPORTANT: Import packages INSIDE the function
import numpy as np
import torch
# Get GPU name
device_name = torch.cuda.get_device_name(0)
# Create random matrices
A = np.random.rand(size, size)
B = np.random.rand(size, size)
# Multiply matrices
C = np.dot(A, B)
return {
"matrix_size": size,
"result_mean": float(np.mean(C)),
"gpu": device_name
}
# Call the function
async def main():
print("Running matrix multiplication on Runpod GPU...")
result = await gpu_matrix_multiply(1000)
print(f"\n✓ Matrix size: {result['matrix_size']}x{result['matrix_size']}")
print(f"✓ Result mean: {result['result_mean']:.4f}")
print(f"✓ GPU used: {result['gpu']}")
if __name__ == "__main__":
asyncio.run(main())
```
Make sure you activate your virtual environment in the same directory where you created the `gpu_demo.py` file. If you open a new terminal, run `source .venv/bin/activate` before executing the script.
## Step 4: Run it
Execute the script:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
python gpu_demo.py
```
You'll see Flash provision a GPU worker and execute your function:
```text theme={"theme":{"light":"github-light","dark":"github-dark"}}
Running matrix multiplication on Runpod GPU...
Creating endpoint: flash-quickstart
Provisioning Serverless endpoint...
Endpoint ready
Executing function on Runpod endpoint ID: xvf32dan8rcilp
Initial job status: IN_QUEUE
Job completed, output received
✓ Matrix size: 1000x1000
✓ Result mean: 249.8286
✓ GPU used: NVIDIA RTX A5000
```
The first run takes 30-60 seconds, while Runpod provisions the endpoint, installs dependencies, and starts a worker. Subsequent runs take 2-3 seconds (because the worker is already running).
If you're having authorization issues, you can set your API key directly in your terminal:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
export RUNPOD_API_KEY="your_key"
```
Replace `your_key` with your actual API key from the [Runpod console](https://console.runpod.io/user/settings).
## Step 5: Update and run again
With your endpoint running, make a change and run the script again:
1. Open `gpu_demo.py` and change the matrix size from `1000` to `2000`:
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
result = await gpu_matrix_multiply(2000)
```
2. Run the script again:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
python gpu_demo.py
```
This time, the result should appear in 1-3 seconds instead of 30-60 seconds, injects the code into the running worker so code changes take effect immediately without reprovisioning.
This instant iteration is one of Flash's key features. You can develop and test GPU code as quickly as local development, even though it runs on remote hardware.
## Step 6: Understand what you just did
Let's break down the code you just ran:
### Imports and setup
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
import asyncio
from runpod_flash import Endpoint, GpuGroup
```
* **`asyncio`**: Enables asynchronous execution (endpoint functions run async).
* **`Endpoint`**: The class that marks functions for remote execution.
* **`GpuGroup`**: Enum for selecting GPU types or groups of GPUs.
Flash automatically loads your credentials from `flash login` or the `RUNPOD_API_KEY` environment variable.
### The `@Endpoint` decorator
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
@Endpoint(
name="flash-quickstart",
gpu=GpuGroup.ANY,
workers=3,
idle_timeout=300,
dependencies=["numpy", "torch"]
)
def gpu_matrix_multiply(size):
import numpy as np
import torch
# Get GPU name
device_name = torch.cuda.get_device_name(0)
# Create random matrices
A = np.random.rand(size, size)
B = np.random.rand(size, size)
# Multiply matrices
C = np.dot(A, B)
return {
"matrix_size": size,
"result_mean": float(np.mean(C)),
"gpu": device_name
}
```
The `@Endpoint` decorator configures everything in one place:
* **`name`**: Identifies your endpoint in the [Runpod console](https://console.runpod.io/serverless).
* **`gpu`**: Which GPU to use (`GpuGroup.ANY` accepts any available GPU for faster provisioning).
* **`workers`**: Maximum parallel workers (allows 3 concurrent executions).
* **`idle_timeout`**: Seconds a worker stays active after completing a request before scaling down. Setting this to 300 (5 minutes) gives you more time to iterate on your code while the worker remains warm.
* **`dependencies`**: Python packages to install on the worker.
* **Function body**: The matrix multiplication code runs on the remote GPU, not your local machine.
* **Return value**: The result is returned to your local machine as a Python dictionary.
See [GPU types](/flash/configuration/gpu-types) for available GPUs or [endpoint functions](/flash/create-endpoints) for all configuration options.
You must import packages **inside the function body**, not at the top of your file. These imports need to happen on the remote worker.
### Calling the function
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
async def main():
print("Running matrix multiplication on Runpod GPU...")
result = await gpu_matrix_multiply(1000)
print(f"\n✓ Matrix size: {result['matrix_size']}x{result['matrix_size']}")
print(f"✓ Result mean: {result['result_mean']:.4f}")
print(f"✓ GPU used: {result['gpu']}")
if __name__ == "__main__":
asyncio.run(main())
```
Here's what happens when you call an `@Endpoint` decorated function:
1. Flash checks if the endpoint specified in your decorator already exists.
* If yes: It updates the endpoint if the configuration has changed.
* If no: It creates a new endpoint, initializes a worker, and installs your dependencies.
2. Flash sends your code to the GPU worker
3. The GPU worker executes the function with the provided inputs.
4. The result is returned to your local machine as a Python dictionary, where it's printed in your terminal.
Everything outside the `@Endpoint` function (all the `print` statements, etc.) runs **locally on your machine**. Only the decorated function runs remotely.
## Step 7: Run multiple operations in parallel
Flash makes it easy to run multiple GPU operations concurrently. Replace your `main()` function with the code below:
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
async def main():
print("Running 3 matrix operations in parallel...")
# Run all three operations at once
results = await asyncio.gather(
gpu_matrix_multiply(500),
gpu_matrix_multiply(1000),
gpu_matrix_multiply(2000)
)
# Print results
for i, result in enumerate(results, 1):
print(f"\n{i}. Size: {result['matrix_size']}x{result['matrix_size']}")
print(f" Mean: {result['result_mean']:.4f}")
print(f" GPU: {result['gpu']}")
```
Run the script again:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
python gpu_demo.py
```
All three operations execute simultaneously:
```text theme={"theme":{"light":"github-light","dark":"github-dark"}}
Running 3 matrix operations in parallel...
Initial job status: IN_QUEUE
Initial job status: IN_QUEUE
Initial job status: IN_QUEUE
Job completed, output received
Job completed, output received
Job completed, output received
1. Size: 500x500
Mean: 125.3097
GPU: NVIDIA RTX A5000
2. Size: 1000x1000
Mean: 249.9442
GPU: NVIDIA RTX A5000
3. Size: 2000x2000
Mean: 500.1321
GPU: NVIDIA RTX A5000
```
## Clean up
When you're done testing, clean up the endpoints:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
# List all endpoints
flash undeploy list
# Remove the quickstart endpoint
flash undeploy flash-quickstart
# Or remove all endpoints
flash undeploy --all
# If using uv:
uv run flash undeploy list
uv run flash undeploy flash-quickstart
uv run flash undeploy --all
```
## Next steps
You've successfully run GPU code on Runpod! Now you're ready to learn more about Flash:
Use Stable Diffusion XL to generate images from text prompts.
Learn how to configure and optimize endpoint functions.
Deploy production APIs.
Browse example Flash scripts and apps on GitHub.
## Troubleshooting
### Authentication error
```text theme={"theme":{"light":"github-light","dark":"github-dark"}}
Error: API key is not set
```
**Solution**: Run `flash login` to authenticate with your Runpod account:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
flash login
# If using uv:
uv run flash login
```
Alternatively, set the `RUNPOD_API_KEY` environment variable:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
export RUNPOD_API_KEY="your_key"
```
### Template name conflict
```text theme={"theme":{"light":"github-light","dark":"github-dark"}}
Error: endpoint template names must be unique
```
**Solution**: Each endpoint needs a unique `name`. If you've deployed an endpoint before with the same name, either:
* Use a different name for your new endpoint
* Undeploy the existing endpoint with `flash undeploy --force`
### Job stuck in queue
```text theme={"theme":{"light":"github-light","dark":"github-dark"}}
Initial job status: IN_QUEUE
[Stays in queue for >60 seconds]
```
**Solution**: No GPUs available. Use `GpuGroup.ANY` to accept any available GPU:
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
@Endpoint(
name="flash-quickstart",
gpu=GpuGroup.ANY,
dependencies=["numpy", "torch"]
)
def gpu_matrix_multiply(size):
...
```
Or add multiple specific GPU types for fallback:
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
@Endpoint(
name="flash-quickstart",
gpu=[
GpuType.NVIDIA_GEFORCE_RTX_4090,
GpuType.NVIDIA_RTX_A5000,
GpuType.NVIDIA_RTX_A6000
],
dependencies=["numpy", "torch"]
)
def gpu_matrix_multiply(size):
...
```
You can also check [GPU availability](https://console.runpod.io/serverless) in the console.
### Import errors
```text theme={"theme":{"light":"github-light","dark":"github-dark"}}
ModuleNotFoundError: No module named 'numpy'
```
**Solution**: Move imports inside the `@Endpoint` function:
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
@Endpoint(name="compute", gpu=GpuGroup.ANY, dependencies=["numpy"])
def my_function():
import numpy as np # Import here, not at top of file
# ...
```
See the [execution model](/flash/execution-model#common-execution-issues) for more troubleshooting.
# Troubleshooting
Source: https://docs.runpod.io/flash/troubleshooting
Monitor, debug, and troubleshoot Flash deployments. Review setup, configuration, deployment, and usage guidance for Runpod Flash.
This guide covers how to monitor your Flash deployments, debug issues, and resolve common errors.
## Monitoring and debugging
### Viewing logs
When running Flash functions, logs are displayed in your terminal:
```text theme={"theme":{"light":"github-light","dark":"github-dark"}}
2025-11-19 12:35:15,109 | INFO | Created endpoint: rb50waqznmn2kg - flash-quickstart
2025-11-19 12:35:15,114 | INFO | Endpoint:rb50waqznmn2kg | API /run
2025-11-19 12:35:15,655 | INFO | Endpoint:rb50waqznmn2kg | Started Job:b0b341e7-...
2025-11-19 12:35:15,762 | INFO | Job:b0b341e7-... | Status: IN_QUEUE
2025-11-19 12:36:09,983 | INFO | Job:b0b341e7-... | Status: COMPLETED
2025-11-19 12:36:10,068 | INFO | Worker:icmkdgnrmdf8gz | Delay Time: 51842 ms
2025-11-19 12:36:10,068 | INFO | Worker:icmkdgnrmdf8gz | Execution Time: 1533 ms
```
Control log verbosity with the `LOG_LEVEL` environment variable:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
LOG_LEVEL=DEBUG python your_script.py
```
Available levels: `DEBUG`, `INFO`, `WARNING`, `ERROR`.
### Runpod console
View detailed metrics and logs in the [Runpod console](https://console.runpod.io/serverless):
1. Navigate to the **Serverless** section.
2. Click on your endpoint to view:
* Active workers and queue depth.
* Request history and job status.
* Worker logs and execution details.
The console provides metrics including request rate, queue depth, latency, worker count, and error rate.
### View worker logs
Access detailed logs for specific workers:
1. Go to the [Serverless console](https://console.runpod.io/serverless).
2. Select your endpoint.
3. Click on a worker to view its logs.
Logs include dependency installation output, function execution output (print statements, errors), and system-level messages.
### Add logging to functions
Include print statements in your endpoint functions for debugging:
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
@Endpoint(name="processor", gpu=GpuGroup.ANY)
async def process(data: dict) -> dict:
print(f"Received data: {data}") # Visible in worker logs
result = do_processing(data)
print(f"Processing complete: {result}")
return result
```
## Configuration errors
### API key not set
**Error:**
```
No Runpod API key found. Set one with:
flash login # interactive setup
or
export RUNPOD_API_KEY= # environment variable
or
echo 'RUNPOD_API_KEY=' >> .env
Get a key: https://docs.runpod.io/get-started/api-keys
```
**Cause:** Flash requires a valid Runpod API key to provision and manage endpoints.
**Solution:**
1. Generate an API key from [Settings > API Keys](https://console.runpod.io/user/settings) in the Runpod console. The key needs **All** access permissions.
2. Authenticate using one of these methods:
**Option 1: Use `flash login` (recommended)**
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
flash login
```
Opens your browser for authentication and saves your credentials.
**Option 2: Environment variable**
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
export RUNPOD_API_KEY="your_api_key"
```
**Option 3: .env file for local CLI use**
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
echo "RUNPOD_API_KEY=your_api_key" >> .env
```
Values in your `.env` file are only available locally for CLI commands. They are not passed to deployed endpoints.
**Option 4: Shell profile for persistent local access**
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
echo 'export RUNPOD_API_KEY="your_api_key"' >> ~/.bashrc
source ~/.bashrc
```
### Corrupted credentials file
**Error:**
```
Error: ~/.runpod/config.toml is corrupted and cannot be parsed.
Run 'flash login' to re-authenticate, or delete the file and retry.
```
**Cause:** The credentials file at `~/.runpod/config.toml` contains invalid TOML and cannot be read. This can also appear as "No API key found" even after a successful `flash login`.
**Solution:** Delete the credentials file and re-authenticate:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
rm ~/.runpod/config.toml
flash login
```
### Invalid route configuration
**Error:**
```
Load-balanced endpoints require route decorators
```
**Cause:** Load-balanced endpoints require HTTP method decorators for each route.
**Solution:** Ensure all routes use the correct decorator pattern:
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
from runpod_flash import Endpoint
api = Endpoint(name="api", cpu="cpu5c-4-8", workers=(1, 5))
# Correct - using route decorators
@api.post("/process")
async def process_data(data: dict) -> dict:
return {"result": "processed"}
@api.get("/health")
async def health_check() -> dict:
return {"status": "healthy"}
```
### Invalid HTTP method
**Error:**
```
method must be one of {'GET', 'POST', 'PUT', 'DELETE', 'PATCH'}
```
**Cause:** The HTTP method specified is not supported.
**Solution:** Use one of the supported HTTP methods: `GET`, `POST`, `PUT`, `DELETE`, or `PATCH`.
### Invalid path format
**Error:**
```
path must start with '/'
```
**Cause:** HTTP paths must begin with a forward slash.
**Solution:** Ensure paths start with `/`:
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
# Correct
@api.get("/health")
# Incorrect
@api.get("health")
```
### Duplicate routes
**Error:**
```
Duplicate route 'POST /process' in endpoint 'my-api'
```
**Cause:** Two functions define the same HTTP method and path combination.
**Solution:** Ensure each route is unique within an endpoint. Either change the path or method of one function.
## Build errors
### Unsupported Python version
**Error:**
```
Local Python 3.9 is not supported by Flash workers (supported: 3.10, 3.11, 3.12, 3.13).
Pass --python-version, declare python_version on a resource config, or run flash from a supported interpreter.
```
**Cause:** Flash supports Python 3.10, 3.11, 3.12, and 3.13. Your local Python version is outside this range.
**Solution:**
You have three options:
1. **Use the `--python-version` CLI flag** to override local detection:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
flash build --python-version 3.12
flash deploy --python-version 3.12
```
2. **Declare `python_version` on your resource configs:**
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
@Endpoint(name="my-endpoint", gpu=GpuGroup.ANY, python_version="3.12")
```
3. **Switch to a supported Python version** using a virtual environment:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
# Using pyenv
pyenv install 3.12
pyenv local 3.12
# Or using uv
uv venv --python 3.12
source .venv/bin/activate
```
Python 3.12 is recommended for best performance with no cold-start overhead. Python 3.10, 3.11, and 3.13 incur additional cold-start overhead on GPU workers because an alternative Python interpreter must be installed.
### Local module could not be resolved
**Error:**
```
endpoint.py: relative import (level=1, module='helpers') could not be resolved to a local file under /path/to/project
```
Or, for an ignore-excluded import:
```
Shipped code imports local modules that the build ignore rules (.gitignore or built-in defaults) exclude:
utils/helpers.py (imported by endpoint.py)
Shipping them would silently override a deliberate exclusion, and omitting them would break the worker with ModuleNotFoundError. Remove the matching ignore pattern or stop importing these modules from shipped code.
```
**Cause:** This `LocalModuleResolutionError` has two variants: an `@Endpoint` file imports a local module that Flash can't resolve, or shipped code imports a local module that an ignore pattern excludes.
**Solution:**
For an unresolvable import:
1. Fix the import so it points to a file that exists under your project root.
2. Move the imported module under the project directory if it lives outside it.
3. Confirm that package imports have an `__init__.py` file.
For an ignore-excluded import, remove the matching ignore pattern, or stop importing that module from your shipped code.
For how Flash resolves and bundles local imports, see [Import local modules](/flash/create-endpoints#import-local-modules).
## Deployment errors
### Tarball too large
**Error:**
```
Tarball exceeds maximum size. File size: 1.6GB, Max: 1.5GB
```
**Cause:** The deployment package exceeds the 1.5GB limit.
**Solution:**
1. Check for large files that shouldn't be included (datasets, model weights, logs).
2. Add large files to `.gitignore` to exclude them from the build.
3. Use [network volumes](/flash/configuration/storage) to store large models instead of bundling them.
### Invalid tarball format
**Error:**
```
File is not a valid gzip file. Expected magic bytes (31, 139)
```
**Cause:** The build artifact is corrupted or not a valid gzip file.
**Solution:** Delete the `.flash` directory and rebuild:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
rm -rf .flash
flash build
```
### SSL certificate verification failed
**Error:**
```
SSL certificate verification failed. This usually means Python cannot find your system's CA certificates.
```
**Cause:** Python cannot locate the system's trusted CA certificates, preventing secure connections during deployment. This commonly occurs on fresh Python installations, especially on macOS.
**Solution:** Try one of these fixes:
1. **Install certifi and set the certificate bundle path:**
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
pip install certifi
export REQUESTS_CA_BUNDLE=$(python -c "import certifi; print(certifi.where())")
```
2. **macOS only:** Run the certificate installer that comes with Python. Find it in your Python installation folder (typically `/Applications/Python 3.x/`) and run `Install Certificates.command`.
3. **Add to shell profile for persistence:**
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
echo 'export REQUESTS_CA_BUNDLE=$(python -c "import certifi; print(certifi.where())")' >> ~/.bashrc
source ~/.bashrc
```
Transient SSL errors (like connection resets) are automatically retried during upload. The certificate verification error requires manual intervention because it indicates a system configuration issue.
### Resource provisioning failed
**Error:**
```
Failed to provision resources: [error details]
```
**Cause:** Flash couldn't create the Serverless endpoint on Runpod.
**Solutions:**
1. **Check GPU availability**: The requested GPU types may not be available. Add fallback options:
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
gpu=[GpuType.NVIDIA_A100_80GB_PCIe, GpuType.NVIDIA_RTX_A6000, GpuType.NVIDIA_GEFORCE_RTX_4090]
```
2. **Check account limits**: You may have hit worker capacity limits. Contact [Runpod support](https://www.runpod.io/contact) to increase limits.
3. **Check network volume**: If using `volume=`, verify the volume exists and is in a compatible datacenter.
## Runtime errors
### Endpoint not deployed
**Error:**
```
Endpoint URL not available - endpoint may not be deployed
```
**Cause:** The endpoint function was called before the endpoint finished provisioning.
**Solutions:**
1. **For standalone scripts**: Ensure the endpoint has time to provision. Flash handles this automatically, but network issues can cause delays.
2. **For Flash apps**: Deploy the app first with `flash deploy`, then call the endpoint.
3. **Check endpoint status**: View your endpoints in the [Serverless console](https://console.runpod.io/serverless).
### Execution timeout
**Error:**
```
Execution timeout on [endpoint] after [N]s
```
**Cause:** The endpoint function took longer than the configured timeout.
**Solutions:**
1. **Increase timeout**: Set `execution_timeout_ms` in your configuration:
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
@Endpoint(
name="long-running",
gpu=GpuType.NVIDIA_A100_80GB_PCIe,
execution_timeout_ms=600000 # 10 minutes
)
```
2. **Optimize function**: Profile your function to identify bottlenecks.
3. **Use queue-based endpoints**: For long-running tasks, use the `@Endpoint` decorator pattern. Queue-based endpoints are designed for longer operations.
### Connection failed
**Error:**
```
Failed to connect to endpoint [name] ([url])
```
**Cause:** Network connectivity issue between your local environment and the Runpod endpoint.
**Solutions:**
1. **Check internet connection**: Verify you have network access.
2. **Retry**: Transient network issues often resolve on retry. Flash includes automatic retry logic.
3. **Check endpoint status**: Verify the endpoint is running in the [Serverless console](https://console.runpod.io/serverless).
### HTTP errors from endpoint
**Error:**
```
HTTP error from endpoint [name]: 500 - Internal Server Error
```
**Cause:** The endpoint function raised an exception during execution.
**Solutions:**
1. **Check logs**: View worker logs in the [Serverless console](https://console.runpod.io/serverless) for detailed error messages.
2. **Test locally**: Use `flash dev` to test your function locally before deploying.
3. **Add error handling**: Wrap your function logic in try/except to provide better error messages:
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
@Endpoint(name="processor", gpu=GpuGroup.ANY)
async def process(data: dict) -> dict:
try:
# Your logic here
return {"result": "success"}
except Exception as e:
return {"error": str(e)}
```
### Serialization errors
**Error:**
```
Failed to deserialize result: [error]
```
**Cause:** The function's return value cannot be serialized/deserialized.
**Solutions:**
1. **Use simple types**: Return dictionaries, lists, strings, numbers, and other JSON-serializable types.
2. **Avoid complex objects**: Don't return PyTorch tensors, NumPy arrays, or custom classes directly. Convert them first:
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
# Correct
return {"result": tensor.tolist()}
# Incorrect - tensor is not serializable
return {"result": tensor}
```
3. **Check argument types**: Input arguments must also be serializable.
### Payload too large
**Error:**
```
Payload size X MB exceeds limit of 10.0 MB
```
**Cause:** The serialized argument exceeds the 10 MB limit. Flash uses base64 encoding, which expands data by approximately 33%, so roughly 7.5 MB of raw data becomes 10 MB when encoded.
**Solutions:**
1. **Use network volumes for large data**: Save large data to a network volume and pass the file path:
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
@Endpoint(name="processor", gpu=GpuGroup.ANY, volume="vol_abc123")
async def process(file_path: str) -> dict:
import numpy as np
data = np.load(file_path) # Load from volume
return {"result": process_data(data)}
```
2. **Compress data before sending**: For data that must be passed directly, use compression:
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
import gzip
compressed = gzip.compress(data.tobytes())
# Pass compressed bytes instead
```
3. **Split large requests**: Break large datasets into smaller chunks and process them in multiple requests.
### Local module payload too large
**Error:**
```
Inline module payload is X bytes, over the 8388608-byte live-serverless cap.
Use `flash deploy` for endpoints with large local dependencies.
```
**Cause:** On the live execution path (calling an `@Endpoint` function directly or during `flash dev`), Flash ships your local module source inline with the request, and the combined source exceeds the 8 MiB cap.
**Solution:** Deploy the app with `flash deploy`, which bundles local modules into the build artifact instead of the request payload:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
flash deploy
# If using uv:
uv run flash deploy
```
See [Live execution size limit](/flash/create-endpoints#live-execution-size-limit) for details on how local module source is shipped.
### Deserialization timeout
**Error:**
```
Deserialization timed out after 30s
```
**Cause:** The deserialization process took longer than 30 seconds. This usually indicates malformed or corrupted serialized data that causes the unpickle operation to hang.
**Solution:** Verify your input data is properly serialized. If you're manually constructing payloads, ensure the data was serialized using `cloudpickle` and encoded with base64. The Flash SDK handles this automatically for programmatic calls.
### Circuit breaker open
**Error:**
```
Circuit breaker is open. Retry in [N] seconds
```
**Cause:** Too many consecutive failures to the endpoint triggered the circuit breaker protection.
**Solutions:**
1. **Wait and retry**: The circuit breaker will automatically attempt recovery after the timeout (typically 60 seconds).
2. **Check endpoint health**: Multiple failures usually indicate an underlying issue. Check logs and endpoint status.
3. **Fix the root cause**: Address whatever is causing the repeated failures before retrying.
## GPU availability issues
### Job stuck in queue
**Symptom:** Job status shows `IN_QUEUE` for extended periods.
**Cause:** The requested GPU types are not available.
**Solutions:**
1. **Add fallback GPUs**: Expand your `gpu` list with additional options:
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
@Endpoint(
name="flexible",
gpu=[
GpuType.NVIDIA_A100_80GB_PCIe, # First choice
GpuType.NVIDIA_RTX_A6000, # Fallback
GpuType.NVIDIA_GEFORCE_RTX_4090 # Second fallback
]
)
```
2. **Use GpuGroup.ANY**: For development, accept any available GPU:
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
gpu=GpuGroup.ANY
```
3. **Check availability**: View GPU availability in the [Serverless console](https://console.runpod.io/serverless).
4. **Contact support**: For guaranteed capacity, contact [Runpod support](https://www.runpod.io/contact).
## Dependency errors
### Module not found
**Error (in worker logs):**
```
ModuleNotFoundError: No module named 'transformers'
```
**Cause:** A required dependency was not specified in the `@Endpoint` decorator.
**Solution:** Add all required packages to the `dependencies` parameter:
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
@Endpoint(
name="processor",
gpu=GpuGroup.ANY,
dependencies=["transformers", "torch", "pillow"]
)
async def process(data: dict) -> dict:
from transformers import pipeline
# ...
```
### Version conflicts
**Symptom:** Function fails with import errors or unexpected behavior.
**Cause:** Dependency version conflicts between packages.
**Solution:** Pin specific versions:
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
@Endpoint(
name="processor",
gpu=GpuGroup.ANY,
dependencies=[
"transformers==4.36.0",
"torch==2.1.0",
"accelerate>=0.25.0"
]
)
```
## Getting help
If you're still stuck:
1. **Discord**: Join the [Runpod Discord](https://discord.com/invite/cUpRmau42V) for community support.
2. **GitHub Issues**: Report bugs or request features on the [Flash repository](https://github.com/runpod/flash).
3. **Support**: Contact [Runpod support](https://www.runpod.io/contact) for account-specific issues.
# Use Flash on Windows
Source: https://docs.runpod.io/flash/windows-wsl2
Set up WSL2 with Ubuntu to run Flash on Windows. Review setup, configuration, deployment, and usage guidance for Runpod Flash.
Flash runs natively on macOS and Linux. On Windows, you can run Flash through Windows Subsystem for Linux (WSL2), which provides a full Linux environment on your machine.
## Requirements
* Windows 10 version 2004 or later, or Windows 11.
* Administrator access to your Windows machine.
* [Runpod account](/accounts-billing/manage-accounts) with a verified email address.
* [An API key](/get-started/api-keys) with **All** access permissions.
## Step 1: Enable WSL2
Open PowerShell or Command Prompt as Administrator (right-click and select "Run as administrator"), then run:
```powershell theme={"theme":{"light":"github-light","dark":"github-dark"}}
wsl --install
```
This command enables WSL, installs the latest Linux kernel, sets WSL2 as the default version, and installs Ubuntu as your Linux distribution.
If WSL is already installed on your system, you can install Ubuntu specifically with:
```powershell theme={"theme":{"light":"github-light","dark":"github-dark"}}
wsl --install -d Ubuntu
```
After the installation completes, restart your computer when prompted.
## Step 2: Set up Ubuntu
After restarting, Ubuntu launches automatically and prompts you to create a Linux username and password. These credentials are separate from your Windows account and are used only within the Linux environment.
If Ubuntu doesn't launch automatically, open it from the Start menu by searching for "Ubuntu".
Once setup is complete, you'll see the Ubuntu terminal prompt:
```text theme={"theme":{"light":"github-light","dark":"github-dark"}}
username@hostname:~$
```
Update your package lists to ensure you have access to the latest software:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
sudo apt update && sudo apt upgrade -y
```
## Step 3: Install Python and uv
Flash requires Python 3.10 or later. Ubuntu typically includes Python, but you should verify the version:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
python3 --version
```
If you need a newer version, install it:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
sudo apt install python3.12 python3.12-venv -y
```
Install [uv](https://docs.astral.sh/uv/), a fast Python package manager:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
curl -LsSf https://astral.sh/uv/install.sh | sh
```
After installation, restart your terminal or run the following to add uv to your path:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
source $HOME/.local/bin/env
```
## Step 4: Install and authenticate Flash
Create a project directory and set up a virtual environment:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
mkdir my-flash-project && cd my-flash-project
uv venv
source .venv/bin/activate
```
Install Flash:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
uv pip install runpod-flash
```
Authenticate with your Runpod account:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
flash login
```
This opens your browser to authorize Flash. After you approve, your credentials are saved.
Alternatively, you can set your API key directly:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
export RUNPOD_API_KEY="your_key"
```
## Step 5: Verify your installation
Test that Flash is working correctly:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
flash --version
```
You're now ready to use Flash. Continue with the [quickstart](/flash/quickstart) to run your first GPU workload.
## Tips for working with WSL2
**Accessing Windows files**: Your Windows drives are mounted under `/mnt/`. For example, `C:\Users\YourName\Documents` is accessible at `/mnt/c/Users/YourName/Documents`.
**Opening WSL from any folder**: In Windows Explorer, type `wsl` in the address bar to open Ubuntu in that directory.
**VS Code integration**: Install the [WSL extension](https://marketplace.visualstudio.com/items?itemName=ms-vscode-remote.remote-wsl) for VS Code to edit files in WSL with full IDE support. Run `code .` from your WSL terminal to open VS Code in the current directory.
**Default terminal**: You can set Ubuntu as your default terminal in Windows Terminal for quicker access.
# Deploy your first Pod
Source: https://docs.runpod.io/get-started
Run code on a remote GPU in minutes. Review setup steps, core concepts, and recommended next actions for building on Runpod.
Follow this guide to learn how to create an account, deploy your first GPU , and use it to execute code remotely.
## Step 1: Create an account
Start by creating a Runpod account:
1. [Sign up here](https://www.console.runpod.io/signup).
2. Verify your email address.
3. Set up two-factor authentication (recommended for security).
## Step 2: Deploy a Pod
Now that you've created your account, you're ready to deploy your first Pod:
Use this path to launch a Pod from the console. Configure your workload, choose a region and GPU, review the pricing, and deploy.
Runpod is rolling out an updated deployment flow through [early access](https://console.runpod.io/user/early-access). Use the tabs below to follow the version that matches what you see in the console.
To open the deploy page, click **+ New** in the top-right corner of the console and select **Pod**.
You can also click **Pods** in the left sidebar.
These instructions describe the early access version of the deployment flow. If you don't see this flow, follow the **Legacy flow** tab, or enable the new flow from [Account → Early access](https://console.runpod.io/user/early-access).
In the **Workload** panel at the top of the page:
* **Template**: select the container image your Pod will run. Use the search box to find a template, or click **Explore all** to browse. The selected template and its image appear below the search bar. Click **Set overrides** to customize environment variables, exposed ports, or container start commands without editing the template itself.
* **Pod name**: a name is auto-generated. You can replace it with any name you prefer.
* **Options**: when you select an [official Runpod template](/pods/templates/overview), two extra options appear. These options are not shown for community templates.
* **Start Jupyter notebook**: launches a Jupyter server when the Pod starts, accessible from the console. Enabled by default.
* **SSH terminal access**: enables SSH into the Pod. If enabled, paste your SSH public key in the field that appears. [Learn how to create an SSH key](/pods/configuration/use-ssh).
Set the **Region** for your Pod. It's set to **Any region** by default. Click to restrict deployment to a specific geographic region.
In the **Compute** panel, choose a GPU from one of four tabs:
* **Available**: shows only GPUs with capacity right now.
* **Recommended**: shows GPUs recommended by Runpod and by the template maintainer.
* **All**: shows all GPUs, including those that are out of capacity and those marked incompatible by the template.
* **Recent**: shows all GPUs you've deployed in the last 7 days. This tab only appears if you have a recent deployment history.
Use the search box, **Network volume** filter, **Filter** button, or **Sort by** dropdown to narrow the list. Click **Compare GPUs** to ask the Runpod assistant for a comparison of the selected GPUs.
The **Network volume** filter narrows the GPU list to compatible GPUs by filtering for GPUs that are in the same data center as the selected network volume. Select any network volume, and GPU availability updates to match it. GPUs that aren't compatible with the selected volume move to the incompatible section of the **All** tab.
If the GPU you want shows **Out of capacity**, you can still select it and deploy it once capacity frees up. Selecting an out-of-capacity GPU changes the final [Deploy](#deploy) step: instead of deploying right away, you can subscribe to deploy when the GPU becomes available.
Pods offer two kinds of storage.
**Container disk**
This is the container's primary storage, and it's wiped whenever the Pod is stopped.
**Persistent storage**
Persistent storage keeps your data across stops and restarts, and you can mount it at any location. It's mounted at `/workspace` by default, but you can change this with a template override, or the template itself can set a different default. It comes in two types, and you can attach one or the other but not both:
* **Volume disk**: a disk attached directly to your Pod that keeps its data across stops and restarts but is deleted when the Pod is terminated.
* **Network volume**: permanent storage that exists independently of any Pod, so you can attach the same volume to different Pods over time.
If you did select a network volume in the **Compute** step, the persistent storage selection in this step is locked to that network volume.
GPUs that aren't compatible with any network volume have volume disk selected automatically, and you can't select a network volume for them.
For a full comparison, see [Storage options](/pods/storage/types).
The **Summary** panel on the right shows:
* The selected template and GPU.
* Total cost per hour (billed per millisecond).
* A breakdown of GPU cost, container disk cost, persistent storage cost, and stopped cost.
Verify the configuration looks correct before deploying.
Click **Deploy Pod** to launch your Pod.
Your Pod appears under **Pods** in the left sidebar. It may take a moment to reach the running state while the container image is pulled.
If you haven't set up payments yet, you'll be prompted to add a payment method and purchase credits for your account.
If you selected an out-of-capacity GPU, the **Summary** panel shows **Instance not available** and a **Deploy when available** button in place of **Deploy Pod**. Runpod queues your Pod and deploys it automatically as soon as that GPU frees up within the time window you choose.
In the right panel, click the **Deploy when available** button to open the subscription modal.
In the **Deploy when available** modal:
* **Notifications**: choose how you want to be notified when your Pod deploys:
* **Email**: sends a notification to one of your account's email addresses. Select which address to use. If you're a member of a team, you can also select the team's email address.
* **In-console notification**: shows an alert inside the Runpod console.
Both are enabled by default.
* **Subscription window**: set the time range during which Runpod monitors for availability and auto-deploys your Pod:
* The default window is 24 hours from now.
* Click the date range to adjust the start and end time.
* Check **Use advanced scheduler** to fine-tune the hours when Runpod can deploy your request, rather than monitoring continuously across the entire window.
Click **Subscribe** to confirm. Runpod monitors for availability and deploys your Pod automatically when the selected GPU becomes free within your subscription window.
You'll receive a notification (via email or in-console, depending on your settings) when the Pod has been deployed.
If you don't have enough funds available when the GPU becomes available, your deploy when available subscription fails and no Pod is deployed. To retry, you'll need to subscribe again.
If the GPU doesn't become available within your subscription window, your subscription expires and no Pod is deployed. You can create a new subscription at any time.
The deploy page opens to a grid of available GPUs. Use the controls at the top of the grid to narrow the list, then click a card to configure it.
* Filter the list with the row of toggles: **Secure Cloud**, **Network volume**, **Any region**, **Global Networking**, and **Additional filters**.
* Drag the **VRAM** slider to set a minimum amount of VRAM.
* Use the search box to find a GPU by name.
* Browse GPUs by group: Featured GPUs, NVIDIA latest generation, NVIDIA previous generation, and AMD.
Each card shows the GPU's price per hour, VRAM, maximum count, and current availability (High, Medium, Low, or Unavailable).
Click a card to select that GPU and open the Configure deployment panel below.
After you select a GPU, the Configure deployment panel opens below the grid. Set the following:
* **Pod name**: a name is auto-generated. You can replace it with any name you prefer.
* **Pod template**: click **Edit** to modify the current template's settings, or **Change template** to swap it for a different one.
* **GPU count**: choose how many GPUs to attach, from 1 to 8.
* **Instance pricing**: choose **On-Demand** for standard pricing, or **Reserved** to reserve capacity (click **Talk to sales** to arrange a reservation).
* **Options**: enable **Encrypt volume**, **SSH terminal access**, or **Start Jupyter notebook**. Start Jupyter notebook is enabled by default.
* **Storage configuration**: set the **Container disk** size, then choose between a **Network volume** and a **Volume disk** for persistent storage.
You must complete the storage configuration before you can deploy. If you haven't selected persistent storage, a warning appears: "No network volume has been selected. Please select a network volume above or switch to volume disk." Select a network volume or switch to a volume disk to enable the deploy button.
Review the **Pricing summary** and **Pod summary** to confirm your configuration, then click **Deploy On-Demand** to deploy and start your Pod. You'll be redirected back to the Pods page after a few seconds.
If you haven't set up payments yet, you'll be prompted to add a payment method and purchase credits for your account.
First, install the [Runpod CLI](/runpodctl/overview) on your local machine and configure it with your API key:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
# Install runpodctl
bash <(wget -qO- cli.runpod.io)
# Configure your API key
runpodctl config --apiKey "your-api-key"
```
Then deploy your Pod:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
runpodctl pod create \
--name "quickstart-pod" \
--gpu-id "NVIDIA A40" \
--image "runpod/pytorch:2.1.0-py3.10-cuda11.8.0-devel-ubuntu22.04"
```
The response includes your Pod ID:
```json theme={"theme":{"light":"github-light","dark":"github-dark"}}
{
"id": "uv9wy55tyv30lo",
"name": "quickstart-pod",
"desiredStatus": "RUNNING",
...
}
```
Save it for later:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
export RUNPOD_POD_ID="uv9wy55tyv30lo"
```
## Step 3: Execute code on your Pod
Once your Pod finishes initializing, connect and run some code:
1. On the [Pods page](https://www.console.runpod.io/pods), click your Pod to open the detail pane.
2. Under **HTTP Services**, click **Jupyter Lab** to open a JupyterLab workspace.
3. Under **Notebook**, select **Python 3 (ipykernel)**.
4. Type `print("Hello, world!")` in the first cell and click the play button.
You'll need an [SSH key added to your account](/pods/configuration/use-ssh) for this to work. You can run `runpodctl doctor` to set this up automatically.
Get the SSH command for your Pod:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
runpodctl ssh info $RUNPOD_POD_ID
```
The output includes the SSH command to connect:
```json theme={"theme":{"light":"github-light","dark":"github-dark"}}
{
"sshCommand": "ssh root@194.68.245.207 -p 22100 -i ~/.ssh/id_ed25519"
}
```
Use this command to connect and run code:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
ssh root@194.68.245.207 -p 22100
python3 -c "print('Hello, world!')"
```
Congratulations! You just ran your first line of code on Runpod.
## Step 4: Clean up
To avoid incurring unnecessary charges, clean up your Pod resources.
Terminating a Pod permanently deletes all data that isn't stored in a . Be sure that you've saved any data you might need to access again.
To stop your Pod:
1. Return to the [Pods page](https://www.console.runpod.io/pods) and click your running Pod.
2. Click the **Stop** button (pause icon) to stop your Pod.
3. Click **Stop Pod** in the modal that opens to confirm.
You'll still be charged a small amount for storage on stopped Pods (\$0.20 per GB per month). If you don't need to retain any data on your Pod, you should terminate it completely.
To terminate your Pod:
1. Click the **Terminate** button (trash icon).
2. Click **Terminate Pod** to confirm.
Stop your Pod:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
runpodctl pod stop $RUNPOD_POD_ID
```
You'll still be charged a small amount for storage on stopped Pods (\$0.20 per GB per month). If you don't need to retain any data on your Pod, terminate it completely:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
runpodctl pod delete $RUNPOD_POD_ID
```
## Next steps
Create API keys for programmatic resource management.
Create teams and invite collaborators.
Learn how to select the best Pod for your workload.
Review pricing options for Pods.
Follow step-by-step guides for specific AI/ML use cases.
Start building production-ready applications.
## Need help?
* Join the Runpod community [on Discord](https://discord.com/invite/cUpRmau42V).
* Submit a support request using our [contact page](https://contact.runpod.io/hc/requests/new).
* Reach out to us via [email](mailto:help@runpod.io).
# Agent skills for AI coding tools
Source: https://docs.runpod.io/get-started/agent-skills
Manage GPU workloads on Runpod with coding agents like Claude Code, Codex, and Cursor. Review setup steps and core concepts for building on Runpod.
The Runpod skills plugin teaches your coding agent how to run GPU workloads on Runpod. Once installed, you can ask your agent to create Pods, deploy Serverless endpoints, transfer files, or deploy your own code with Flash, all in natural language. A built-in router sends each request to the right skill, so you don't need to know which tool applies.
It works with [Claude Code](https://code.claude.com/docs), [Codex](https://github.com/openai/codex), [Cursor](https://cursor.com/), [GitHub Copilot](https://github.com/features/copilot), [Windsurf](https://codeium.com/windsurf), [Cline](https://github.com/cline/cline), and [many other AI agents](https://www.skills.sh/).
## Quick start
Install the skills with a single command that works with every agent:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
npx skills add runpod/runpod-plugins-official
```
Next, install the Runpod CLI that the skills rely on (skip this if you already have it):
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
curl -sSL https://cli.runpod.net | bash
# or with Homebrew:
brew install runpod/runpodctl/runpodctl
```
Then authenticate with your [Runpod API key](/get-started/api-keys). The same key works for the CLI, Flash, and the bundled MCP server:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
# Set the key for the current shell (add to ~/.zshrc or ~/.bashrc to persist):
export RUNPOD_API_KEY=
# Or save it permanently to ~/.runpod/config.toml:
runpodctl doctor
```
That's it — restart your agent and [get started](#getting-started).
## Getting started
There are no commands to memorize: describe what you want, and the router picks the right skill (prompting you if something like an API key is still missing). Confirm it's wired up by asking:
* "List my Runpod endpoints"
If your endpoints come back, you're set — see [What you can do](#what-you-can-do) for more examples.
## What's included
The plugin installs a router and a set of focused skills:
| Skill | Description |
| ------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **runpod** | Router and entry point. Reads your task and hands it to the right skill. |
| **runpod-mcp** | Manages Pods, endpoints, templates, network volumes, registries, and billing through the Runpod MCP server. |
| **runpodctl** | Manages the same resources from the [Runpod CLI](/runpodctl/overview), plus Hub deployments, file transfers, SSH keys, and model caching. |
| **flash** | Writes and deploys your own Python code to Runpod Serverless using the [runpod-flash](/flash/overview) SDK. |
| **companion-clis** | Uses supporting CLIs such as Hugging Face, Docker, and the AWS CLI when a task needs them. |
| **runpod-usage** | Provides conceptual knowledge about Pods, Serverless, storage, and GPU selection. |
| **runpod-migrate** | Migrates a codebase from the GraphQL API or REST v1 to REST v2. It inventories which API each call site uses, rewrites the call sites, and verifies the result. |
The plugin also includes a collection of worked, end-to-end reference tasks, such as deploying a Whisper endpoint or running a model on a Pod, that guide your agent through common workflows.
## What you can do
Once installed, you can ask your agent to perform tasks like the following:
| Category | Example prompt |
| ----------------- | ------------------------------------------------- |
| Create resources | "Create a Pod with an RTX 4090" |
| List resources | "List my Pods" or "Show my Serverless endpoints" |
| GPU availability | "What GPUs are available?" |
| Account info | "Show my account balance" |
| Deploy endpoints | "Deploy a Serverless endpoint using my template" |
| Manage Pods | "Stop my Pod" or "SSH into my Pod" |
| Deploy with Flash | "Deploy this function to a remote GPU with Flash" |
| Local development | "Start a local dev server with Flash" |
## Migrate an existing integration
If you already have an integration built on the GraphQL API or REST v1, the `/runpod:migrate` command moves it to REST v2. It inventories which API version each part of your code uses, then rewrites the call sites, flags breaking changes, and verifies the result. It migrates one file at a time, with one commit per file.
Because it edits and commits your code as it goes, run it on a feature branch and review each commit before you merge or deploy.
The command takes an optional scope and path:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
/runpod:migrate [scope: all | rest | graphql] [path]
```
`scope` defaults to `all`, which covers both REST v1 and GraphQL. Use `rest` to target REST v1 only, or `graphql` to target GraphQL only. `path` defaults to the current directory. For example, to migrate only the REST v1 code under `src/`:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
/runpod:migrate rest src/
```
Running the command requires the Runpod skills plugin installed in your coding agent (see [Quick start](#quick-start)). For a manual walkthrough of the API changes, see the [migration guide](/api-reference-v2/migrate-from-v1).
## Native install options
The `npx skills add` command above works everywhere. If you'd rather install the plugin through your agent's native marketplace, use the route for your agent below. Each route installs the same router and skills.
### Claude Code
Add the marketplace, install the plugin, then reload:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
/plugin marketplace add runpod/runpod-plugins-official
/plugin install runpod@runpod
/reload-plugins
```
Installing the plugin also wires up the hosted Runpod MCP server. To authenticate it, run `/mcp`, select **runpod**, and choose **Sign in with Runpod**.
### Codex
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
codex plugin marketplace add https://github.com/runpod/runpod-plugins-official.git
```
Run `codex /plugins`, open the **Runpod** tab, and install (reload if prompted). If the Runpod MCP tools don't appear, add the hosted server manually:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
codex mcp add runpod --transport http https://mcp.getrunpod.io/
```
### Gemini
Gemini can install the plugin natively through the bundled `gemini-extension.json`. Follow your client's extension documentation to add it.
## Connecting the MCP server on other agents
The hosted Runpod MCP server gives your agent structured control-plane tools for managing Pods, endpoints, and other resources. Claude Code sets it up automatically during a native install. On other agents, run the guided installer, which detects your agent and configures the connection:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
npx @runpod/mcp-server@latest add
```
The installer authenticates the MCP server for you. To reuse the API key you already set instead, pass it as a bearer header when you add the server. For example, in Claude Code:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
claude mcp add --transport http runpod -s user https://mcp.getrunpod.io/ \
--header "Authorization: Bearer $RUNPOD_API_KEY"
```
## Update and uninstall
To update the plugin to the latest version:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
# Claude Code:
/plugin marketplace update runpod
# Codex:
codex plugin marketplace upgrade runpod
# skills.sh:
npx skills add runpod/runpod-plugins-official
```
In Claude Code, run `/reload-plugins` after updating.
To uninstall:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
# Claude Code:
/plugin uninstall runpod@runpod
# Codex:
codex plugin marketplace remove runpod
# skills.sh:
npx skills remove runpod
```
If a command reports a name mismatch, list what's installed first with `/plugin marketplace list` (Claude Code), `codex plugin marketplace list` (Codex), or `npx skills list` (skills.sh), then use the name shown.
## Learn more
Source code and full skill definitions.
The skills platform with the full list of compatible AI agents.
Full runpodctl documentation.
Another way to integrate AI tools with Runpod.
# Manage API keys
Source: https://docs.runpod.io/get-started/api-keys
Create, edit, and disable Runpod API keys. Review setup steps, core concepts, and recommended next actions for building on Runpod.
Legacy API keys generated before November 11, 2024 have either Read/Write or Read Only access to GraphQL based on what was set for that key. All legacy keys have full access to AI API. To improve security, generate a new key with **Restricted** permission and select the minimum permission needed for your use case.
## Create an API key
Follow these steps to create a new Runpod API key:
1. In the Runpod console, navigate to the [Settings page](https://www.console.runpod.io/user/settings).
2. Expand the **API Keys** section and select **Create API Key**.
3. Give your key a name and set its permissions (**All**, **Restricted**, or **Read Only**). If you choose **Restricted**, you can customize access for each Runpod API:
* **None**: No access
* **Restricted**: Customize access for each of your endpoints. (Default: None.)
* **Read/Write**: Full access to your endpoints.
* **Read Only**: Read access without write access.
4. Select **Create**, then select your newly-generated key to copy it to your clipboard.
Runpod does not store your API key, so you may wish to save it elsewhere (e.g., in your password manager, or in a GitHub secret). Treat your API key like a password and don't share it with anyone.
## Edit API key permissions
To edit an API key:
1. Navigate to the [Settings page](https://www.console.runpod.io/user/settings).
2. Under **API Keys**, select the pencil icon for the key you wish to update
3. Update the key with your desired permissions, then select **Update**.
## Enable/disable an API key
To enable/disable an API key:
1. Navigate to the [Settings page](https://www.console.runpod.io/user/settings).
2. Under **API Keys**, select the toggle for the API key you wish to enable/disable, then select **Yes** in the confirmation modal.
## Delete an API key
To delete an API key:
1. From the console, select **Settings**.
2. Under **API Keys**, select the trash can icon and select **Revoke Key** in the confirmation modal.
# Concepts
Source: https://docs.runpod.io/get-started/concepts
Key concepts and terminology for understanding Runpod's platform and products. Review setup steps and core concepts for building on Runpod.
## [Runpod console](https://console.runpod.io)
The web interface for managing your compute resources, account, teams, and billing.
## [Serverless](/serverless/overview)
A pay-as-you-go compute solution designed for dynamic autoscaling in production AI/ML apps.
## [Flash](/flash/overview)
A framework for building distributed GPU applications using local Python scripts. Write functions with the `@Endpoint` decorator, and Flash automatically executes them on Runpod's infrastructure.
## [Pod](/pods/overview)
A dedicated GPU or CPU instance for containerized AI/ML workloads, such as training models, running inference, or other compute-intensive tasks.
## [Public Endpoint](/public-endpoints/overview)
An AI model API hosted by Runpod that you can access directly without deploying your own infrastructure.
## [Instant Cluster](/instant-clusters)
A managed compute cluster with high-speed networking for multi-node distributed workloads like training large AI models.
## [Network volume](/storage/network-volumes)
Persistent storage that exists independently of your other compute resources and can be attached to multiple Pods or Serverless endpoints to share data between machines. Available in standard and [high-performance](/storage/high-performance-storage) tiers.
## [S3-compatible API](/storage/s3-api)
A storage interface compatible with Amazon S3 for uploading, downloading, and managing files in your network volumes.
## [Runpod Hub](/hub/overview)
A repository for discovering, deploying, and sharing preconfigured AI projects optimized for Runpod.
## Container
A Docker-based environment that packages your code, dependencies, and runtime into a portable unit that runs consistently across machines.
## Data center
Physical facilities where Runpod's GPU and CPU hardware is located. Your choice of data center can affect latency, available GPU types, and pricing.
## Machine
The physical server hardware within a data center that hosts your workloads. Each machine contains CPUs, GPUs, memory, and storage.
## Training
The foundational phase of AI development, where a model analyzes a massive dataset to learn patterns and relationships.
## Fine-tuning
The process of adapting a pre-trained model to a specific task using a smaller, specialized dataset.
## Inference
The execution phase where a trained model makes predictions on new data. When you prompt a model and it responds, that's inference.
## Serving
The process of deploying and managing a model for inference. When you deploy a model to a Serverless endpoint, that's serving.
# Choose a workflow
Source: https://docs.runpod.io/get-started/connect-to-runpod
Review available methods for accessing and managing Runpod resources. Review setup steps and core concepts for building on Runpod.
Runpod offers multiple ways to access and manage your compute resources. Choose the method that best fits your workflow:
If you are deciding between Pods, Serverless, Public Endpoints, and Clusters, start with the [Runpod product overview](/get-started/products).
## Runpod console
The Runpod console provides an intuitive web interface to manage and s, access Pod terminals, send endpoint requests, monitor resource usage, and view billing and usage history.
[Launch the Runpod console →](https://www.console.runpod.io)
## Connect directly to Pods
You can connect directly to your running Pods and execute code on them using a variety of methods, including a built-in web terminal, an SSH connection from your local machine, a JupyterLab instance, or a remote VSCode/Cursor development environment.
[Learn more about Pod connection options →](/pods/connect-to-a-pod)
## REST API
The Runpod REST API allows you to programmatically manage and control compute resources. Use the API to manage Pod lifecycles and endpoints, monitor resource utilization, and integrate Runpod into your applications.
[Explore the API reference →](/api-reference/docs/GET/openapi-json)
## SDKs
Runpod provides SDKs in Python, JavaScript, Go, and GraphQL to help you integrate Runpod services into your applications.
[Explore the SDKs →](/serverless/sdks)
## Command-line interface (CLI)
The Runpod CLI allows you to manage Pods from your terminal, execute code on Pods, transfer data between Runpod and local systems, and programmatically manage Serverless endpoints.
Every Pod comes pre-installed with the `runpodctl` command and includes a Pod-scoped API key for seamless command-line management.
[Learn more about runpodctl →](/runpodctl/overview)
## Coding agents
Use the [agent onboarding guide](/agent-setup) to install Runpod skills and connect the Runpod MCP server to a supported coding agent.
# Early access features
Source: https://docs.runpod.io/get-started/early-access
Opt in to experimental Runpod console features before they're generally available. Review setup steps and core concepts for building on Runpod.
Early access lets you opt in to experimental Runpod console features before they're generally available. These features are still in active development, so they may change, behave unexpectedly, or be removed before a full release. Enabling them is a good way to try new workflows early and help shape them with your feedback.
Early access settings apply only to your console experience. Turning a feature on or off doesn't affect your running Pods, endpoints, or other resources.
## What is Early access
Early access is a section of the Runpod console where you can enable and disable in-development features one at a time. Each feature is opt-in: it stays off until you turn it on, and you can turn it off again at any time. Because these features are experimental, the available set changes over time as features graduate to general availability or are retired.
## Enable or disable a feature
To manage your Early access features:
1. In the Runpod console, open your account menu and select **Early access**.
2. Find the feature you want to try in the list of available experiments.
3. Toggle the feature on to enable it, or off to disable it.
Your choice takes effect immediately, and you can change it as often as you like.
## Submit feedback
Early access features include an in-page feedback form so you can tell the Runpod team what's working and what isn't. Use it to report bugs, suggest improvements, or share your overall impressions. Your feedback directly informs how each feature evolves and whether it moves toward general availability.
# Use Runpod's MCP servers
Source: https://docs.runpod.io/get-started/mcp-servers
Connect AI tools to Runpod using the Model Context Protocol for infrastructure management and documentation access.
Runpod provides two [Model Context Protocol (MCP)](https://modelcontextprotocol.io/docs/getting-started/intro) servers that connect AI tools and coding agents directly to Runpod:
* **[Runpod API MCP server](#runpod-api-mcp-server):** Manage Pods, endpoints, templates, volumes, and registries through the Runpod REST API. Authenticate with Sign in with Runpod or a [Runpod API key](/get-started/api-keys).
* **[Runpod docs MCP server](#runpod-docs-mcp-server):** Search Runpod documentation for features, code examples, and guides. No authentication required.
## Runpod API MCP server
The Runpod API MCP server gives AI tools access to the [Runpod REST API](/api-reference/overview), letting you create and manage Pods, Serverless endpoints, templates, network volumes, and container registries through natural language.
**Hosted endpoint:** `https://mcp.getrunpod.io/` (Streamable HTTP transport)
**Local package:** `@runpod/mcp-server` (via npx)
**Source code:** [github.com/runpod/runpod-mcp](https://github.com/runpod/runpod-mcp)
**Authentication:** Sign in with Runpod (OAuth) for the hosted server, or a [Runpod API key](/get-started/api-keys) for the local server or as a hosted-server override.
You can connect in two ways: the recommended hosted server, or a local server that runs from an npm package with an API key.
### Hosted server (recommended)
The hosted server runs at `https://mcp.getrunpod.io/` and provides the same capabilities as the local server. OAuth-capable clients run a "Sign in with Runpod" flow on first connect: a browser opens, you log in to the Runpod console and approve the request, and the server obtains a session-scoped API key. Nothing is stored on disk.
The guided installer is the recommended path for supported clients. It detects your installed clients — Claude Code, Claude Desktop, Cursor, Windsurf, and VS Code — and configures each one, handling client-specific config-key differences automatically:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
npx @runpod/mcp-server@latest add
```
To undo these changes, run `npx @runpod/mcp-server@latest remove`.
#### Claude Code (hosted)
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
claude mcp add --transport http runpod -s user https://mcp.getrunpod.io/
```
#### Claude Desktop (hosted)
1. Open **Settings** in Claude Desktop.
2. Navigate to **Connectors** and select **Add custom connector**.
3. Enter `https://mcp.getrunpod.io/` as the URL and click **Add**.
#### VS Code with Copilot (hosted)
1. Open the Command Palette (Ctrl+Shift+P on Windows/Linux or Cmd+Shift+P on macOS).
2. Run **MCP: Add Server** and select **HTTP**.
3. Enter `https://mcp.getrunpod.io/` as the URL and `Runpod` as the name.
4. Select **Global** or **Workspace** and click **Add**.
#### Other clients (hosted)
For any other MCP-compatible client, use the remote URL `https://mcp.getrunpod.io/`:
```json theme={"theme":{"light":"github-light","dark":"github-dark"}}
{
"mcpServers": {
"runpod": {
"url": "https://mcp.getrunpod.io/"
}
}
}
```
Clients differ in the exact key name they use for a remote server URL. For the clients the guided installer supports, it writes the correct configuration automatically; for any other client, consult that client's MCP documentation for how to add a remote HTTP server, using the URL above.
To skip the OAuth flow, pass your API key as a bearer header. For example, in Claude Code:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
claude mcp add --transport http runpod -s user https://mcp.getrunpod.io/ \
--header "Authorization: Bearer $RUNPOD_API_KEY"
```
Clients configured with JSON use a `headers` block instead.
### Local server (API key)
Use this method to run the server locally with an API key stored in your client config — for example, if your client doesn't support the hosted server's sign-in flow, or you prefer to manage the credential yourself.
#### Supported clients
* [Claude Code](#claude-code)
* [Codex CLI](#codex-cli)
* [Cursor](#cursor)
* [VS Code with Copilot](#vs-code-with-copilot)
* [Claude Desktop](#claude-desktop)
* [Windsurf](#windsurf)
* [Cline](#cline)
* [Gemini CLI](#gemini-cli)
#### Claude Code
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
claude mcp add runpod --scope user -e RUNPOD_API_KEY=your_api_key_here -- npx -y @runpod/mcp-server@latest
```
Replace `your_api_key_here` with your Runpod API key. The `--scope user` flag makes the server available across all your projects. Run `/mcp` inside Claude Code to verify the connection.
#### Codex CLI
[Codex CLI](https://github.com/openai/codex) is OpenAI's terminal-based coding agent.
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
codex mcp add runpod --env RUNPOD_API_KEY=your_api_key_here -- npx -y @runpod/mcp-server@latest
```
#### Cursor
Add the following to `.cursor/mcp.json` (project-level) or `~/.cursor/mcp.json` (global). This configuration works with both the Cursor IDE and the [Cursor Agent](https://cursor.com/docs/cli/mcp):
```json theme={"theme":{"light":"github-light","dark":"github-dark"}}
{
"mcpServers": {
"runpod": {
"command": "npx",
"args": ["-y", "@runpod/mcp-server@latest"],
"env": {
"RUNPOD_API_KEY": "your_api_key_here"
}
}
}
}
```
#### VS Code with Copilot
1. Open the Command Palette (Ctrl+Shift+P on Windows/Linux or Cmd+Shift+P on macOS).
2. Run **MCP: Add Server** and select **stdio**.
3. Enter the following details:
* **Name:** `Runpod`
* **Command:** `npx`
* **Arguments:** `-y @runpod/mcp-server@latest`
4. Add environment variable `RUNPOD_API_KEY` with your Runpod API key.
5. Select **Global** or **Workspace** and click **Add**.
#### Claude Desktop
Add the following to your Claude Desktop config file:
* **macOS**: `~/Library/Application Support/Claude/claude_desktop_config.json`
* **Windows**: `%APPDATA%\Claude\claude_desktop_config.json`
```json theme={"theme":{"light":"github-light","dark":"github-dark"}}
{
"mcpServers": {
"runpod": {
"command": "npx",
"args": ["-y", "@runpod/mcp-server@latest"],
"env": {
"RUNPOD_API_KEY": "your_api_key_here"
}
}
}
}
```
Restart Claude Desktop after saving the file.
#### Windsurf
Edit `~/.codeium/windsurf/mcp_config.json` (or open from **Settings** > **Cascade** > **MCP Servers** > **View raw config**):
```json theme={"theme":{"light":"github-light","dark":"github-dark"}}
{
"mcpServers": {
"runpod": {
"command": "npx",
"args": ["-y", "@runpod/mcp-server@latest"],
"env": {
"RUNPOD_API_KEY": "your_api_key_here"
}
}
}
}
```
#### Cline
Open the Cline sidebar in VS Code, click the **MCP Servers** icon, then select **Configure MCP Servers** to edit `cline_mcp_settings.json`:
```json theme={"theme":{"light":"github-light","dark":"github-dark"}}
{
"mcpServers": {
"runpod": {
"command": "npx",
"args": ["-y", "@runpod/mcp-server@latest"],
"env": {
"RUNPOD_API_KEY": "your_api_key_here"
},
"disabled": false
}
}
}
```
#### Gemini CLI
Add to `~/.gemini/settings.json` (global) or `.gemini/settings.json` (project-level):
```json theme={"theme":{"light":"github-light","dark":"github-dark"}}
{
"mcpServers": {
"runpod": {
"command": "npx",
"args": ["-y", "@runpod/mcp-server@latest"],
"env": {
"RUNPOD_API_KEY": "your_api_key_here"
}
}
}
}
```
#### Other clients
For any other MCP-compatible client, use the following connection details:
* **Command:** `npx`
* **Args:** `-y @runpod/mcp-server@latest`
* **Environment:** `RUNPOD_API_KEY=your_api_key_here`
### Usage examples
Once connected, you can interact with your Runpod resources using natural language:
```
List all my Runpod Pods
```
```
Create a new Runpod Pod with the following specifications:
- Name: ml-training-pod
- Image: runpod/pytorch:2.1.0-py3.10-cuda11.8.0-devel-ubuntu22.04
- GPU Type: NVIDIA GeForce RTX 4090
- GPU Count: 1
- Cloud Type: SECURE
```
```
Create a Runpod Serverless endpoint with the following configuration:
- Name: image-generation-endpoint
- Template ID: 30zmvf89kd
- Minimum workers: 0
- Maximum workers: 5
```
```
Stop the Pod named "ml-training-pod"
```
## Runpod docs MCP server
The Runpod docs MCP server provides access to Runpod's documentation knowledge base, making it easier to get answers about features and how to use them.
**Endpoint:** `https://docs.runpod.io/mcp`
**Authentication:** None required
### Claude Code
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
claude mcp add runpod-docs --scope user --transport http https://docs.runpod.io/mcp
```
### Codex CLI
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
codex mcp add runpod-docs --url https://docs.runpod.io/mcp
```
### Cursor
Add to your `.cursor/mcp.json` file:
```json theme={"theme":{"light":"github-light","dark":"github-dark"}}
{
"mcpServers": {
"runpod-docs": {
"url": "https://docs.runpod.io/mcp"
}
}
}
```
### VS Code with Copilot
1. Open the Command Palette (Ctrl+Shift+P on Windows/Linux or Cmd+Shift+P on macOS).
2. Run **MCP: Add Server** and select **HTTP**.
3. Enter `https://docs.runpod.io/mcp` as the URL and `Runpod Docs` as the name.
4. Select **Global** or **Workspace** and click **Add**.
### Claude Desktop
1. Open **Settings** in Claude Desktop.
2. Navigate to **Connectors** and select **Add custom connector**.
3. Enter `https://docs.runpod.io/mcp` as the URL and click **Add**.
### Windsurf
Add to `~/.codeium/windsurf/mcp_config.json`:
```json theme={"theme":{"light":"github-light","dark":"github-dark"}}
{
"mcpServers": {
"runpod-docs": {
"serverUrl": "https://docs.runpod.io/mcp"
}
}
}
```
### Cline
Add to `cline_mcp_settings.json`:
```json theme={"theme":{"light":"github-light","dark":"github-dark"}}
{
"mcpServers": {
"runpod-docs": {
"url": "https://docs.runpod.io/mcp",
"disabled": false
}
}
}
```
### Gemini CLI
Add to `~/.gemini/settings.json`. Note that Gemini CLI uses `httpUrl` instead of `url`:
```json theme={"theme":{"light":"github-light","dark":"github-dark"}}
{
"mcpServers": {
"runpod-docs": {
"httpUrl": "https://docs.runpod.io/mcp"
}
}
}
```
### Other clients
For any other MCP-compatible client, use URL `https://docs.runpod.io/mcp` (HTTP transport).
### Usage examples
With the docs MCP server connected, you can ask questions about Runpod features:
```
Explain the Runpod Serverless model caching feature
```
```
How do I configure environment variables for a Serverless endpoint?
```
```
How does global networking work in Runpod?
```
# Runpod product overview
Source: https://docs.runpod.io/get-started/products
Compare Runpod Pods, Serverless, Clusters, Public Endpoints, and storage options to choose the right infrastructure for your AI workload.
Runpod offers cloud computing resources for AI and machine learning workloads. You can choose from instant GPUs for development, auto-scaling computing, pre-deployed AI models, or multi-node clusters for distributed training.
## [Serverless](/serverless/overview)
Serverless provides pay-per-second computing with automatic scaling for production AI workloads. You only pay for actual compute time when your code runs, with no idle costs, making Serverless ideal for variable workloads and cost-efficient production deployments.
## [Pods](/pods/overview)
give you dedicated GPU or CPU instances for containerized workloads. Pods are billed by the minute and stay available as long as you keep them running, making them perfect for development, training, and workloads that need continuous access.
## [Public Endpoints](/public-endpoints/overview)
s provide instant API access to pre-deployed AI models for image, video, and text generation without any setup. You only pay for what you generate, making it easy to integrate AI into your applications without managing infrastructure. Browse all available models in the [model reference](/public-endpoints/reference).
## [Instant Clusters](/instant-clusters)
s deliver fully managed multi-node compute clusters for large-scale distributed workloads. With high-speed networking between nodes, you can run multi-node training, fine-tune large language models, and handle other tasks that require multiple GPUs working in parallel.
## Choosing the right option
Choose **Serverless** when you need auto-scaling for inference workloads with variable traffic. Pay-per-second billing minimizes costs, and automatic worker management handles unpredictable workloads and API services efficiently.
Choose **Pods** when you need full control for development and experimentation. They work best for training models, iterative development, and custom workflows that require persistent storage and long-running processes.
Choose **Public Endpoints** when you want to quickly integrate AI capabilities without managing infrastructure. They're ideal for prototyping and production applications that use popular AI models with simple pay-per-use pricing.
Choose **Instant Clusters** when your workload requires multiple GPUs across multiple nodes. They provide the infrastructure for training or fine-tuning large language models and other distributed computing tasks with high-speed networking.
You can combine these products as well. For example, use Pods for development and experimentation, Serverless for production inference, and Instant Clusters for large-scale training runs.
# Overview
Source: https://docs.runpod.io/hub/overview
Discover, deploy, and share preconfigured AI repos using the Runpod Hub. Review publishing and management guidance for the Runpod Hub.
The [Runpod Hub](https://console.runpod.io/hub) is a centralized repository for discovering, sharing, and deploying preconfigured AI repos optimized for and infrastructure.
## Why use the Hub?
**For users:**
* **Production-ready solutions**: Vetted, open-source repos with minimal setup required.
* **One-click deployment**: Go from discovery to running services in minutes.
* **Configurable**: Customize parameters without diving into code.
**For creators:**
* **Showcase your work**: Share projects with the AI community.
* **Automated pipeline**: The Hub builds and tests your releases automatically.
* **Earn revenue**: Generate up to 7% of compute revenue when users deploy your repos. See [revenue sharing](/hub/revenue-sharing).
## Public Endpoints
The Hub also offers s for popular AI models. These are ready-to-use APIs with instant access, interactive playgrounds, and usage-based pricing. Browse available models in the [model reference](/public-endpoints/reference).
## Deploy a repo
1. Go to the [Hub](https://www.console.runpod.io/hub) and select a repo.
2. Review hardware requirements and configuration options.
3. Click **Deploy** → **Create Endpoint**.
Your endpoint will be ready for integration within minutes.
1. Go to the [Hub](https://www.console.runpod.io/hub) and select a repo.
2. Click **Deploy**, select **Pod** as the deployment type.
3. Click **Deploy Pod**.
After deployment, find a sample request in the Pod details pane:
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
import requests
response = requests.post(
'https://POD_ID-80.proxy.runpod.net/v2/LOCAL/run',
headers={
'Content-Type': 'application/json',
'Authorization': 'Bearer YOUR_API_KEY'
},
json={'input': {"prompt": "Your prompt"}}
)
```
## Publish your own repo
Publish your GitHub repository on the Hub by preparing a with a and `Dockerfile`.
New to building Serverless workers? Follow the [quickstart guide](/serverless/quickstart).
1. Add configuration files in a `.runpod` directory per the [publishing guide](/hub/publishing-guide).
2. Create a GitHub release.
3. Submit your repository through the Runpod console.
4. After review and approval, your repo appears in the Hub.
Once published, earn revenue from user deployments. Link your GitHub profile to your Runpod account for verified maintainer status. Revenue tiers range from 1% (100-999 compute hours) to 7% (10,000+ hours), paid monthly as Runpod credits.
## How Hub repos work
1. **Repository integration**: Connects with GitHub repos using releases (not commits) for versioning.
2. **Configuration**: Repos use `hub.json` and `tests.json` in a `.runpod` directory to define metadata and test procedures.
3. **Automated pipeline**: The Hub builds and tests repos on submission and monitors for new releases.
4. **Deployment**: Users browse, customize, and deploy with minimal configuration.
# Runpod Hub publishing guide
Source: https://docs.runpod.io/hub/publishing-guide
Publish your repositories to the Runpod Hub. Review setup, publishing, and management guidance for resources in the Runpod Hub.
Learn how to publish your repositories to the [Runpod Hub](https://console.runpod.io/hub), including how to configure your repository with the required `hub.json` and `tests.json` files.
After you publish your repository to the Hub, you can start [earning revenue](/hub/revenue-sharing) from your users' compute usage.
## How to publish your repo
Follow these steps to add your repository to the Hub:
1. Navigate to the [Hub page](https://www.console.runpod.io/hub) in the Runpod console.
2. Under **Add your repo** click **Get Started**.
3. Enter your GitHub repo URL.
4. Follow the UI steps to add your repo to the Hub.
The Hub page will guide you through the following steps:
1. Create your `hub.json` and `tests.json` files.
2. Ensure your repository contains a `handler.py`, `Dockerfile`, and `README.md` file (in either the `.runpod` or root directory).
3. Create a new GitHub release (the Hub indexes releases, not commits).
4. (Optional) Add a Runpod Hub badge into your GitHub `README.md` file, so that users can instantly deploy your repo from GitHub.
After all the necessary files are in place and a release has been created, your repo will be marked "Pending" during building/testing. After testing is complete, the Runpod team will manually review the repo for publication.
## Update a repo
To update your repo on the Hub, just **create a new GitHub release**, and the Hub listing will be automatically indexed and built (usually within an hour).
## Required files
Aside from a working [Serverless implementation](/serverless/overview), every Hub repo requires two additional configuration files:
1. `hub.json` - Defines metadata and deployment settings for your repo.
2. `tests.json` - Specifies how to test your repo.
These files should be placed in the `.runpod` directory at the root of your repository. This directory takes precedence over the root directory, allowing you to override common files like `Dockerfile` and `README.md` specifically for the Hub.
## hub.json reference
The `hub.json` file defines how your listing appears and functions in the Hub.
You can build your `hub.json` from scratch, or use [this template](#hubjson-template) as a starting point.
### General metadata
| Field | Description | Required | Values |
| ------------- | ---------------------------------- | -------- | --------------------------------------------------------------- |
| `title` | Name of your tool | Yes | String |
| `description` | Brief explanation of functionality | Yes | String |
| `type` | Deployment type | Yes | `"serverless"` |
| `category` | Tool category | Yes | `"audio"`, `"embedding"`, `"language"`, `"video"`, or `"image"` |
| `iconUrl` | URL to tool icon | No | Valid URL |
| `config` | Runpod configuration | Yes | Object ([see below](#runpod-configuration)) |
### Runpod configuration
| Field | Description | Required | Values |
| --------------------- | ----------------------------------------------------------------------------------------------------------------------------------------- | --------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `runsOn` | Machine type | Yes | `"GPU"` or `"CPU"` |
| `endpointType` | Endpoint deployment type. When set to `"LB"`, users can deploy your listing as a Serverless endpoint or a Pod directly from the Hub page. | No | `"LB"` |
| `containerDiskInGb` | Container disk space allocation | Yes | Integer (GB) |
| `cpuFlavor` | CPU configuration | Only if `runsOn` is `"CPU"` | Valid CPU flavor string. For a complete list of available CPU flavors, see [CPU types](/references/cpu-types) |
| `gpuCount` | Number of GPUs | Only if `runsOn` is `"GPU"` | Integer |
| `gpuIds` | GPU pool specification | Only if `runsOn` is `"GPU"` | Comma-separated pool IDs (e.g., `"ADA_24"`) with optional GPU ID negations (e.g., `"-NVIDIA GeForce RTX 4090"`). For a list of GPU pools and IDs, see [GPU types](/references/gpu-types#gpu-pools). |
| `allowedCudaVersions` | Supported CUDA versions | No | Array of version strings |
| `env` | Environment variable definitions | No | Object ([see below](#environment-variables)) |
| `presets` | Default environment variable values | No | Object ([see below](#presets)) |
### Environment variables
Environment variables can be defined in several ways:
1. **Static variables**: Direct value assignment. For example:
```json theme={"theme":{"light":"github-light","dark":"github-dark"}}
{
"key": "API_KEY",
"value": "default-api-key-value"
}
```
2. **String inputs**: User-entered text fields. For example:
```json theme={"theme":{"light":"github-light","dark":"github-dark"}}
{
"key": "MODEL_PATH",
"input": {
"name": "Model path",
"type": "string",
"description": "Path to the model weights on disk",
"default": "/models/stable-diffusion-v1-5",
"advanced": false
}
}
```
3. **Hugging Face inputs:** Fields for model selection from Hugging Face Hub. For example:
```json theme={"theme":{"light":"github-light","dark":"github-dark"}}
{
"key": "HF_MODEL",
"input": {
"type": "huggingface",
"name": "Hugging Face Model",
"description": "Model organization/name as listed on Huggingface Hub",
"default": "runwayml/stable-diffusion-v1-5",
"required": true
}
}
```
4. **Option inputs**: User selected option fields. For example:
```json theme={"theme":{"light":"github-light","dark":"github-dark"}}
{
"key": "PRECISION",
"input": {
"name": "Model precision",
"type": "string",
"description": "The numerical precision for model inference",
"options": [
{"label": "Full Precision (FP32)", "value": "fp32"},
{"label": "Half Precision (FP16)", "value": "fp16"},
{"label": "8-bit Quantization", "value": "int8"}
],
"default": "fp16"
}
}
```
5. **Number Inputs**: User-entered numeric fields. For example:
```json theme={"theme":{"light":"github-light","dark":"github-dark"}}
{
"key": "MAX_TOKENS",
"input": {
"name": "Maximum tokens",
"type": "number",
"description": "Maximum number of tokens to generate",
"min": 32,
"max": 4096,
"default": 1024
}
}
```
6. **Boolean Inputs**: User-toggled boolean fields. For example:
```json theme={"theme":{"light":"github-light","dark":"github-dark"}}
{
"key": "USE_FLASH_ATTENTION",
"input": {
"type": "boolean",
"name": "Flash attention",
"description": "Enable Flash Attention for faster inference on supported GPUs",
"default": true,
"trueValue": "true",
"falseValue": "false"
}
}
```
Advanced options will be hidden by default. Hide an option by setting: `"advanced": true` .
### Presets
Presets allow you to define groups of default environment variable values. When a user deploys your repo, they'll be offered a dropdown menu with any preset options you've defined.
Here are some example presets:
```json theme={"theme":{"light":"github-light","dark":"github-dark"}}
"presets": [
{
"name": "Quality Optimized",
"defaults": {
"MODEL_NAME": "runpod-stable-diffusion-xl",
"INFERENCE_MODE": "quality",
"BATCH_SIZE": 1,
"ENABLE_CACHING": false,
"USE_FLASH_ATTENTION": true
}
},
{
"name": "Performance Optimized",
"defaults": {
"MODEL_NAME": "runpod-stable-diffusion-v1-5",
"INFERENCE_MODE": "fast",
"BATCH_SIZE": 8,
"ENABLE_CACHING": true,
"USE_FLASH_ATTENTION": true
}
}
]
```
## hub.json template
Here’s an example `hub.json` file that you can use as a starting point:
```json title="hub.json" theme={"theme":{"light":"github-light","dark":"github-dark"}}
{
"title": "Your Tool's Name",
"description": "A brief explanation of what your tool does",
"type": "serverless",
"category": "language",
"iconUrl": "https://your-icon-url.com/icon.png",
"config": {
"runsOn": "GPU",
"containerDiskInGb": 20,
"gpuCount": 1,
"gpuIds": "RTX A4000,-NVIDIA GeForce RTX 4090",
"allowedCudaVersions": [
"12.8", "12.7", "12.6", "12.5", "12.4",
"12.3", "12.2", "12.1", "12.0"
],
"presets": [
{
"name": "Preset Name",
"defaults": {
"STRING_ENV_VAR": "value1",
"INT_ENV_VAR": 10,
"BOOL_ENV_VAR": true
}
}
],
"env": [
{
"key": "STATIC_ENV_VAR",
"value": "static_value"
},
{
"key": "STRING_ENV_VAR",
"input": {
"name": "User-friendly Name",
"type": "string",
"description": "Description of this input",
"default": "default value",
"advanced": false
}
},
{
"key": "OPTION_ENV_VAR",
"input": {
"name": "Select Option",
"type": "string",
"description": "Choose from available options",
"options": [
{"label": "Option 1", "value": "value1"},
{"label": "Option 2", "value": "value2"}
],
"default": "value1"
}
},
{
"key": "INT_ENV_VAR",
"input": {
"name": "Numeric Value",
"type": "number",
"description": "Enter a number",
"min": 1,
"max": 100,
"default": 50
}
},
{
"key": "BOOL_ENV_VAR",
"input": {
"type": "boolean",
"name": "Enable Feature",
"description": "Toggle this feature on/off",
"default": false,
"trueValue": "enabled",
"falseValue": "disabled"
}
}
]
}
}
```
## tests.json reference
The `tests.json` file defines test cases to validate your tool's functionality. Tests are executed during the build step after [a release has been created](#publish-your-repo-to-the-runpod-hub). A test is considered valid by the Hub if the endpoint returns a [200 response](https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Status/200).
You can build your `tests.json` from scratch, or use [this template](#testsjson-template) as a starting point.
### Test cases
Each test case should include:
| Field | Description | Required | Values |
| --------- | --------------------- | -------- | ---------------------- |
| `name` | Test identifier | Yes | String |
| `input` | Raw job input payload | Yes | Object |
| `timeout` | Max execution time | No | Integer (milliseconds) |
### Test environment configuration
| Field | Description | Required | Values |
| --------------------- | ----------------------------- | ------------------ | ---------------------------------------------------------------- |
| `gpuTypeId` | GPU type for testing | Only for GPU tests | Valid GPU ID (see [GPU types](/references/gpu-types)) |
| `gpuCount` | Number of GPUs | Only for GPU tests | Integer |
| `cpuFlavor` | CPU configuration for testing | Only for CPU tests | Valid CPU flavor string (see [CPU types](/references/cpu-types)) |
| `env` | Test environment variables | No | Array of key-value pairs |
| `allowedCudaVersions` | Supported CUDA versions | No | Array of version strings |
## tests.json template
Here’s an example `tests.json` file that you can use as a starting point:
```json title="tests.json" theme={"theme":{"light":"github-light","dark":"github-dark"}}
{
"tests": [
{
"name": "test_case_name",
"input": {
"param1": "value1",
"param2": "value2"
},
"timeout": 10000
}
],
"config": {
"gpuTypeId": "NVIDIA GeForce RTX 4090",
"gpuCount": 1,
"env": [
{
"key": "TEST_ENV_VAR",
"value": "test_value"
}
],
"allowedCudaVersions": [
"12.7", "12.6", "12.5", "12.4",
"12.3", "12.2", "12.1", "12.0", "11.7"
]
}
}
```
# Revenue sharing
Source: https://docs.runpod.io/hub/revenue-sharing
Earn Runpod credits from your repositories published to the Runpod Hub. Review publishing and management guidance for the Runpod Hub.
Starting in September 2025, every repository [published to the Runpod Hub](/hub/publishing-guide) automatically generates revenue for its maintainers. When users deploy your repositories from the Hub to run on Runpod infrastructure, you earn up to 7% of the compute revenue they generate, paid directly to your Runpod credit balance.
### How it works
Revenue share is calculated based on the total compute hours generated by users running your repositories each month. The percentage you earn depends on your monthly usage tier, with higher tiers offering better revenue rates.
1. Users deploy your repos: When users find and deploy your repositories from the Hub, they generate compute hours on Runpod infrastructure.
2. Usage is tracked: The platform tracks all compute hours generated by deployments of your repositories.
3. Monthly calculations: At the end of each month, your total compute hours are calculated and assigned to a revenue tier.
4. Credits deposited: Your revenue share is automatically deposited into your Runpod account balance.
### Revenue tiers
Revenue tiers reset monthly based on the total compute hours generated by all your published repositories:
* 10,000+ hours: 7% revenue share
* 5,000-9,999 hours: 5% revenue share
* 1,000-4,999 hours: 3% revenue share
* 100-999 hours: 1% revenue share
* Below 100 hours: 0% revenue share
For example, if users generate 2,500 compute hours using your repositories in a month, you would earn \$75 in Runpod credits from those hours (3% of the total compute revenue).
### Requirements
To participate in the revenue sharing program, you must:
1. Link your GitHub profile: Connect your Runpod account to your GitHub profile for verified maintainer status. This ensures you receive credits for repositories you maintain.
2. Have published repositories: Your repositories must be successfully published and approved in the Hub.
3. Maintain active repositories: Keep your repositories up to date with working releases.
### Payment timing
Credits are deposited into your Runpod account balance during the first week of each month. Revenue is calculated based on the previous month's activity.
# Overview
Source: https://docs.runpod.io/instant-clusters
Fully managed compute clusters for multi-node training and AI inference. Review configuration and operations guidance for Runpod Clusters.
Instant Clusters provide fully managed multi-node compute with high-performance networking for distributed workloads. Deploy jobs or large-scale without managing infrastructure, networking, or cluster configuration.
* **Scale beyond single machines**: Train models too large for one GPU, or accelerate training across multiple nodes.
* **High-speed networking**: 1600-3200 Gbps between nodes for efficient gradient synchronization and data movement.
* **Zero configuration**: Pre-configured static IPs, environment variables, and framework support.
* **On-demand**: Deploy in minutes, pay only for what you use.
## Get started
For distributed inference across multiple nodes, follow the [Ray and vLLM deployment guide](/instant-clusters/ray-vllm).
Managed Slurm for HPC workloads.
Multi-node PyTorch for deep learning.
Fine-tune LLMs across multiple GPUs.
Serve models across multiple nodes with Ray and vLLM.
## How it works
Runpod provisions multiple GPU nodes in the same connected with high-speed networking. One node is designated primary (`NODE_RANK=0`), and all nodes receive pre-configured environment variables for distributed communication.
The high-speed interfaces (`ens1`-`ens8`) handle inter-node communication for , , and . The `eth0` interface on the primary node handles external traffic. See the [configuration reference](/instant-clusters/configuration) for environment variables and network details.
## Supported hardware
| GPU | Network speed | Nodes |
| ---- | ------------- | ---------------------- |
| B200 | 3200 Gbps | 2-8 nodes (16-64 GPUs) |
| H200 | 3200 Gbps | 2-8 nodes (16-64 GPUs) |
| H100 | 3200 Gbps | 2-8 nodes (16-64 GPUs) |
| A100 | 1600 Gbps | 2-8 nodes (16-64 GPUs) |
For clusters larger than 8 nodes (up to 512 GPUs), [contact our sales team](https://ecykq.share.hsforms.com/2MZdZATC3Rb62Dgci7knjbA).
## Pricing
Pricing is based on GPU type and number of nodes. See [Instant Clusters pricing](https://www.runpod.io/pricing) for current rates.
Custom pricing is available for enterprise workloads. [Contact our sales team](https://ecykq.share.hsforms.com/2MZdZATC3Rb62Dgci7knjbA) for details.
All accounts have a default spending limit. To deploy larger clusters, contact [help@runpod.io](mailto:help@runpod.io).
# Deploy an Instant Cluster with Axolotl
Source: https://docs.runpod.io/instant-clusters/axolotl
Deploy a Runpod Cluster with Axolotl and configure multi-GPU distributed training for fine-tuning large language models.
This tutorial demonstrates how to use Instant Clusters with [Axolotl](https://axolotl.ai/) to fine-tune large language models (LLMs) across multiple GPUs. By leveraging PyTorch's distributed capabilities and Runpod's high-speed networking infrastructure, you can significantly accelerate your training process compared to single-GPU setups.
Follow the steps below to deploy a cluster and start training your models efficiently.
## Step 1: Deploy an Instant Cluster
1. Open the [Instant Clusters page](https://www.console.runpod.io/cluster) on the Runpod web interface.
2. Click **Create Cluster**.
3. Use the UI to name and configure your Cluster. For this walkthrough, keep **Pod Count** at **2** and select the option for **16x H100 SXM** GPUs. Keep the **Pod Template** at its default setting (Runpod PyTorch).
4. Click **Deploy Cluster**. You should be redirected to the Instant Clusters page after a few seconds.
## Step 2: Set up Axolotl on each Pod
1. Click your cluster to expand the list of Pods.
2. Click on a Pod, for example `CLUSTERNAME-pod-0`, to expand the Pod.
3. Click **Connect**, then click **Web Terminal**.
4. In the terminal that opens, run this command to clone the Axolotl repository into the Pod's main directory:
```sh theme={"theme":{"light":"github-light","dark":"github-dark"}}
git clone https://github.com/axolotl-ai-cloud/axolotl
```
5. Navigate to the `axolotl` directory:
```sh theme={"theme":{"light":"github-light","dark":"github-dark"}}
cd axolotl
```
6. Install the required packages:
```sh theme={"theme":{"light":"github-light","dark":"github-dark"}}
pip3 install -U packaging setuptools wheel ninja
pip3 install --no-build-isolation -e '.[flash-attn,deepspeed]'
```
7. Navigate to the `examples/llama-3` directory:
```sh theme={"theme":{"light":"github-light","dark":"github-dark"}}
cd examples/llama-3
```
Repeat these steps for **each Pod** in your cluster.
## Step 3: Start the training process on each Pod
Run this command in the web terminal of **each Pod**:
```php theme={"theme":{"light":"github-light","dark":"github-dark"}}
torchrun \
--nnodes $NUM_NODES \
--node_rank $NODE_RANK \
--nproc_per_node $NUM_TRAINERS \
--rdzv_id "myjob" \
--rdzv_backend static \
--rdzv_endpoint "$PRIMARY_ADDR:$PRIMARY_PORT" -m axolotl.cli.train lora-1b.yml
```
Currently, the dynamic `c10d` backend is not supported. Please keep the `rdzv_backend` flag set to `static`.
After running the command on the last Pod, you should see output similar to this after the training process is complete:
```csharp theme={"theme":{"light":"github-light","dark":"github-dark"}}
...
{'loss': 1.2569, 'grad_norm': 0.11112671345472336, 'learning_rate': 5.418275829936537e-06, 'epoch': 0.9}
{'loss': 1.2091, 'grad_norm': 0.11100614815950394, 'learning_rate': 3.7731999690749585e-06, 'epoch': 0.92}
{'loss': 1.2216, 'grad_norm': 0.10450132936239243, 'learning_rate': 2.420361737256438e-06, 'epoch': 0.93}
{'loss': 1.223, 'grad_norm': 0.10873789340257645, 'learning_rate': 1.3638696597277679e-06, 'epoch': 0.95}
{'loss': 1.2529, 'grad_norm': 0.1063728854060173, 'learning_rate': 6.069322682050516e-07, 'epoch': 0.96}
{'loss': 1.2304, 'grad_norm': 0.10996092110872269, 'learning_rate': 1.518483566683826e-07, 'epoch': 0.98}
{'loss': 1.2334, 'grad_norm': 0.10642101615667343, 'learning_rate': 0.0, 'epoch': 0.99}
{'train_runtime': 61.7602, 'train_samples_per_second': 795.189, 'train_steps_per_second': 1.085, 'train_loss': 1.255359119443751, 'epoch': 0.99}
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 67/67 [01:00<00:00, 1.11it/s]
[2025-04-01 19:24:22,603] [INFO] [axolotl.train.save_trained_model:211] [PID:1009] [RANK:0] Training completed! Saving pre-trained model to ./outputs/lora-out.
```
Congratulations! You've successfully trained a model using Axolotl on an Instant Cluster. Your fine-tuned model has been saved to the `./outputs/lora-out` directory. You can now use this model for or continue training with different parameters.
## Step 4: Clean up
If you no longer need your cluster, make sure you return to the [Instant Clusters page](https://www.console.runpod.io/cluster) and delete your cluster to avoid incurring extra charges.
You can monitor your cluster usage and spending using the **Billing Explorer** at the bottom of the [Billing page](https://www.console.runpod.io/user/billing) section under the **Cluster** tab.
## Next steps
Now that you've successfully deployed and tested an Axolotl distributed training job on an Instant Cluster, you can:
* Review the [fine-tuning guide](/fine-tune) for a complete Axolotl workflow and deployment path.
* **Fine-tune your own models** by modifying the configuration files in Axolotl to suit your specific requirements.
* **Scale your training** by adjusting the number of Pods in your cluster (and the size of their containers and volumes) to handle larger models or datasets.
* **Try different optimization techniques** such as DeepSpeed, FSDP (Fully Sharded Data Parallel), or other distributed training strategies.
For more information on fine-tuning with Axolotl, refer to the [Axolotl documentation](https://github.com/axolotl-ai-cloud/axolotl).
# Cluster observability
Source: https://docs.runpod.io/instant-clusters/cluster-observability
Monitor Runpod Cluster training workloads in real time with built-in Grafana dashboards for GPU, CPU, memory, network, and storage metrics.
Instant Clusters include a built-in observability engine that streams cluster metrics at 5-second granularity directly to a Grafana dashboard. No configuration is required. Authentication is handled automatically when you launch a cluster.
## Access the dashboard
1. In the left navigation, go to **Resources → Clusters** and select your cluster.
2. Click on a pod row in the cluster list. A side panel opens on the right.
3. In the side panel, select the **Connect** tab.
4. Under **HTTP services**, click **Grafana** (Port 8889).
## Available metrics
Metrics are collected from two sources: NVIDIA DCGM for GPU telemetry and Node Exporter for host-level telemetry.
### GPU metrics (DCGM)
| Category | Metrics |
| ------------- | ----------------------------------------------------------------------------------------------------- |
| Utilization | GPU core, memory copy, encoder, decoder utilization |
| Frame buffer | Used, free, total, reserved VRAM; used percentage; BAR1 used |
| Temperature | GPU temp, memory temp, slowdown threshold, shutdown threshold |
| Power | Usage, management limit, enforced limit, total energy consumption, power violation, thermal violation |
| Clocks | SM clock, memory clock, P-state, clock event reasons |
| PCIe | Link generation, link width, replay counter |
| ECC | Single-bit and double-bit aggregate errors, volatile double-bit errors |
| Row remapping | Correctable, uncorrectable, pending, failure |
| Fan | Fan speed |
### Host metrics
| Category | Metrics |
| ---------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| CPU / scheduling | CPU seconds by mode, frequency, load averages (1/5/15 min), running/blocked processes, forks, context switches, interrupts, scheduler run/wait time |
| Pressure (PSI) | CPU, I/O, and memory waiting and stalled seconds |
| Memory | Total, free, available, buffers, cached, swap, active/inactive pages, anonymous pages, huge pages, slab, kernel stack, page tables, dirty, writeback, committed memory, and more |
| vmstat | Page faults (minor/major), page in/out, swap in/out, OOM kills |
| Disk | Read/write bytes, completed reads/writes, merged ops, read/write time, I/O time, in-progress I/O, discards, flushes |
| Filesystem / FDs | Available, free, and total bytes; inode counts; file descriptor allocation |
| Network (device) | Receive/transmit bytes, packets, drops, errors, FIFO, frame, multicast, compressed; carrier changes, MTU, speed, queue length, interface info |
| Network (stack) | TCP open/established/segment/retransmit counters, UDP, ICMP, IP, socket stats, conntrack, ARP entries, softnet |
| InfiniBand / RoCE | Port state, physical state, rate |
| ethtool | Interface info, received counters |
| Thermal / power (host) | hwmon temperatures (current, min, crit), power average, thermal zone temp, cooling device state |
| EDAC (RAM ECC) | Correctable and uncorrectable memory errors per controller and CSROW |
| Time / clock | Current time, boot time, timezone offset, clocksource info, NTP error, frequency adjustment, TAI offset |
| XFS | Allocation, block-map, directory, inode, vnode, read/write operation counters |
| Entropy / misc | Available entropy bits, pool size, uname/OS/DMI/NVMe info, scrape metadata |
The dashboards shown are a first pass. Coverage and layout will expand over time as new metrics and pre-built views are added.
## What to look for
| Signal | What it indicates |
| ------------------------------------------------------ | ----------------------------------------------------------------------- |
| High VRAM pressure (`DCGM_FI_DEV_FB_USED_PERCENT`) | Model or batch size may be too large for GPU memory |
| Low GPU utilization (`DCGM_FI_DEV_GPU_UTIL`) | Data pipeline bottleneck, worker idle time, or checkpointing pauses |
| GPU temperature near slowdown threshold | Thermal throttling likely; check cooling and power limits |
| PCIe replay counter increasing | Hardware link issue — contact support |
| ECC double-bit errors | Uncorrectable GPU memory errors — contact support |
| NTP clock drift (`node_timex_estimated_error_seconds`) | Nodes out of sync; can affect distributed training coordination |
| High memory pressure (PSI stalled) | Host memory contention; consider reducing CPU-side batch pre-processing |
| Disk IOPS spikes | Dataset loading or checkpoint writes competing for I/O |
# Configuration reference
Source: https://docs.runpod.io/instant-clusters/configuration
Environment variables, network interfaces, and NCCL configuration for Clusters. Review configuration and operations guidance for Runpod Clusters.
This page provides reference information for configuring and troubleshooting Instant Clusters.
## Environment variables
The following environment variables are automatically set on all nodes in an Instant Cluster:
| Environment Variable | Description |
| ------------------------------ | ------------------------------------------------------------------------------------------------ |
| `PRIMARY_ADDR` / `MASTER_ADDR` | The address of the primary node. |
| `PRIMARY_PORT` / `MASTER_PORT` | The port of the primary node. All ports are available. |
| `NODE_ADDR` | The static IP of this node within the cluster network. |
| `NODE_RANK` | The cluster rank (i.e. global rank) assigned to this node. `NODE_RANK = 0` for the primary node. |
| `NUM_NODES` | The number of nodes in the cluster. |
| `NUM_TRAINERS` | The number of GPUs per node. |
| `HOST_NODE_ADDR` | A convenience variable, defined as `PRIMARY_ADDR:PRIMARY_PORT`. |
| `WORLD_SIZE` | The total number of GPUs in the cluster (`NUM_NODES` \* `NUM_TRAINERS`). |
Each node receives a static IP address (`NODE_ADDR`) on the overlay network. When a cluster is deployed, the system designates one node as the primary node by setting the `PRIMARY_ADDR` and `PRIMARY_PORT` environment variables. This simplifies working with multiprocessing libraries that require a primary node.
The following variables are equivalent:
* `MASTER_ADDR` and `PRIMARY_ADDR`
* `MASTER_PORT` and `PRIMARY_PORT`
`MASTER_*` variables are available to provide compatibility with tools that expect these legacy names.
## Network interfaces
Instant Clusters use dedicated high-bandwidth network interfaces for inter-node communication, separate from the management interface used for external traffic.
| Interface | Purpose |
| --------------- | ------------------------------------------------------------------------------------------------------------- |
| `ens1` - `ens8` | High-bandwidth interfaces for inter-node communication. Each interface provides a private network connection. |
| `eth0` | Management interface for external traffic (internet connectivity). |
Instant Clusters support up to 8 high-bandwidth interfaces per node. The `PRIMARY_ADDR` environment variable corresponds to `ens1`, which enables launching and bootstrapping distributed processes.
Do not use `eth0` for inter-node communication. The 172.xxx IP addresses on `eth0` are reserved for internet connectivity only and will result in connection timeouts for distributed workloads.
## NCCL configuration
[NCCL](https://developer.nvidia.com/nccl) (NVIDIA Collective Communications Library) handles GPU-to-GPU communication in distributed training. You must configure NCCL to use the internal network interfaces.
### Required configuration
Set the `NCCL_SOCKET_IFNAME` environment variable to use the internal network:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
export NCCL_SOCKET_IFNAME=ens1
```
By default, `NCCL_SOCKET_IFNAME` uses all available interfaces. However, explicitly setting it to `ens1` ensures NCCL uses the high-bandwidth internal network.
Without this configuration, nodes may attempt to communicate using external IP addresses in the 172.xxx range, which are reserved for internet connectivity only. This will result in connection timeouts and failed distributed training jobs.
### Debugging NCCL
To troubleshoot multi-node communication issues, enable NCCL debug logging:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
export NCCL_DEBUG=INFO
```
This outputs detailed information about NCCL's network discovery and communication attempts, helping identify configuration issues.
## Troubleshooting
### Connection timeouts during distributed training
**Symptom:** Training jobs fail with connection timeout errors between nodes.
**Cause:** NCCL is attempting to communicate over the external network interface (`eth0`) instead of the internal interfaces (`ens1`-`ens8`).
**Solution:** Set the `NCCL_SOCKET_IFNAME` environment variable:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
export NCCL_SOCKET_IFNAME=ens1
```
### Nodes cannot find the primary node
**Symptom:** Worker nodes fail to connect to the primary node during initialization.
**Cause:** The `PRIMARY_ADDR` or `MASTER_ADDR` environment variable is not being used correctly in your distributed training script.
**Solution:** Verify your script uses the `PRIMARY_ADDR` environment variable for the rendezvous address. For PyTorch distributed training:
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
import os
master_addr = os.environ.get("PRIMARY_ADDR", "localhost")
master_port = os.environ.get("PRIMARY_PORT", "29500")
```
### Inconsistent training performance
**Symptom:** Training speed varies significantly between runs or degrades over time.
**Cause:** Network congestion or suboptimal NCCL configuration.
**Solution:**
1. Ensure all nodes are in the same data center (Runpod handles this automatically).
2. Enable NCCL debugging to identify bottlenecks: `export NCCL_DEBUG=INFO`
3. Verify your batch sizes are appropriate for the cluster size to maintain efficient GPU utilization.
# Deploy an Instant Cluster with PyTorch
Source: https://docs.runpod.io/instant-clusters/pytorch
Deploy a Runpod Cluster with PyTorch and configure distributed processing across multiple Pods for multi-GPU workloads.
This tutorial demonstrates how to use [Instant Clusters](/instant-clusters) with to run distributed workloads across multiple GPUs. By leveraging PyTorch's distributed processing capabilities and Runpod's high-speed networking infrastructure, you can significantly accelerate your training process compared to single-GPU setups.
## Requirements
* A Runpod account with sufficient credits for a multi-node cluster
* Basic familiarity with PyTorch and distributed training concepts
## Step 1: Deploy an Instant Cluster
1. Open the [Instant Clusters page](https://www.console.runpod.io/cluster) on the Runpod web interface.
2. Click **Create Cluster**.
3. Use the UI to name and configure your Cluster. For this walkthrough, keep **Pod Count** at **2** and select the option for **16x H100 SXM** GPUs. Keep the **Pod Template** at its default setting (Runpod PyTorch).
4. Click **Deploy Cluster**. You should be redirected to the Instant Clusters page after a few seconds.
## Step 2: Clone the PyTorch demo into each Pod
1. Click your cluster to expand the list of Pods.
2. Click on a Pod, for example `CLUSTERNAME-pod-0`, to expand the Pod.
3. Click **Connect**, then click **Web Terminal**.
4. In the terminal that opens, run this command to clone a basic `main.py` file into the Pod's main directory:
```sh theme={"theme":{"light":"github-light","dark":"github-dark"}}
git clone https://github.com/murat-runpod/torch-demo.git
```
Repeat these steps for **each Pod** in your cluster.
## Step 3: Examine the main.py file
Let's look at the code in our `main.py` file:
```python main.py theme={"theme":{"light":"github-light","dark":"github-dark"}}
import os
import torch
import torch.distributed as dist
def init_distributed():
"""Initialize the distributed training environment"""
# Initialize the process group
dist.init_process_group(backend="nccl")
# Get local rank and global rank
local_rank = int(os.environ["LOCAL_RANK"])
global_rank = dist.get_rank()
world_size = dist.get_world_size()
# Set device for this process
device = torch.device(f"cuda:{local_rank}")
torch.cuda.set_device(device)
return local_rank, global_rank, world_size, device
def cleanup_distributed():
"""Clean up the distributed environment"""
dist.destroy_process_group()
def main():
# Initialize distributed environment
local_rank, global_rank, world_size, device = init_distributed()
print(f"Running on rank {global_rank}/{world_size-1} (local rank: {local_rank}), device: {device}")
# Your code here
# Clean up distributed environment when done
cleanup_distributed()
if __name__ == "__main__":
main()
```
This is the minimal code necessary for initializing a distributed environment. The `main()` function prints the local and global rank for each GPU process (this is also where you can add your own code).
PyTorch assigns `LOCAL_RANK` dynamically to each process. All other environment variables are set automatically by Runpod when you deploy your cluster:
| Variable | Description |
| ------------------------------ | ---------------------------------------------------- |
| `MASTER_ADDR` / `PRIMARY_ADDR` | Address of the primary node for process coordination |
| `MASTER_PORT` / `PRIMARY_PORT` | Port on the primary node |
| `NUM_NODES` | Number of nodes in your cluster |
| `NUM_TRAINERS` | Number of GPUs per node |
| `NODE_RANK` | This node's rank in the cluster (0 for primary) |
| `WORLD_SIZE` | Total GPUs across all nodes |
For a complete list of environment variables, see the [configuration reference](/instant-clusters/configuration#environment-variables).
## Step 4: Start the PyTorch process on each Pod
Run this command in the web terminal of **each Pod** to start the PyTorch process:
```bash launcher.sh theme={"theme":{"light":"github-light","dark":"github-dark"}}
export NCCL_DEBUG=INFO
export NCCL_SOCKET_IFNAME=ens1
torchrun \
--nproc_per_node=$NUM_TRAINERS \
--nnodes=$NUM_NODES \
--node_rank=$NODE_RANK \
--master_addr=$MASTER_ADDR \
--master_port=$MASTER_PORT \
torch-demo/main.py
```
This command launches eight `main.py` processes per node (one per GPU in the Pod).
The `NCCL_SOCKET_IFNAME=ens1` setting tells NCCL to use the high-speed internal network interface (`ens1`) for GPU-to-GPU communication between nodes. Instant Clusters provide up to 8 high-bandwidth interfaces (`ens1`-`ens8`) for inter-node traffic, separate from `eth0` which handles external internet traffic.
The `NCCL_DEBUG=INFO` setting enables detailed logging, which is helpful for troubleshooting communication issues. For more information on NCCL configuration and troubleshooting, see the [configuration reference](/instant-clusters/configuration#nccl-configuration).
The `NCCL_SOCKET_IFNAME=ens1` setting is critical for proper inter-node communication. Without this configuration, nodes may attempt to communicate using external IP addresses (172.xxx range) instead of the internal network interface, leading to connection timeouts and failed distributed training jobs.
### Expected output
After running the command on the last Pod, you should see output similar to this:
```csharp theme={"theme":{"light":"github-light","dark":"github-dark"}}
Running on rank 8/15 (local rank: 0), device: cuda:0
Running on rank 15/15 (local rank: 7), device: cuda:7
Running on rank 9/15 (local rank: 1), device: cuda:1
Running on rank 12/15 (local rank: 4), device: cuda:4
Running on rank 13/15 (local rank: 5), device: cuda:5
Running on rank 11/15 (local rank: 3), device: cuda:3
Running on rank 14/15 (local rank: 6), device: cuda:6
Running on rank 10/15 (local rank: 2), device: cuda:2
```
The first number refers to the global rank of the thread, spanning from `0` to `WORLD_SIZE-1` (`WORLD_SIZE` = the total number of GPUs in the cluster). In our example there are two Pods of eight GPUs, so the global rank spans from 0-15. The second number is the local rank, which defines the order of GPUs within a single Pod (0-7 for this example).
The specific number and order of ranks may be different in your terminal, and the global ranks listed will be different for each Pod.
This diagram illustrates how local and global ranks are distributed across multiple Pods:
## Step 5: Clean up
If you no longer need your cluster, make sure you return to the [Instant Clusters page](https://www.console.runpod.io/cluster) and delete your cluster to avoid incurring extra charges.
You can monitor your cluster usage and spending using the **Billing Explorer** at the bottom of the [Billing page](https://www.console.runpod.io/user/billing) section under the **Cluster** tab.
## Next steps
Now that you've successfully deployed and tested a PyTorch distributed application on an Instant Cluster, you can:
* **Adapt your own PyTorch code** to run on the cluster by modifying the distributed initialization in your scripts.
* **Scale your training** by adjusting the number of Pods in your cluster to handle larger models or datasets.
* **Try different frameworks** like [Axolotl](/instant-clusters/axolotl) for fine-tuning large language models.
* **Optimize performance** by experimenting with different distributed training strategies like Data Parallel (DP), Distributed Data Parallel (DDP), or Fully Sharded Data Parallel (FSDP).
* **Review the [configuration reference](/instant-clusters/configuration)** for detailed information on environment variables, network interfaces, and troubleshooting.
For more information on distributed training with PyTorch, refer to the [PyTorch Distributed Training documentation](https://pytorch.org/tutorials/beginner/dist_overview.html).
# Deploy an Instant Cluster with Ray and vLLM
Source: https://docs.runpod.io/instant-clusters/ray-vllm
Deploy a Runpod Cluster with Ray and vLLM to serve distributed inference across multiple nodes, including setup, networking, and validation.
This tutorial shows how to use Instant Clusters with Ray to run distributed inference on large language models. By combining Ray's cluster management with vLLM's tensor and pipeline parallelism, you can serve models that exceed the memory of a single node — for example, a 70B parameter model across multiple 8×H100 pods.
Ray handles the cluster topology; vLLM uses it to split the model across GPUs both within each node (tensor parallelism) and across nodes (pipeline parallelism).
Distributed inference with Ray and vLLM on Instant Clusters is currently in beta. Join our [Discord](https://discord.gg/runpod) to provide feedback and get support.
***
## Requirements
* A Runpod account with sufficient credits for a multi-node cluster
* Basic familiarity with large language model inference and distributed GPU setups
***
## Step 1: Deploy an Instant Cluster
1. Open the [Instant Clusters page](https://console.runpod.io/instant-clusters).
2. Click **Create Cluster**.
3. Name your cluster and configure it. For this walkthrough, set **Pod Count** to **2** and select **8× H100 SXM GPUs** per pod. Use the **Runpod PyTorch** template as your base image.
Increase `/dev/shm` when configuring your pod. The default (64 MB) is too small for large tensor-parallel workloads. Set it to at least 8 GB. In the pod configuration, add the environment variable `MALLOC_ARENA_MAX=1` and set `--shm-size` to `8g` in your Docker run options.
4. Click **Deploy Cluster**. You are redirected to the Instant Clusters page.
***
## Step 2: Start the Ray head on pod-0
The first pod (`CLUSTERNAME-pod-0`) runs the Ray head node. All other pods connect to it as workers.
1. Click your cluster to expand the pod list.
2. Click **CLUSTERNAME-pod-0**, then click **Connect → Web Terminal**.
3. In the terminal, clone the reference scripts:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
{/* [STO-463] Replace with canonical repo URL once scripts are published. */}
git clone https://github.com/runpod/ray-vllm-cluster.git
```
4. Run the head startup script:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
bash ray-vllm-cluster/head.sh
```
The script sets the correct NIC address and starts Ray:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
# head.sh (excerpt — see full script in the repo)
export RAY_NODE_IP_ADDRESS=$(hostname -I | awk '{print $1}')
export VLLM_HOST_IP=$RAY_NODE_IP_ADDRESS
ray start --head \
--node-ip-address=$RAY_NODE_IP_ADDRESS \
--port=6379 \
--num-gpus=$NUM_TRAINERS
```
`RAY_NODE_IP_ADDRESS` and `VLLM_HOST_IP` must be set to the pod's internal network IP — not `0.0.0.0`. Setting them prevents Ray and vLLM from binding to the wrong interface on multi-NIC pods.
***
## Step 3: Join the worker pods to the cluster
Repeat this for each remaining pod in the cluster (`pod-1`, `pod-2`, …).
1. In the Instant Clusters page, click the next pod and open its **Web Terminal**.
2. Clone the same scripts:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
{/* [STO-463] Replace with canonical repo URL once scripts are published. */}
git clone https://github.com/runpod/ray-vllm-cluster.git
```
3. Run the worker startup script:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
bash ray-vllm-cluster/worker.sh
```
The script waits for the head to be reachable, then joins:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
# worker.sh (excerpt — see full script in the repo)
export RAY_NODE_IP_ADDRESS=$(hostname -I | awk '{print $1}')
export VLLM_HOST_IP=$RAY_NODE_IP_ADDRESS
# Wait for Ray head
until bash -c ">/dev/tcp/$MASTER_ADDR/6379" 2>/dev/null; do
echo "Waiting for Ray head at $MASTER_ADDR:6379..."
sleep 2
done
ray start \
--address=$MASTER_ADDR:6379 \
--node-ip-address=$RAY_NODE_IP_ADDRESS \
--num-gpus=$NUM_TRAINERS
```
`$MASTER_ADDR` is injected automatically by Runpod into all pods in the cluster — it resolves to `pod-0`.
***
## Step 4: Verify the cluster
Run this on `pod-0` to confirm all nodes have joined:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
ray status
```
Expected output for a two-pod cluster with 8 GPUs each:
```
======== Autoscaler status: ... ========
Node status
---------------------------------------------------------------
Healthy:
2 node(s) with resources: {"GPU": 8.0, ...}
```
You can also open the Ray dashboard. The console shows the dashboard port in the **Connect** dialog for `pod-0`.
***
## Step 5: Launch distributed inference with vLLM
Run this on `pod-0` only. vLLM uses the Ray cluster that is already running.
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
bash ray-vllm-cluster/serve.sh
```
The script launches vLLM with tensor parallelism across GPUs within each node and pipeline parallelism across nodes:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
# serve.sh (excerpt — see full script in the repo)
python -m vllm.entrypoints.openai.api_server \
--model meta-llama/Meta-Llama-3-70B \
--tensor-parallel-size $NUM_TRAINERS \ # GPUs per node (e.g., 8)
--pipeline-parallel-size $NUM_NODES \ # number of nodes (e.g., 2)
--host 0.0.0.0 \
--port 8000
```
`--tensor-parallel-size` should equal the number of GPUs per node (`$NUM_TRAINERS`). `--pipeline-parallel-size` should equal the number of nodes (`$NUM_NODES`). Both are injected as environment variables by Runpod.
vLLM connects to the running Ray cluster automatically. It may take several minutes to load model weights across all nodes.
***
## Step 6: Test the endpoint
Once vLLM reports that it is ready, validate from `pod-0`:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
# Check the server is healthy
curl http://localhost:8000/health
# Confirm the model is loaded
curl http://localhost:8000/v1/models
```
Then send a test request:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
curl http://localhost:8000/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "meta-llama/Meta-Llama-3-70B",
"messages": [{"role": "user", "content": "Hello!"}]
}'
```
***
## Step 7: Clean up
When you are done, return to the [Instant Clusters page](https://console.runpod.io/instant-clusters) and delete your cluster. Leaving it running continues to incur charges.
***
## Environment variables reference
Runpod injects these environment variables into every pod in the cluster. The startup scripts rely on them.
| Variable | Description |
| -------------- | ---------------------------------------------------- |
| `MASTER_ADDR` | Hostname of `pod-0`, the Ray head node |
| `MASTER_PORT` | Port for inter-node communication (default: `29500`) |
| `NUM_NODES` | Total number of pods in the cluster |
| `NUM_TRAINERS` | Number of GPUs per pod |
| `NODE_RANK` | Index of this pod (`0` for head, `1+` for workers) |
***
## Common issues
**Ray workers don't join**
Confirm `$MASTER_ADDR` resolves from each worker pod. Run `ping $MASTER_ADDR` in a worker terminal. If it fails, the cluster network may still be initializing — wait 30 seconds and try again.
**vLLM OOM during model load**
Check that `/dev/shm` is large enough (at least 8 GB for 70B models). Also verify that `--tensor-parallel-size` matches the number of GPUs per node — a mismatch causes uneven shard sizes.
**`VLLM_HOST_IP` binding error**
This error occurs when vLLM tries to bind to `0.0.0.0` on a pod with multiple network interfaces. Make sure `VLLM_HOST_IP` is set to the internal IP (`hostname -I | awk '{print $1}'`) before starting the server.
**Stale Ray cluster after restart**
If you restart a pod, Ray does not automatically rejoin the cluster. Rerun `head.sh` on `pod-0` first, then `worker.sh` on all other pods.
***
## Next steps
* Adapt the serve script to load your own model from a [Network Volume](/storage/network-volumes) mount.
* Scale up by increasing the pod count and adjusting `--pipeline-parallel-size` accordingly.
* Try [Axolotl on an Instant Cluster](/instant-clusters/axolotl) for distributed fine-tuning.
* Review the [Instant Cluster configuration reference](/instant-clusters/configuration) for full details on environment variables and networking.
# Scale an Instant Cluster
Source: https://docs.runpod.io/instant-clusters/scale-clusters
Add pods to a running Cluster to increase GPU capacity. Review setup, configuration, scaling, and operations guidance for Runpod Clusters.
You can scale an existing Instant Cluster by adding more pods to it. New pods join the cluster's private network automatically and use the same GPU type, template, and network storage as the existing nodes.
Scaling is not available for clusters running on reserved or contracted hardware. To expand your cluster, contact your account manager or reach out to the sales team.
## Scale a cluster
1. In the left navigation, go to **Resources** → **Clusters**.
2. Select the cluster you want to expand.
3. Click **Scale cluster**.
4. Under **Additional Pods**, set how many pods to add using the **−** and **+** controls.
5. Review the **Summary** — it shows your new total GPU count, VRAM, and updated hourly cost.
6. Click **Scale this cluster** to confirm.
The new pod or pods start provisioning immediately. They appear in the cluster's pod list once they're running.
To downsize your cluster, we recommend terminating the pods on the nodes you want to remove.
## What changes when you scale
Each pod you add contributes the same GPU configuration as your existing cluster nodes. For example, a cluster with 2 pods running 8× H100 SXMs each (16× H100 SXMs / 1280 GB total) becomes a 3-pod cluster with 24× H100 SXMs (1920 GB) after adding one pod.
Cost updates to reflect the new total as soon as the additional pods are running.
## Capacity limits
The number of pods you can add depends on available GPU inventory at the time of scaling. If you need more capacity than the stepper allows, click **Talk to our Sales team** in the scaling dialog to discuss a reserved arrangement.
# Deploy Slurm on a Cluster (unmanaged)
Source: https://docs.runpod.io/instant-clusters/slurm
Deploy and configure an unmanaged Runpod Cluster with Slurm to schedule distributed workloads across multiple GPU nodes.
This guide is for advanced users who want to configure and manage their own Slurm deployment on Instant Clusters. If you're looking for a pre-configured solution, see [Slurm Clusters](/instant-clusters/slurm-clusters).
This tutorial shows how to configure Runpod Instant Clusters with to manage and schedule distributed workloads across multiple nodes.
## Requirements
* You've created a [Runpod account](https://www.console.runpod.io/home) and funded it with sufficient credits.
* You have basic familiarity with Linux command line.
* You're comfortable working with [Pods](/pods/overview) and understand the basics of [Slurm](https://slurm.schedmd.com/).
## Step 1: Deploy an Instant Cluster
1. Open the [Instant Clusters page](https://www.console.runpod.io/cluster) on the Runpod web interface.
2. Click **Create Cluster**.
3. Use the UI to name and configure your cluster. For this walkthrough, keep **Pod Count** at **2** and select the option for **16x H100 SXM** GPUs. Keep the **Pod Template** at its default setting (Runpod PyTorch).
4. Click **Deploy Cluster**. You should be redirected to the Instant Clusters page after a few seconds.
## Step 2: Clone demo and install Slurm on each Pod
To connect to a Pod:
1. On the Instant Clusters page, click on the cluster you created to expand the list of Pods.
2. Click on a Pod, for example `CLUSTERNAME-pod-0`, to expand the Pod.
**On each Pod:**
1. Click **Connect**, then click **Web Terminal**.
2. In the terminal that opens, run this command to clone the Slurm demo files into the Pod's main directory:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
git clone https://github.com/pandyamarut/slurm_example.git && cd slurm_example
```
3. Run this command to install Slurm:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
apt update && apt install -y slurm-wlm slurm-client munge
```
## Step 3: Overview of Slurm demo scripts
The repository contains several essential scripts for setting up Slurm. Let's examine what each script does:
* `create_gres_conf.sh`: Generates the Slurm Generic Resource (GRES) configuration file that defines GPU resources for each node.
* `create_slurm_conf.sh`: Creates the main Slurm configuration file with cluster settings, node definitions, and partition setup.
* `install.sh`: The primary installation script that sets up MUNGE authentication, configures Slurm, and prepares the environment.
* `test_batch.sh`: A sample Slurm job script for testing cluster functionality.
## Step 4: Install Slurm on each Pod
Now run the installation script **on each Pod**, replacing `[MUNGE_SECRET_KEY]` with any secure random string (like a password). The secret key is used for authentication between nodes, and must be identical across all Pods in your cluster.
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
./install.sh "[MUNGE_SECRET_KEY]" node-0 node-1 10.65.0.2 10.65.0.3
```
This script automates the complex process of configuring a two-node Slurm cluster with GPU support, handling everything from system dependencies to authentication and resource configuration. It implements the necessary setup for both the primary (i.e. master/control) and secondary (i.e compute/worker) nodes.
## Step 5: Start Slurm services
If you're not sure which Pod is the primary node, run the command `echo $HOSTNAME` on the web terminal of each Pod and look for `node-0`.
1. **On the primary node** (`node-0`), run both Slurm services:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
slurmctld -D
```
2. Use the web interface to open a second terminal **on the primary node** and run:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
slurmd -D
```
3. **On the secondary node** (`node-1`), run:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
slurmd -D
```
After running these commands, you should see output indicating that the services have started successfully. The `-D` flag keeps the services running in the foreground, so each command needs its own terminal.
## Step 6: Test your Slurm Cluster
1. Run this command **on the primary node** (`node-0`) to check the status of your nodes:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
sinfo
```
You should see output showing both nodes in your cluster, with a state of "idle" if everything is working correctly.
2. Run this command to test GPU availability across both nodes:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
srun --nodes=2 --gres=gpu:1 nvidia-smi -L
```
This command should list all GPUs across both nodes.
## Step 7: Submit the Slurm job script
Run the following command **on the primary node** (`node-0`) to submit the test job script and confirm that your cluster is working properly:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
sbatch test_batch.sh
```
Check the output file created by the test (`test_simple_[JOBID].out`) and look for the hostnames of both nodes. This confirms that the job ran successfully across the cluster.
## Step 8: Clean up
If you no longer need your cluster, make sure you return to the [Instant Clusters page](https://www.console.runpod.io/cluster) and delete your cluster to avoid incurring extra charges.
You can monitor your cluster usage and spending using the **Billing Explorer** at the bottom of the [Billing page](https://www.console.runpod.io/user/billing) section under the **Cluster** tab.
## Next steps
Now that you've successfully deployed and tested a Slurm cluster on Runpod, you can:
* **Adapt your own distributed workloads** to run using Slurm job scripts.
* **Scale your cluster** by adjusting the number of Pods to handle larger models or datasets.
* **Try different frameworks** like [Axolotl](/instant-clusters/axolotl) for fine-tuning large language models.
* **Optimize performance** by experimenting with different distributed training strategies.
# Slurm Clusters
Source: https://docs.runpod.io/instant-clusters/slurm-clusters
Deploy Slurm Clusters on Runpod with zero configuration. Review setup, configuration, scaling, and operations guidance for Runpod Clusters.
Runpod Slurm Clusters provide a managed high-performance computing and scheduling solution that enables you to rapidly create and manage Slurm Clusters with minimal setup.
Slurm Clusters provide:
* **Zero configuration setup:** Slurm and munge are pre-installed and fully configured.
* **Instant provisioning:** Clusters deploy rapidly with minimal setup.
* **Automatic role assignment:** Runpod automatically designates controller and agent nodes.
For more information on working with Slurm, refer to the [Slurm documentation](https://slurm.schedmd.com/documentation.html).
## Deploy a Slurm Cluster
If you prefer to manually configure your Slurm deployment, see [Deploy an Instant Cluster with Slurm (unmanaged)](/instant-clusters/slurm) for a step-by-step guide.
1. Open the [Instant Clusters page](https://console.runpod.io/cluster) on the Runpod console.
2. Click **Create Cluster**.
3. Select **Slurm Cluster** from the cluster type dropdown menu.
4. Configure your cluster specifications:
* **Cluster name**: Enter a descriptive name for your cluster.
* **Pod count**: Choose the number of Pods in your cluster.
* **GPU type**: Select your preferred [GPU type](/references/gpu-types).
* **Region**: Choose your deployment region.
* **Network volume** (optional): Add a [network volume](/storage/network-volumes) for persistent/shared storage. If using a network volume, ensure the region matches your cluster region.
* **Pod template**: Select a [Pod template](/pods/templates/overview) or click **Edit Template** to customize start commands, environment variables, ports, or [container/volume disk](/pods/storage/types) capacity.
Slurm Clusters currently only support official Runpod Pytorch images. If you deploy using a different image, the Slurm process will not start.
5. Click **Deploy Cluster**.
## Connect to a Slurm Cluster
Once deployment completes, you can access your cluster from the [Instant Clusters page](https://console.runpod.io/cluster).
From this page you can select a cluster to view it's component nodes, including a label indicating the **Slurm controller** (primary node) and **Slurm agents** (secondary nodes). Expand a node to view details like availability, GPU/storage utilization, and options for connection and management.
Connect to a node using the **Connect** button, or using any of the [connection methods supported by Pods](/pods/connect-to-a-pod).
## Submit and manage jobs
All standard Slurm commands are available without configuration. For example, you can:
Check cluster status and available resources:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
sinfo
```
Submit a job to the cluster from the Slurm controller node:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
sbatch your-job-script.sh
```
Monitor job queue and status:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
squeue
```
View detailed job information from the Slurm controller node:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
scontrol show job JOB_ID
```
You can find the output of Slurm agents in their individual container logs.
## Advanced configuration
While Runpod's Slurm Clusters work out-of-the-box, you can customize your configuration by connecting to the Slurm controller node using the [web terminal or SSH](/pods/connect-to-a-pod).
Access Slurm configuration files in their standard locations:
* `/etc/slurm/slurm.conf` - Main configuration file.
* `/etc/slurm/gres.conf` - Generic resource configuration.
Modify these files as needed for your specific requirements.
## Troubleshooting
If you encounter issues with your Slurm Cluster, try the following:
* **Jobs stuck in pending state:** Check resource availability with `sinfo` and ensure requested resources are available. If you need more resources, you can add more nodes to your cluster.
* **Authentication errors:** Munge is pre-configured, but if issues arise, verify the munge service is running on all nodes.
For additional support, contact [Runpod support](https://contact.runpod.io/hc/en-us) with your cluster ID and specific error messages.
# Manage Pods with dstack on Runpod
Source: https://docs.runpod.io/integrations/dstack
Use dstack to automate Pod orchestration for AI and ML workloads on Runpod. Review configuration and usage guidance for this Runpod integration.
[dstack](https://dstack.ai/) is an open-source tool that automates Pod orchestration for AI and ML workloads. It lets you define your application and resource requirements in YAML files, then handles provisioning and managing cloud resources on Runpod so you can focus on your application instead of infrastructure.
This guide shows you how to set up dstack with Runpod and deploy [vLLM](https://github.com/vllm-project/vllm) to serve the `meta-llama/Llama-3.1-8B-Instruct` model from Hugging Face.
## Requirements
You'll need:
* [A Runpod account with an API key](/get-started/api-keys).
* Python 3.8 or higher installed on your local machine.
* `pip` (or `pip3` on macOS).
* Basic utilities like `curl`.
These instructions work on macOS, Linux, and Windows.
**Windows users**
Use [WSL (Windows Subsystem for Linux)](https://learn.microsoft.com/en-us/windows/wsl/install) or [Git Bash](https://gitforwindows.org/) to follow along with the Unix-like commands in this guide. Alternatively, use PowerShell or Command Prompt and adjust commands as needed.
## Set up dstack
### Install and configure the server
Open a terminal and create a new directory:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
mkdir runpod-dstack-tutorial
cd runpod-dstack-tutorial
```
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
python3 -m venv .venv
source .venv/bin/activate
```
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
python3 -m venv .venv
source .venv/bin/activate
```
**Command Prompt:**
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
python -m venv .venv
.venv\Scripts\activate
```
**PowerShell:**
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
python -m venv .venv
.venv\Scripts\Activate.ps1
```
Install dstack using `pip`:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
pip3 install -U "dstack[all]"
```
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
pip install -U "dstack[all]"
```
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
pip install -U "dstack[all]"
```
### Configure dstack for Runpod
Create a `config.yml` file in the dstack configuration directory. This file stores your Runpod credentials for all dstack deployments.
* **Create the configuration directory:**
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
mkdir -p ~/.dstack/server
```
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
mkdir -p ~/.dstack/server
```
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
mkdir %USERPROFILE%\.dstack\server
```
* **Navigate to the configuration directory:**
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
cd ~/.dstack/server
```
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
cd ~/.dstack/server
```
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
cd %USERPROFILE%\.dstack\server
```
Create a file named `config.yml` with the following content:
```yml theme={"theme":{"light":"github-light","dark":"github-dark"}}
projects:
- name: main
backends:
- type: runpod
creds:
type: api_key
api_key: YOUR_RUNPOD_API_KEY
```
Replace `YOUR_RUNPOD_API_KEY` with your actual Runpod API key.
Start the dstack server:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
dstack server
```
You'll see output like this:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
[INFO] Applying ~/.dstack/server/config.yml...
[INFO] The admin token is ADMIN-TOKEN
[INFO] The dstack server is running at http://127.0.0.1:3000
```
Save the `ADMIN-TOKEN` to access the dstack web UI.
Open your browser and go to `http://127.0.0.1:3000`. Enter the `ADMIN-TOKEN` from the server output to access the web UI where you can monitor and manage deployments.
## Deploy vLLM
### Configure the deployment
Open a new terminal and navigate to your tutorial directory:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
cd runpod-dstack-tutorial
```
Activate the Python virtual environment:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
source .venv/bin/activate
```
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
source .venv/bin/activate
```
**Command Prompt:**
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
.venv\Scripts\activate
```
**PowerShell:**
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
.venv\Scripts\Activate.ps1
```
Create a new directory for the deployment:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
mkdir task-vllm-llama
cd task-vllm-llama
```
Create a file named `.dstack.yml` with the following content:
```yml theme={"theme":{"light":"github-light","dark":"github-dark"}}
type: task
name: vllm-llama-3.1-8b-instruct
python: "3.10"
env:
- HUGGING_FACE_HUB_TOKEN=YOUR_HUGGING_FACE_HUB_TOKEN
- MODEL_NAME=meta-llama/Llama-3.1-8B-Instruct
- MAX_MODEL_LEN=8192
commands:
- pip install vllm
- vllm serve $MODEL_NAME --port 8000 --max-model-len $MAX_MODEL_LEN
ports:
- 8000
spot_policy: on-demand
resources:
gpu:
name: "RTX4090"
memory: "24GB"
cpu: 16..
```
Replace `YOUR_HUGGING_FACE_HUB_TOKEN` with your [Hugging Face access token](https://huggingface.co/settings/tokens). The model is gated and requires authentication to download.
### Initialize and deploy
In the directory with your `.dstack.yml` file, run:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
dstack init
```
Deploy the task:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
dstack apply
```
You'll see the deployment configuration and available instances. When prompted:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
Submit the run vllm-llama-3.1-8b-instruct? [y/n]:
```
Type `y` and press Enter.
The `ports` configuration forwards the deployed Pod's port to `localhost:8000` on your machine.
dstack will provision the Pod, download the Docker image, install packages, download the model, and start the vLLM server. You'll see progress logs in the terminal.
To view logs at any time, run:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
dstack logs vllm-llama-3.1-8b-instruct
```
Wait until you see logs indicating the server is ready:
```
INFO: Started server process [1]
INFO: Waiting for application startup.
INFO: Application startup complete.
INFO: Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit)
```
### Test the deployment
The vLLM server is now accessible at `http://localhost:8000`.
Test it with `curl`:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
curl -X POST http://localhost:8000/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "meta-llama/Llama-3.1-8B-Instruct",
"messages": [
{"role": "system", "content": "You are Poddy, a helpful assistant."},
{"role": "user", "content": "What is your name?"}
],
"temperature": 0,
"max_tokens": 150
}'
```
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
curl -X POST http://localhost:8000/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "meta-llama/Llama-3.1-8B-Instruct",
"messages": [
{"role": "system", "content": "You are Poddy, a helpful assistant."},
{"role": "user", "content": "What is your name?"}
],
"temperature": 0,
"max_tokens": 150
}'
```
**Command Prompt:**
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
curl -X POST http://localhost:8000/v1/chat/completions -H "Content-Type: application/json" -d "{ \"model\": \"meta-llama/Llama-3.1-8B-Instruct\", \"messages\": [ {\"role\": \"system\", \"content\": \"You are Poddy, a helpful assistant.\"}, {\"role\": \"user\", \"content\": \"What is your name?\"} ], \"temperature\": 0, \"max_tokens\": 150 }"
```
**PowerShell:**
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
curl.exe -Method Post http://localhost:8000/v1/chat/completions `
-Headers @{ "Content-Type" = "application/json" } `
-Body '{ "model": "meta-llama/Llama-3.1-8B-Instruct", "messages": [ {"role": "system", "content": "You are Poddy, a helpful assistant."}, {"role": "user", "content": "What is your name?"} ], "temperature": 0, "max_tokens": 150 }'
```
You'll receive a JSON response:
```json theme={"theme":{"light":"github-light","dark":"github-dark"}}
{
"id": "chat-f0566a5143244d34a0c64c968f03f80c",
"object": "chat.completion",
"created": 1727902323,
"model": "meta-llama/Llama-3.1-8B-Instruct",
"choices": [
{
"index": 0,
"message": {
"role": "assistant",
"content": "My name is Poddy, and I'm here to assist you with any questions or information you may need.",
"tool_calls": []
},
"logprobs": null,
"finish_reason": "stop",
"stop_reason": null
}
],
"usage": {
"prompt_tokens": 49,
"total_tokens": 199,
"completion_tokens": 150
},
"prompt_logprobs": null
}
```
### Clean up
Stop the task when you're done to avoid charges.
Press `Ctrl + C` in the terminal where you ran `dstack apply`. When prompted:
```
Stop the run vllm-llama-3.1-8b-instruct before detaching? [y/n]:
```
Type `y` and press Enter.
The instance will terminate automatically. To ensure immediate termination, run:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
dstack stop vllm-llama-3.1-8b-instruct
```
Verify termination in your Runpod dashboard or the dstack web UI.
## Use volumes for persistent storage
Volumes let you store data between runs and cache models to reduce startup times.
### Create a volume
Create a file named `volume.dstack.yml`:
```yml theme={"theme":{"light":"github-light","dark":"github-dark"}}
type: volume
name: llama31-volume
backend: runpod
region: EUR-IS-1
# Required size
size: 100GB
```
The `region` ties your volume to a specific region, which also ties your Pod to that region.
Apply the volume configuration:
```sh theme={"theme":{"light":"github-light","dark":"github-dark"}}
dstack apply -f volume.dstack.yml
```
### Use the volume in your task
Modify your `.dstack.yml` file to include the volume:
```yml theme={"theme":{"light":"github-light","dark":"github-dark"}}
volumes:
- name: llama31-volume
path: /data
```
This mounts the volume to the `/data` directory inside your container, letting you store models and data persistently. This is useful for large models that take time to download.
For more information, see the [dstack blog on volumes](https://dstack.ai/blog/volumes-on-runpod/).
# Running Runpod on Mods
Source: https://docs.runpod.io/integrations/mods
Use Mods to interact with language models hosted on Runpod from the command line. Review configuration and usage guidance for this Runpod integration.
[Mods](https://github.com/charmbracelet/mods) is a command-line tool for interacting with language models. It integrates with Unix pipelines, letting you send command output directly to LLMs from your terminal.
## How Mods works
Mods reads standard input (or a prompt you provide as an argument), sends it to a language model, and prints the result. You can prefix the input with a prompt from the Mods arguments, and optionally format the output as Markdown. This lets you pipe command output to an LLM for analysis or transformation.
## Get started
Get your API key from the [Runpod Settings](https://www.console.runpod.io/user/settings) page.
Follow the installation instructions for [Mods](https://github.com/charmbracelet/mods) based on your system.
Update the `config_template.yml` file with your Runpod configuration:
```yml theme={"theme":{"light":"github-light","dark":"github-dark"}}
runpod:
# https://docs.runpod.io/serverless/vllm/openai-compatibility
base-url: https://api.runpod.ai/v2/${YOUR_ENDPOINT}/openai/v1
api-key:
api-key-env: RUNPOD_API_KEY
models:
# Add your model name
openchat/openchat-3.5-1210:
aliases: ["openchat"]
max-input-chars: 8192
```
Replace the following values:
* `base-url`: Your specific endpoint URL.
* `api-key-env`: Your Runpod API key.
* `openchat/openchat-3.5-1210`: The model name you want to use.
* `aliases: ["openchat"]`: Your preferred model alias.
* `max-input-chars`: The maximum input characters for your model.
Test your setup by piping command output to Mods:
```sh theme={"theme":{"light":"github-light","dark":"github-dark"}}
ls ~/Downloads | mods --api runpod --model openchat -f "tell my fortune based on these files" | glow
```
This lists files in your `~/Downloads` directory, sends them to Mods using the Runpod API and specified model, and pipes the output to `glow` for formatted display.
# Integrate Runpod with n8n
Source: https://docs.runpod.io/integrations/n8n-integration
Deploy a vLLM worker on Runpod and connect it to n8n to build AI-powered workflows with authenticated inference requests and reusable automations.
Learn how to integrate Runpod Serverless with n8n, a workflow automation tool. By the end of this tutorial, you'll have a vLLM endpoint running on Runpod that you can use within your n8n workflows.
For a faster start, you can point your n8n workflow to an OpenAI-compatible [Public Endpoint](/public-endpoints/overview) instead of deploying a vLLM worker. To do this, skip to [step 2](#step-2%3A-create-an-n8n-workflow) to create your workflow, then in step 3, set the base URL to the Public Endpoint URL for [Qwen3 32B AWQ](/public-endpoints/models/qwen3-32b):
```
https://api.runpod.ai/v2/qwen3-32b-awq/openai/v1
```
## What you'll learn
In this tutorial, you'll learn how to:
* Deploy a vLLM worker serving the `Qwen/qwen3-32b-awq` model.
* Configure your environment variables for n8n compatibility.
* Create a simple n8n workflow to test your integration.
* Connect your workflow to your Runpod endpoint.
## Requirements
Before you begin, you'll need:
* A [Runpod account](/accounts-billing/manage-accounts) (with available credits).
* A [Runpod API key](/get-started/api-keys).
* An [n8n](https://n8n.io/) account.
## Step 1: Deploy a vLLM worker on Runpod
First, you'll deploy a vLLM worker to serve the `Qwen/qwen3-32b-awq` model.
Open the [Runpod console](https://www.console.runpod.io/serverless) and navigate to the Serverless page.
Click **New Endpoint** and select **vLLM** under **Ready-to-Deploy Repos**.
For more details on vLLM deployment options, see [Deploy a vLLM worker](/serverless/vllm/get-started).
In the deployment modal:
* In the **Model** field, enter `Qwen/qwen3-32b-awq`.
* Expand the **Advanced** section to configure your vLLM environment variables:
* Set **Max Model Length** to `8192`.
* Near the bottom of the page, check **Enable Auto Tool Choice**.
* Set **Tool Call Parser** to `Hermes`.
* Set **Reasoning Parser** to `Qwen3`.
* Click **Next**.
* Click **Create Endpoint**.
When using a different model, you may need to adjust your vLLM environment variables to ensure your model returns responses in the format that n8n expects.
Your endpoint will now begin initializing. This may take several minutes while Runpod provisions resources and downloads your model. Wait until the status shows as **Running**.
Once deployed, you'll be taken to the detail page for your endpoint in the Runpod console. You can find your endpoint ID in the **Overview** tab:
You can also find your endpoint ID in the URL of the endpoint detail page. For example, if the URL for your endpoint is `https://console.runpod.io/serverless/user/endpoint/isapbl1e254mbj`, the endpoint ID is `isapbl1e254mbj`.
Copy your endpoint ID to your clipboard. You'll need it to configure your n8n workflow.
## Step 2: Create an n8n workflow
Next, you'll create a simple n8n workflow to test your integration.
Open n8n and navigate to your workspace, then click **Create Workflow**.
Click **Add first step** and select **On chat message**. Click **Test chat** to confirm.
Click the **+** button and search for **AI Agent** and select it. Click **Execute step** to confirm.
Click the **+** button labeled **Chat Model**. Search for **OpenAI Chat Model** and select it.
Click the dropdown under **Credential to connect with** and select **Create new credential**.
## Step 3: Configure the OpenAI Chat Model node
Now you'll configure the n8n OpenAI Chat Model node to use the model running on your Runpod endpoint.
Under **API Key**, add your Runpod API Key. You can create an API key on the settings page of the [Runpod console](https://console.runpod.io/user/settings).
Under **Base URL**, replace the default OpenAI URL with your Runpod endpoint URL:
```
https://api.runpod.ai/v2/ENDPOINT_ID/openai/v1
```
Replace `ENDPOINT_ID` with your vLLM endpoint ID from Step 1.
Click **Save**, and n8n will automatically test your endpoint connection.
It may take a few minutes for your endpoint to scale up a worker to process the request. You can monitor the request using the **Workers** and **Requests** tabs for your vLLM endpoint in the Runpod console.
If you see the message "Connection tested successfully," that means your endpoint is reachable, but it doesn't gaurantee that it's fully compatible with n8n—we'll do that in the next step.
Press escape to return to the OpenAI Chat Model configuration modal.
Under **Model**, select `qwen/qwen3-32b-awq`, then press escape to return to the workflow canvas.
Type a test message into the chat box like "Hello, how are you?" and press enter.
If everything is working correctly, you should see each of the nodes in your workflow go green to indicate successful execution, and a response from the model in the chat box.
Make sure to **Save** your workflow before closing it, as n8n may not save changes to your model node configuration automatically.
## Next steps
Congratulations! You've successfully used Runpod to power an AI agent on n8n.
Now that you've integrated with n8n, you can:
* Build complex AI-powered workflows using your Runpod endpoints.
* Explore other [integration options](/integrations/overview) with Runpod.
* Learn about [OpenAI compatibility](/serverless/vllm/openai-compatibility) features in vLLM.
# Integrate your applications with Runpod
Source: https://docs.runpod.io/integrations/overview
Integrate Runpod compute resources with your applications, external tools, and agentic frameworks. See related Runpod setup and usage details.
You can integrate Runpod compute resources with any system that supports custom endpoint configuration. This guide provides an overview of the many different methods for doing so.
## Integrate with Serverless
[Runpod Serverless endpoints](/serverless/overview) are REST APIs that accept HTTP requests, execute your code, and return the result via HTTP response. Each endpoint provides a unique URL and abstracts away the complexity of managing individual GPUs/CPUs.
To integrate with Serverless:
1. Create a [handler function](/serverless/workers/handler-functions) with the code for your application.
2. [Create a Dockerfile](/serverless/workers/create-dockerfile) to package your handler function and all its dependencies.
3. [Package your worker](/serverless/workers/deploy) into a Docker image and push it to a Docker registry.
4. [Deploy a Serverless endpoint](/serverless/endpoints/overview) using the Runpod console or [REST API](/api-reference/endpoints/POST/endpoints).
5. Start [sending requests](/serverless/endpoints/send-requests) to the endpoint.
For a full walkthrough of how to create and test custom endpoints, try the [Serverless quickstart](/serverless/quickstart).
## Integrate with Pods
[Pods](/pods/overview) are self-contained compute environments, providing instant access to powerful GPU and CPU resources. They're ideal for applications that require a consistent, predictable environment, such as web applications or backend services with a constant workload.
There are two primary methods for integrating a Pod with your application:
### HTTP proxy
For web-based APIs or UIs, Runpod provides an automated [HTTP proxy](/pods/configuration/expose-ports#http-access-via-runpod-proxy). Any port you expose as an HTTP port in your template or Pod configuration is accessible via a unique URL.
The URL follows this format:
```bash title="HTTP proxy URL format" theme={"theme":{"light":"github-light","dark":"github-dark"}}
https://POD_ID-INTERNAL_PORT.proxy.runpod.net
```
For example, if your Pod's ID is `abc123xyz` and you exposed port 8000, your application would send requests to:
```bash title="HTTP proxy URL example" theme={"theme":{"light":"github-light","dark":"github-dark"}}
https://abc123xyz-8000.proxy.runpod.net
```
### Direct TCP
For protocols that require persistent connections or fall outside of standard HTTP, use the [Direct TCP Ports](/pods/configuration/expose-ports#tcp-access-via-public-ip). When you expose a TCP port, Runpod assigns a public IP address and a mapped external port. You can find these details using the [`GET /pods/POD_ID`](/api-reference/pods/GET/pods/podId) endpoint or the [Pod connection menu](/pods/connect-to-a-pod) in the Runpod console.
## Integrate with Public Endpoints
[Public Endpoints](/public-endpoints/overview) are pre-deployed AI models that you can use for without setting up your own Serverless endpoint. They are extremely simple to integrate, requiring zero infrastructure configuration, and you can start using them immediately by pointing your application to the Public Endpoint URL.
The easiest way to get started is to use the [Public Endpoint playground](https://console.runpod.io/hub?tabSelected=public_endpoints) to configure your request parameters, then click the `API` tab to copy the code to your application.
## Integrate external tools with OpenAI-compatible endpoints
Many external tools and agentic frameworks support OpenAI-compatible endpoints with little-to-no configuration required. Integration is usually straightforward: any library or framework that accepts a custom base URL for API calls will work with Runpod without specialized adapters or connectors.
This means you can integrate Runpod with tools like n8n, CrewAI, LangChain, and many others by simply pointing them to your Runpod endpoint URL and providing your Runpod API key for authentication:
```bash title="Base URL format" theme={"theme":{"light":"github-light","dark":"github-dark"}}
https://api.runpod.ai/v2/ENDPOINT_ID/openai/v1
```
You can integrate OpenAI-compatible tools with Runpod using any of the following methods:
### Public Endpoints
[Public Endpoints](/public-endpoints/overview) are pre-deployed AI models that you can use without setting up your own Serverless endpoint. They're vLLM-compatible and return OpenAI-compatible responses, so you can get started quickly without deploying
The following Public Endpoint URLs are available for OpenAI-compatible models:
```bash title="Public Endpoint base URLs" theme={"theme":{"light":"github-light","dark":"github-dark"}}
# Qwen3 32B AWQ base URL
https://api.runpod.ai/v2/qwen3-32b-awq/openai/v1
# IBM Granite-4.0-H-Small base URL
https://api.runpod.ai/v2/granite-4-0-h-small/openai/v1
```
See the [Qwen3 32B](/public-endpoints/models/qwen3-32b) and [IBM Granite 4.0](/public-endpoints/models/granite-4) model reference pages for parameters and pricing.
For more information on the parameters and responses for each model, check the [Public Endpoint model reference](/public-endpoints/reference).
### vLLM endpoints
[Serverless vLLM workers](/serverless/vllm/overview) are optimized for running large language models and return [OpenAI-compatible responses](/serverless/vllm/openai-compatibility), making them ideal for tools that expect OpenAI's API format.
When you deploy a vLLM worker, you can access it using the OpenAI-compatible API at this base URL:
```bash title="vLLM endpoint base URL" theme={"theme":{"light":"github-light","dark":"github-dark"}}
https://api.runpod.ai/v2/ENDPOINT_ID/openai/v1
```
Where `ENDPOINT_ID` is your Serverless endpoint ID.
Not all models support tool calling, which is required to integrate with OpenAI-compatible tools. For more information, see the [vLLM tool calling documentation](https://docs.vllm.ai/en/latest/features/tool_calling.html).
You may need to set a custom chat template or tool call parser to ensure your model returns responses in the format your integration expects. For example, you can configure the `Qwen/qwen3-32b-awq` model for OpenAI compatibility by adding these environment variables to your [vLLM endpoint settings](/serverless/vllm/environment-variables):
```bash title="Qwen3 32B AWQ vLLM environment variables" theme={"theme":{"light":"github-light","dark":"github-dark"}}
ENABLE_AUTO_TOOL_CHOICE=true
REASONING_PARSER=qwen3
TOOL_CALL_PARSER=hermes
```
### SGLang endpoints
[SGLang workers](https://github.com/runpod-workers/worker-sglang) also return OpenAI-compatible responses, offering optimized performance for certain model types and use cases.
### Load balancing endpoints
[Load balancing endpoints](/serverless/load-balancing/overview) let you create custom endpoints where you define your own inputs and outputs. This gives you complete control over the API contract and is ideal when you need custom behavior beyond standard patterns.
## Third-party integrations
For infrastructure management and orchestration, you can also integrate Runpod with:
* [**dstack**](/integrations/dstack): Simplified Pod orchestration for AI/ML workloads.
* [**SkyPilot**](/integrations/skypilot): Multi-cloud execution framework.
* [**Mods**](/integrations/mods): AI-powered command-line tool.
# Running Runpod on SkyPilot
Source: https://docs.runpod.io/integrations/skypilot
Use SkyPilot to run LLMs, AI, and batch jobs on Runpod Pods and Serverless endpoints. Review configuration and usage guidance for this Runpod integration.
[SkyPilot](https://skypilot.readthedocs.io/en/latest/) is a framework for running LLMs, AI, and batch jobs on any cloud.
This integration uses the Runpod CLI infrastructure to spin up on-demand Pods and deploy Serverless endpoints with SkyPilot.
## Get started
Get your API key from the [Runpod Settings](https://www.console.runpod.io/user/settings) page.
Install the latest version of Runpod:
```sh theme={"theme":{"light":"github-light","dark":"github-dark"}}
pip install "runpod>=1.6"
```
Run `runpod config` and paste your API key when prompted.
Install the [SkyPilot Runpod cloud](https://skypilot.readthedocs.io/en/latest/getting-started/installation.html#runpod):
```sh theme={"theme":{"light":"github-light","dark":"github-dark"}}
pip install "skypilot-nightly[runpod]"
```
Run `sky check` to verify your credentials are set up correctly.
## Run a project
Create a new directory for your project:
```sh theme={"theme":{"light":"github-light","dark":"github-dark"}}
mkdir hello-sky
cd hello-sky
```
Create a file named `hello_sky.yaml` with the following content:
```yml theme={"theme":{"light":"github-light","dark":"github-dark"}}
resources:
cloud: runpod
# Working directory (optional) containing the project codebase.
# Its contents are synced to ~/sky_workdir/ on the cluster.
workdir: .
# Setup commands (optional).
# Typical use: pip install -r requirements.txt
# Invoked under the workdir (i.e., can use its files).
setup: |
echo "Running setup."
# Run commands.
# Typical use: make use of resources, such as running training.
# Invoked under the workdir (i.e., can use its files).
run: |
echo "Hello, SkyPilot!"
conda env list
```
Launch your project on the cluster:
```sh theme={"theme":{"light":"github-light","dark":"github-dark"}}
sky launch -c mycluster hello_sky.yaml
```
You'll see the available GPU options. Confirm your GPU type and the cluster will start spinning up.
# Run Transformer Lab experiments on Runpod
Source: https://docs.runpod.io/integrations/transformer-lab
Configure Transformer Lab to run ML training and inference workloads on Runpod GPUs. Review configuration and usage guidance for this Runpod integration.
[Transformer Lab](https://lab.cloud/) is an open-source research environment for AI researchers to train, fine-tune and evaluate models. It allows you to easily scale training from local hardware to cloud GPUs. It provides a unified interface to all your compute resources and simplifies experiment/checkpoint tracking, job scheduling, auto-recovery, centralized artifact storage and more.
This guide shows you how to configure Transformer Lab to run ML workloads on Runpod GPUs.
## Requirements
You'll need:
* [A Runpod account with an API key](/get-started/api-keys).
* macOS, Linux, or Windows with WSL2.
* Python 3.8 or higher.
* Git and curl installed.
**Windows users**
Transformer Lab requires [WSL2 (Windows Subsystem for Linux)](https://learn.microsoft.com/en-us/windows/wsl/install). Install WSL2 first, then follow the Linux instructions within your WSL2 environment.
## Install Transformer Lab
Open a terminal and run:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
curl -fsSL https://lab.cloud/install.sh | bash -s -- multiuser_setup
```
This installs Transformer Lab to `~/.transformerlab`, sets up a conda environment with all dependencies, and enables the Team Settings features needed for cloud provider configuration.
Start the Transformer Lab server:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
cd ~/.transformerlab/src
./run.sh
```
Open your browser to `http://localhost:8338`.
Use the default credentials:
* **Email**: `admin@example.com`
* **Password**: `admin123`
Change these credentials after your first login for security.
## Configure shared storage
For remote task execution, Transformer Lab requires shared storage so your local instance can communicate with remote Pods. Configure one of the following:
* **Amazon S3**: Create an S3 bucket and configure credentials.
* **Google Cloud Storage**: Create a GCS bucket and configure service account.
* **Azure Blob Storage**: Create a storage container and configure credentials.
Refer to the [Transformer Lab documentation](https://lab.cloud/for-teams/advanced-install/cloud-storage/) for detailed shared storage setup instructions.
## Configure Runpod as a compute provider
In the Runpod console, go to [Settings](https://www.console.runpod.io/user/settings) and create an API key with **All** permissions or **Restricted** permissions that include Pod access.
Copy the API key. Runpod doesn't store it, so save it securely.
In Transformer Lab, click your profile icon in the top right corner and select **Team Settings**.
Navigate to **Compute Providers** and click **Add Provider**.
In the modal that opens:
1. Enter a name for your provider (e.g., "runpod-provider"). Remember this name—you'll use it in your task.yaml files.
2. Select **Runpod** as the provider type.
3. In the configuration JSON field, add your [Runpod API key](/get-started/api-keys):
```json theme={"theme":{"light":"github-light","dark":"github-dark"}}
{
"api_key": "YOUR_RUNPOD_API_KEY",
"api_base_url": "https://rest.runpod.io/v1"
}
```
Leave the base URL as is.
Click **Add Compute Provider** to save the provider.
## Run a task on Runpod
Transformer Lab uses task files to define cloud workloads. Tasks specify the resources, setup commands, and run commands for your job.
For detailed information on task configuration, see the [Task YAML Structure](https://lab.cloud/for-teams/running-a-task/task-yaml-structure/) documentation. You can also browse the [Task Gallery](https://lab.cloud/for-teams/running-a-task/quick-start/#4-import-a-task-from-tasks-gallery) for pre-built templates.
### Create a task
In the Transformer Lab sidebar, click **Tasks** to open the task management interface.
Click **New** to add a new task. Select **Start with a blank task template**, then click **Submit**.
In the task editor, paste the following YAML configuration:
```yaml theme={"theme":{"light":"github-light","dark":"github-dark"}}
name: hello-runpod
resources:
compute_provider: runpod-provider
cpus: 4
memory: 16
accelerators: "A40:1"
setup: |
echo "Setting up environment..."
pip install torch
run: |
echo "Hello from Runpod!"
python -c "import torch; print(f'CUDA available: {torch.cuda.is_available()}'); print(f'GPU: {torch.cuda.get_device_name(0)}' if torch.cuda.is_available() else 'GPU: None')"
```
Replace `runpod-provider` with the name you gave your Runpod provider in Team Settings.
This configuration requests a single NVIDIA A40 GPU on Runpod, installs PyTorch, and runs a simple script to verify GPU access.
Click **Queue** to submit the task you just created. Select your Runpod compute provider and click **Submit** to start the job. Transformer Lab provisions a Pod on Runpod, runs your task, and displays the output in the task logs.
### Monitor task progress
Once queued, your task appears in the Tasks list with its current status. Click **Output** to view the task logs.
The output modal has two tabs:
* **Lab SDK Output**: Shows output from scripts that use the `transformerlab` Python package.
* **Machine Logs**: Shows raw stdout/stderr from the Pod. Use this tab to see output from standard `print()` statements.
For the examples in this guide, check the **Machine Logs** tab to see your task output.
### Stop a running task
To stop a task before it completes, click the stop button (square icon). This terminates the Runpod Pod and releases the resources.
You can also verify that no Pods are running by checking the [Runpod console](https://www.console.runpod.io/pods).
## Specify GPU types
Use the `accelerators` field to specify the GPU type:
| Accelerator | Description |
| --------------- | ------------------------------ |
| `"RTX4090:1"` | NVIDIA GeForce RTX 4090 (24GB) |
| `"A40:1"` | NVIDIA A40 (48GB) |
| `"A100:1"` | NVIDIA A100 (40GB or 80GB) |
| `"A100-80GB:1"` | NVIDIA A100 80GB |
| `"H100:1"` | NVIDIA H100 (80GB) |
| `"L40S:1"` | NVIDIA L40S (48GB) |
For multiple GPUs, change the count: `"A100:4"` for 4x A100 GPUs.
## Clean up
When your tasks complete, Transformer Lab automatically releases the Runpod resources. To manually stop a running task, select it from the Tasks list and click **Stop**.
You can also verify that no Pods are running by checking the [Runpod console](https://www.console.runpod.io/pods).
# Welcome to Runpod
Source: https://docs.runpod.io/overview
Explore our guides and examples to deploy your AI/ML application on Runpod. Review setup and usage guidance in the Runpod documentation.
Runpod is a cloud computing platform built for AI, machine learning, and general compute needs. Whether you're or AI models, or deploying cloud-based applications for , Runpod provides scalable, high-performance GPU and CPU resources to power your workloads.
## Access GPUs instantly
Create an account, deploy your first GPU Pod, and use it to execute code.
Create API keys to manage your access to Runpod resources.
Learn about the key concepts and terminology for the Runpod platform.
Run Python functions on remote GPUs directly from your local terminal.
Pay-per-second computing with automatic scaling for production AI/ML apps.
Dedicated GPU or CPU instances for containerized AI/ML workloads.
## Use our model endpoints
Runpod offers [Public Endpoints](/public-endpoints/overview) for instant API access to pre-deployed AI models for image, video, audio, and text generation. No deployment or infrastructure required—just [create an API key](/get-started/api-keys) and make a request:
```python Python theme={"theme":{"light":"github-light","dark":"github-dark"}}
import requests
response = requests.post(
"https://api.runpod.ai/v2/black-forest-labs-flux-1-schnell/runsync",
headers={
"Authorization": "Bearer YOUR_API_KEY", # Replace YOUR_API_KEY with your actual API key
"Content-Type": "application/json"
},
json={
"input": {
"prompt": "A beautiful sunset over mountains", # Customize your prompt
"width": 1024,
"height": 1024
}
}
)
result = response.json()
print(result["output"]["image_url"])
```
```bash cURL theme={"theme":{"light":"github-light","dark":"github-dark"}}
# Replace YOUR_API_KEY with your actual API key
curl -X POST "https://api.runpod.ai/v2/black-forest-labs-flux-1-schnell/runsync" \
-H "Authorization: Bearer YOUR_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"input": {
"prompt": "A beautiful sunset over mountains",
"width": 1024,
"height": 1024
}
}'
```
For a list of available models, see the [model reference](/public-endpoints/reference).
## Guides and examples
Deploy a dedicated GPU with ComfyUI pre-installed and start generating images.
Build a ComfyUI worker and deploy it as a Serverless endpoint.
Use a hybrid local/remote script to generate images with SDXL.
Create a multi-model pipeline for video generation.
Create a REST API with automatic load balancing using Flash.
Deploy a large language model in minutes using vLLM on Serverless.
## High-performance clusters
Create a multi-node [Instant Cluster](/instant-clusters) for fully managed distributed GPU computing with high-speed networking between nodes.
Learn how Instant Clusters work and when to use them.
Set up managed Slurm for HPC workloads.
Run distributed PyTorch training across multiple nodes.
## Support
Submit a support request using our contact page.
Check the status of Runpod services and infrastructure.
Join the Runpod community on Discord.
# Choose a Pod
Source: https://docs.runpod.io/pods/choose-a-pod
Select the right Pod by evaluating your resource requirements. Review setup, configuration, and operations guidance for Runpod Pods.
Selecting the right Pod configuration maximizes performance and cost efficiency for your workload. This guide helps you match your requirements to the right GPU, VRAM, and storage configuration.
## Quick selection by workload
Start by identifying your primary workload type:
| Workload | Recommended GPU tier | Minimum VRAM | Notes |
| ---------------------------------- | ------------------------------------------- | ------------ | ------------------------------------ |
| **LLM inference** (7B–13B params) | Mid-range (RTX 4090, L4, PRO 6000 MIG 24GB) | 24 GB | Sufficient for most quantized models |
| **LLM inference** (30B–70B params) | High-end (A100, H100, PRO 6000 MIG 48GB) | 48–80 GB | May require multi-GPU setup |
| **LLM training/fine-tuning** | High-end (A100, H100) | 40–80 GB | Memory bandwidth critical |
| **Image generation** (SDXL, Flux) | Mid-range (RTX 4090, L4, PRO 6000 MIG 24GB) | 16–24 GB | Benefits from fast inference |
| **Computer vision** | Entry to mid-range | 8–16 GB | Depends on model and batch size |
| **3D rendering** | Mid-range with RT cores | 16–24 GB | RT cores accelerate ray tracing |
| **Data processing** | CPU-focused or entry GPU | 8 GB+ | Prioritize CPU cores and RAM |
For a full list of available GPUs and their specifications, see [GPU types](/references/gpu-types).
RTX PRO 6000 Multi-Instance GPU (MIG) slices are partitioned GPU instances with dedicated memory and compute. Available on Secure Cloud only. All MIG slices use Blackwell architecture. Verify your CUDA version and framework versions support Blackwell before deploying.
## Estimate VRAM requirements
VRAM is the most common bottleneck. Use these guidelines:
**For LLMs:** Allocate approximately **2 GB of VRAM per billion parameters**. For example:
* 7B model → \~14 GB VRAM
* 13B model → \~26 GB VRAM
* 70B model → \~140 GB VRAM (requires multi-GPU)
Quantization reduces VRAM requirements significantly. A 4-bit quantized 70B model can run on \~35 GB VRAM.
**For image models:** SDXL requires \~8 GB minimum, but 16–24 GB provides headroom for larger batch sizes and LoRA training.
### Resource calculators
Use these tools to estimate your specific requirements:
* [Hugging Face Model Memory Calculator](https://huggingface.co/spaces/hf-accelerate/model-memory-usage): Memory estimates for transformer models
* [Can it run LLM?](https://huggingface.co/spaces/Vokturz/can-it-run-llm): Check if hardware can run specific language models
* [VRAM Estimator](https://vram.asmirnov.xyz): GPU memory requirement approximations
## Storage configuration
Choose storage based on your data persistence needs:
| Storage type | Persists after stop? | Persists after delete? | Best for |
| ------------------ | -------------------- | ---------------------- | ------------------------------------------ |
| **Container disk** | No | No | OS, temporary files |
| **Volume disk** | Yes | No | Working files, checkpoints |
| **Network volume** | Yes | Yes | Datasets, model weights, long-term storage |
For data-intensive workloads, ensure sufficient volume disk or network volume capacity for your datasets, model weights, and output files.
## Optimize for cost
1. **Right-size your resources**: Start with the minimum viable configuration, then scale up based on actual usage. Development and testing often need less power than production.
2. **Consider savings plans**: For extended usage, Runpod's [savings plans](/pods/pricing#savings-plans) reduce costs for committed usage.
## Secure Cloud vs Community Cloud
| | Secure Cloud | Community Cloud |
| ------------------ | -------------------------- | ------------------------ |
| **Infrastructure** | T3/T4 data centers | Peer-to-peer providers |
| **Reliability** | High redundancy | Variable |
| **Best for** | Production, sensitive data | Cost-sensitive workloads |
| **Pricing** | Standard | Competitive |
Runpod is no longer accepting new hosts for Community Cloud. Existing Community Cloud resources remain available.
## Next steps
Create your first Pod with your chosen configuration.
Compare all available GPUs and specifications.
Learn more about storage types and pricing.
Learn how to create, start, stop, and delete Pods.
# Connect to a Pod with VSCode or Cursor
Source: https://docs.runpod.io/pods/configuration/connect-to-ide
Set up remote development on your Pod using VSCode or Cursor. Review setup, configuration, and operations guidance for Runpod Pods.
This guide explains how to connect directly to your Pod through VSCode or Cursor using the Remote-SSH extension, allowing you to work within your Pod's volume directories as if the files were stored on your local machine.
## Requirements
Before you begin, you'll need:
* A local development environment with VSCode or Cursor installed.
* [Download VSCode](https://code.visualstudio.com/download).
* [Download Cursor](https://cursor.com/).
* Familiarity with basic command-line operations and SSH.
## Step 1: Install the Remote-SSH extension
To connect to a Pod, you'll need to install the Remote-SSH extension for your IDE:
1. Open VSCode or Cursor and navigate to the **Extensions** view (Ctrl+Shift+X or Cmd+Shift+X).
2. Search for and install the Remote-SSH extension:
* VSCode: **Remote - SSH** by **ms-vscode-remote**.
* Cursor: **Remote-SSH** by **Anysphere**.
## Step 2: Generate an SSH key
Before you can connect to a Pod, you'll need an SSH key that is paired with your Runpod account. If you don't have one, follow these steps:
1. Generate an SSH key using this command on your local terminal:
```sh theme={"theme":{"light":"github-light","dark":"github-dark"}}
ssh-keygen -t ed25519 -f ~/.ssh/id_ed25519 -C "YOUR_EMAIL@DOMAIN.COM"
```
2. To retrieve your public SSH key, run this command:
```sh theme={"theme":{"light":"github-light","dark":"github-dark"}}
cat ~/.ssh/id_ed25519.pub
```
This will output something similar to this:
```sh theme={"theme":{"light":"github-light","dark":"github-dark"}}
ssh-ed25519 AAAAC4NzaC1lZDI1JTE5AAAAIGP+L8hnjIcBqUb8NRrDiC32FuJBvRA0m8jLShzgq6BQ YOUR_EMAIL@DOMAIN.COM
```
3. Copy and paste the output into the **SSH Public Keys** field in your [Runpod user account settings](https://console.runpod.io/user/settings).
To enable SSH access, your public key must be present in the `~/.ssh/authorized_keys` file on your Pod. If you upload your public key to the settings page before your Pod starts, the system will automatically inject it into that file at startup.
If your Pod is already running when you upload the key, the system will not perform this injection. To enable SSH access, you'll need to either terminate/redeploy the Pod, or open a [web terminal](/pods/connect-to-a-pod#web-terminal-connection) on the running Pod and run the following commands:
```sh theme={"theme":{"light":"github-light","dark":"github-dark"}}
export PUBLIC_KEY=""
echo "$PUBLIC_KEY" >> ~/.ssh/authorized_keys
```
## Step 3: Deploy a Pod
Next, deploy the Pod you want to connect to. For detailed deployment instructions, see [Manage Pods -> Create a Pod](/pods/manage-pods#create-a-pod).
To connect with VSCode/Cursor, your Pod template must support SSH over exposed TCP. To determine whether your Pod template supports this, during deployment, after selecting a template, look for a checkbox under **Instance Pricing** labeled **SSH Terminal Access** and make sure it's checked.
All official Runpod Pytorch templates support SSH over exposed TCP.
## Step 4: Configure SSH for your IDE
Next, you'll configure SSH access to your Pod using the Remote-SSH extension. The instructions are different for VSCode and Cursor:
1. From the [Pods](https://console.runpod.io/pods) page, select the Pod you deployed.
2. Select **Connect**, then select the **SSH** tab.
3. Copy the second command, under **SSH over exposed TCP**. It will look similar to this:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
ssh root@123.456.789.80 -p 12345 -i ~/.ssh/id_ed25519
```
If you only see one command under SSH, then SSH over exposed TCP is not supported by your selected Pod template. This means you won't be able to connect to your Pod directly through VSCode/Cursor, but you can still connect using [basic SSH](/pods/connect-to-a-pod#basic-ssh-connection) via the terminal.
4. In VSCode, open the **Command Palette** (Ctrl+Shift+P or Cmd+Shift+P) and choose **Remote-SSH: Connect to Host**, then select **Add New SSH Host**.
5. Enter the copied SSH command from step 3 (`ssh root@***.***.***.** -p ***** -i ~/.ssh/id_ed25519`) and press **Enter**. This will add a new entry to your SSH config file.
1. From the [Pods](https://console.runpod.io/pods) page, select the Pod you deployed.
2. Select **Connect**.
3. Under **Direct TCP Ports**, look for a line similar to:
```
TCP port -> 69.48.159.6:25634 -> :22
```
If you don't see a **Direct TCP Ports** section, then SSH over exposed TCP is not supported by your selected Pod template. This means you won't be able to connect to your Pod directly through VSCode/Cursor, but you can still connect using [basic SSH](/pods/configuration/use-ssh#basic-ssh-connection) via the terminal.
Here's what these values mean:
* `69.48.159.6` is the IP address of your Pod.
* `25634` is the port number for the Pod's SSH service.
Make a note of these values (they will likely be different for your Pod), as you'll need them for the following steps.
4. In Cursor, open the **Command Palette** (Ctrl+Shift+P or Cmd+Shift+P) and choose **Remote-SSH: Connect to Host**, then select **Add New SSH Host**. This opens the SSH config file in Cursor.
5. Add the following to the SSH config file:
```
Host POD_NAME
HostName POD_IP
User root
Port POD_PORT
IdentityFile ~/.ssh/id_ed25519
```
Replace:
* `POD_NAME` with a descriptive name for your Pod. This will be used to identify your Pod in the SSH config file, and does not need to match the name you gave your Pod in the Runpod console.
* `POD_IP` with the IP address of your Pod from step 3.
* `POD_PORT` with the port number of your Pod from step 3.
So, for the example Pod, the SSH config file will look like:
```
Host my-pod
HostName 69.48.159.6
User root
Port 25634
IdentityFile ~/.ssh/id_ed25519
```
If you are using a custom SSH key, replace `~/.ssh/id_ed25519` with the path to your SSH key.
6. Save and close the file.
## Step 5: Connect to your Pod
Now you can connect to your Pod with the Remote-SSH extension.
1. Open the Command Palette (Ctrl+Shift+P or Cmd+Shift+P).
2. Select **Remote-SSH: Connect to Host**.
3. Choose your Pod from the list (either by IP or custom name if you configured one).
4. VSCode/Cursor will open a new window and connect to your Pod.
5. When prompted, select the platform (Linux).
6. Once connected, click **Open Folder** and navigate to your workspace directory (typically `/workspace`).
You should now be connected to your Pod instance, where you can edit files in your volume directories as if they were local.
If you stop and then resume your Pod, the port numbers may change. If so, you'll need to go back to the previous step and update your SSH config file using the new port numbers before reconnecting.
## Working with your Pod
Once connected through Remote-SSH, you can:
* Edit files with full IntelliSense and language support.
* Run and debug applications with access to GPU resources.
* Use integrated terminal for command execution.
* Install extensions that run on the remote host.
* Forward ports to access services locally.
* Commit and push code using integrated Git support.
Here are some important directories to be aware of:
* `/workspace`: Default [persistent storage](/pods/storage/types) directory.
* `/tmp`: Temporary files (cleared when Pod stops).
* `/root`: Home directory for the root user.
## Troubleshooting
If you can't connect to your Pod:
1. Verify your Pod is running and fully initialized.
2. Check that your SSH key is properly configured in Runpod settings.
3. Ensure the Pod has SSH enabled in its template.
If the VSCode/Cursor server fails to install:
1. Check that your Pod has sufficient disk space.
2. Ensure your Pod has internet connectivity.
3. Try manually removing the `.vscode-server` or `.cursor-server` directory and reconnecting:
```sh theme={"theme":{"light":"github-light","dark":"github-dark"}}
rm -rf ~/.vscode-server
```
# Expose ports
Source: https://docs.runpod.io/pods/configuration/expose-ports
Learn how to make your Pod services accessible from the internet using HTTP proxy and TCP port forwarding
Runpod provides flexible options for exposing your Pod services to the internet. This guide explains how to configure port exposure for different use cases and requirements.
## Understanding port mapping
When exposing services from your Pod, it's important to understand that the publicly accessible port usually differs from your internal service port. This mapping ensures security and allows multiple Pods to coexist on the same infrastructure.
For example, if you run a web API inside your Pod on port 4000 like this:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
uvicorn main:app --host 0.0.0.0 --port 4000
```
The external port users connect to will be different, depending on your chosen exposure method.
## HTTP access via Runpod proxy
Runpod's HTTP proxy provides the easiest way to expose web services from your Pod. This method works well for REST APIs, web applications, and any HTTP-based service.
### Configure external HTTP ports
To configure HTTP ports during Pod deployment, click **Edit Template** and add a comma-separated list of ports to the **Expose HTTP Ports (Max 10)** field.
To configure HTTP ports for an existing Pod, navigate to the [Pod page](https://www.console.runpod.io/pods), expand your Pod, click the hamburger menu on the bottom-left, select **Edit Pod**, then add your port(s) to the **Expose HTTP Ports (Max 10)** field.
You can also configure HTTP ports for a Pod template in the [My Templates](https://www.console.runpod.io/user/templates) section of the console.
### Access your service
Once your Pod is running and your service is active, access it using the proxy URL format:
```
https://[POD_ID]-[INTERNAL_PORT].proxy.runpod.net
```
Replace `[POD_ID]` with your Pod's unique identifier and `[INTERNAL_PORT]` with your service's internal port. For example:
* Pod ID: `abc123xyz`
* Internal port: `4000`
* Access URL: `https://abc123xyz-4000.proxy.runpod.net`
A Pod that's listed as **Running** in the console (with a green dot in the Pod UI) may not be ready to use. The best way to check if your Pod is ready is by checking the **Telemetry** tab in the Pod details page in the Runpod console.
If a Pod is receiving telemetry, it should be ready to use, but individual services (JupyterLab, HTTP services, etc.) may take a few minutes to start up.
### Proxy limitations and behavior
The HTTP proxy route includes several intermediaries that affect connection behavior:
```
User → Cloudflare → Runpod Load Balancer → Your Pod
```
This architecture introduces important limitations:
* **100-second timeout**: Cloudflare enforces a maximum connection time of 100 seconds. If your service doesn't respond within this time, the connection closes with a `524` error.
* **HTTPS only**: All connections are secured with HTTPS, even if your internal service uses HTTP.
* **Public accessibility**: Your service becomes publicly accessible. While the Pod ID provides some obscurity, implement proper authentication in your application.
Design your application with these constraints in mind. For long-running operations, consider:
* Implementing progress endpoints that return status updates.
* Using background job queues with status polling.
* Breaking large operations into smaller chunks.
* Returning immediate responses with job IDs for later retrieval.
## TCP access via public IP
Pods do not support UDP connections. If your application relies on UDP, you'll need to modify your application to use TCP-based communication instead.
For services requiring direct TCP connections, lower latency, or protocols other than HTTP, use TCP port exposure with public IP addresses.
### Configure TCP ports
In your Pod or template configuration, follow the same steps as for [HTTP ports](#configure-external-http-ports), but add ports to the **Expose TCP Ports** field. This enables direct TCP forwarding with a public IP address.
### Find your connection details
After your Pod starts, check the **Connect** menu to find your assigned public IP and external port mapping under **Direct TCP Ports**. For example:
```
TCP port 213.173.109.39:13007 -> :22
```
Public IP addresses may change for Community Cloud Pods if your Pod is migrated or restarted, but they should remain stable for Secure Cloud Pods.
External port mappings change whenever your Pod resets.
## Symmetrical port mapping
Some applications require the external port to match the internal port. Runpod supports this through a special configuration syntax.
### Requesting symmetrical ports
To request symmetrical mapping, specify port numbers above 70000 in your TCP configuration. These aren't valid port numbers, but signal Runpod to allocate matching internal and external ports.
After Pod creation, check the **Connect** menu to see which symmetrical ports were assigned under **Direct TCP Ports**.
### Accessing port mappings programmatically
Your application can discover assigned ports through environment variables. For example, if you specify `70000` and `70001` in your Pod configuration, you could use the following commands to retrieve the assigned ports:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
echo $RUNPOD_TCP_PORT_70000
echo $RUNPOD_TCP_PORT_70001
```
You can use these environment variables in your application configuration to automatically adapt to assigned ports:
**Python example:**
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
import os
# Get the assigned port or use a default
port = os.environ.get('RUNPOD_TCP_PORT_70000', '8000')
app.run(host='0.0.0.0', port=int(port))
```
**Configuration file example:**
```yaml theme={"theme":{"light":"github-light","dark":"github-dark"}}
server:
host: 0.0.0.0
port: ${RUNPOD_TCP_PORT_70000}
```
## Best practices
When exposing ports from your Pods, follow these guidelines for security and reliability:
### Security considerations
* **Implement authentication**: Both HTTP proxy and TCP access make your services publicly accessible. Always implement proper authentication and authorization in your applications.
* **Use HTTPS for sensitive data**: While the proxy automatically provides HTTPS, TCP connections do not. Implement TLS in your application when handling sensitive data over TCP.
* **Validate input**: Public endpoints are targets for malicious traffic. Implement robust input validation and rate limiting.
### Performance optimization
* **Choose the right method**: Use HTTP proxy for web services and TCP for everything else. The proxy adds latency but provides automatic HTTPS and load balancing.
* **Handle timeouts gracefully**: Design your application to work within the 100-second proxy timeout or use TCP for long-running connections.
* **Monitor your services**: Implement health checks and monitoring to ensure your exposed services remain accessible.
### Configuration tips
* **Document your ports**: Maintain clear documentation of which services run on which ports, especially in complex deployments.
* **Use templates**: Define port configurations in templates for consistent deployments across multiple Pods.
* **Test thoroughly**: Verify your port configurations work correctly before deploying production workloads.
## Common use cases
Different types of applications benefit from different exposure methods:
* **Web APIs and REST services**: Use HTTP proxy for automatic HTTPS and simple configuration.
* **WebSocket applications**: TCP exposure often works better for persistent connections that might exceed timeout limits.
* **Database connections**: Use TCP with proper security measures. Consider using Runpod's global networking for internal-only databases.
* **Development environments**: HTTP proxy works well for web-based IDEs and development servers.
## Troubleshooting
Try these fixes if you're having issues with port exposure:
* **Service not accessible via proxy**: Ensure your service binds to `0.0.0.0` (all interfaces) not just `localhost` or `127.0.0.1`.
* **524 timeout errors**: If your service takes longer than 100 seconds to respond, consider using TCP or restructuring your application for faster responses.
* **Connection refused**: Verify your service is running and listening on the correct port inside the Pod.
* **Port already in use**: Check that no other services in your Pod are using the same port.
* **Unstable connections**: For Community Cloud Pods, implement reconnection logic to handle IP address changes.
## Next steps
Once you've exposed your ports, consider:
* Setting up [SSH access](/pods/configuration/use-ssh) for secure Pod administration.
* Implementing [global networking](/pods/networking) for secure Pod-to-Pod communication.
* Configuring health checks and monitoring for your exposed services.
# Connect to a Pod with SSH
Source: https://docs.runpod.io/pods/configuration/use-ssh
Manage Pods from your local machine using SSH. Review setup, configuration, storage, networking, and operations guidance for Runpod Pods.
SSH provides secure, reliable access to your Pod for long-running processes and full shell capabilities.
## Connection methods
| Method | SCP/SFTP support | Setup | Best for |
| --------------------------------------------------------------------------- | ---------------- | ------------------- | --------------------------------- |
| [**Basic SSH**](#basic-ssh-with-key-authentication) | No | SSH key | Quick access, most Pods |
| [**Full SSH (public IP)**](#full-ssh-via-public-ip-with-key-authentication) | Yes | SSH key + public IP | File transfers, full SSH features |
| [**Password-based**](#password-based-ssh) | Yes | Script + public IP | Quick setup, temporary access |
SSH key authentication is recommended for security and convenience.
## Generate an SSH key and add it to your Runpod account
Run this command on your local terminal to generate an SSH key, replacing `YOUR_EMAIL@DOMAIN.COM` with your actual email:
```sh theme={"theme":{"light":"github-light","dark":"github-dark"}}
ssh-keygen -t ed25519 -C "YOUR_EMAIL@DOMAIN.COM"
```
This saves a public/private key pair on your local machine to `~/.ssh/id_ed25519.pub` and `~/.ssh/id_ed25519` respectively.
If you are using Command Prompt on Windows instead of the Linux terminal or WSL, your public and private key pair will be saved to `C:\Users\YOUR_USER_ACCOUNT\.ssh\id_ed25519.pub` and `C:\Users\YOUR_USER_ACCOUNT\.ssh\id_ed25519`, respectively.
1. Run `cat ~/.ssh/id_ed25519.pub` to display your public key.
2. Copy the output (starts with `ssh-ed25519`).
3. Paste it into the **SSH Public Keys** field in your [Runpod account settings](https://www.console.runpod.io/user/settings).
If you need to add multiple SSH keys, make sure each key is on its own line.
Use [runpodctl](/runpodctl/overview) to add your key directly:
```sh theme={"theme":{"light":"github-light","dark":"github-dark"}}
runpodctl ssh add-key --key-file ~/.ssh/id_ed25519.pub
```
Verify it was added:
```sh theme={"theme":{"light":"github-light","dark":"github-dark"}}
runpodctl ssh list-keys
```
### Override your public key for a specific Pod
Runpod will attempt to automatically inject the public SSH keys added in your account settings for authentication when connecting using the [basic terminal method](#basic-ssh-with-key-authentication). If you prefer to use a different public key for a specific Pod, you can override the default by setting the `SSH_PUBLIC_KEY` [environment variable](/pods/templates/environment-variables) for that Pod.
## Basic SSH with key authentication
All Pods provide a basic SSH connection that is proxied through Runpod's systems. This method does not support commands like SCP (Secure Copy Protocol) or SFTP (SSH File Transfer Protocol).
Ensure you have an [SSH key pair](#generate-an-ssh-key-and-add-it-to-your-runpod-account) generated on your local machine and added to your Runpod account.
Navigate to the [Pods page](https://console.runpod.io/pods) in the Runpod console. Select the Pod you want to connect to from the list to open its connection options.
In the Pod's **Connect** tab, copy the command listed under **SSH**. It should look something like this:
```sh theme={"theme":{"light":"github-light","dark":"github-dark"}}
ssh 8y5rumuyb50m78-6441103b@ssh.runpod.io -i ~/.ssh/id_ed25519
```
If you saved your key to a custom location, use that specific path after the `-i` flag instead.
Run the copied command in your local terminal to connect to your Pod.
## Full SSH via public IP with key authentication
For full SSH capabilities, including SCP and SFTP for file transfers, you need to rent an instance that supports a public IP address and ensure an SSH daemon is running within your Pod.
If you're using a Runpod official template such as Runpod PyTorch or Stable Diffusion, full SSH access is already configured for you, so no additional setup is required.
However, if you're using a custom template, ensure that TCP port 22 is exposed and that the SSH daemon is running inside your Pod. If it isn't, add the Docker command below to your template. Or, if you already have a custom start command, replace `sleep infinity` at the end of your command with this one:
```sh theme={"theme":{"light":"github-light","dark":"github-dark"}}
bash -c 'apt update; \
DEBIAN_FRONTEND=noninteractive apt-get install openssh-server -y; \
mkdir -p ~/.ssh; \
cd ~/.ssh; \
chmod 700 ~/.ssh; \
echo "$PUBLIC_KEY" >> authorized_keys; \
chmod 700 authorized_keys; \
service ssh start; \
sleep infinity'
```
Once you're sure that the SSH daemon is running, you can connect to your Pod by following these steps:
Ensure you have an [SSH key pair](#generate-an-ssh-key-and-add-it-to-your-runpod-account) generated on your local machine and added to your Runpod account.
An SSH daemon must be started in your Pod. Runpod official templates, such as "Runpod PyTorch", often have this pre-configured. If you're using a custom template, ensure TCP port 22 is exposed and the SSH daemon is started. Refer to the [Use SSH guide](/pods/configuration/use-ssh) for commands to include in your custom Docker template.
Navigate to the [Pods page](https://console.runpod.io/pods) in the Runpod console. Select the Pod you want to connect to from the list to open its connection options.
In the Pod's **Connect** tab, copy the command listed under **SSH over exposed TCP**. It should look something like this:
```sh theme={"theme":{"light":"github-light","dark":"github-dark"}}
ssh root@213.173.108.12 -p 17445 -i ~/.ssh/id_ed25519
```
If you saved your key to a custom location, use that specific path after the `-i` flag instead.
Run the copied command in your local terminal to connect to your Pod.
The SSH command above has the following structure:
```sh theme={"theme":{"light":"github-light","dark":"github-dark"}}
ssh root@[POD_IP_ADDRESS] -p [SSH_PORT] -i [PATH_TO_SSH_KEY]
```
Where:
* `root`: Your assigned username for the Pod (typically `root`).
* `[POD_IP_ADDRESS]`: The public IP address of your Pod.
* `[SSH_PORT]`: The designated public SSH port for your Pod.
* `[PATH_TO_SSH_KEY]`: The local file path to your private SSH key.
## Troubleshooting SSH key authentication
If you're asked for a password when connecting to your Pod via SSH, this means something is not set up correctly. Runpod does not require a password for SSH connections, as authentication is handled entirely through your SSH key pair.
Here are some common reasons why this might happen:
* If you copy and paste the key *fingerprint* (which starts with `SHA256:`) into your Runpod user settings instead of the actual public key (the contents of your `id_ed25519.pub` file), authentication will fail.
* If you omit the encryption type at the beginning of your public key when pasting it into your Runpod user settings (for example, leaving out `ssh-ed25519`), the key will not be recognized.
* If you add multiple public keys to your Runpod user settings but do not separate them with a newline, only the first key will work. Each key must be on its own line.
* If you specify the wrong file path to your private key when connecting, SSH will not be able to find the correct key (`No such file or directory` error).
* If your private key file is accessible by other users on your machine, SSH may refuse to use it for security reasons (`bad permissions` error).
* If your SSH configuration file (`~/.ssh/config`) points to the wrong private key, you will also be prompted for a password. Make sure the `IdentityFile` entry in your config file matches the private key that corresponds to the public key you added to your Runpod account.
## Password-based SSH
To use this method, your Pod must have a public IP address and expose TCP port 22. SSH will be accessible through a mapped external port.
To quickly set up password-based SSH, run this command to download and execute a [helper script](https://github.com/justinwlin/Runpod-SSH-Password/blob/main/passwordrunpod.sh) for password setup:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
wget https://raw.githubusercontent.com/justinwlin/Runpod-SSH-Password/main/passwordrunpod.sh && chmod +x passwordrunpod.sh && ./passwordrunpod.sh
```
While SSH operates on port 22 within your Pod, Runpod assigns a different external port for access. The setup script below automatically detects and uses the correct external port by referencing the `RUNPOD_TCP_PORT_22` environment variable.
If you see the message `Environment variables RUNPOD_PUBLIC_IP or RUNPOD_TCP_PORT_22 are missing` when running the script, it means one or more of the required environment variables are not set. Please ensure you have met all the necessary requirements described above.
After running the script and entering a password, you'll see example commands for SSH or SCP which you can use to connect to your Pod and transfer files from your local machine:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
========================================
SSH CONNECTION
========================================
Connect using: ssh root@38.80.152.73 -p 32061
Password: helloworld
========================================
FILE TRANSFER EXAMPLES (SCP)
========================================
Copy file TO pod:
scp -P 32061 yourfile.txt root@38.80.152.73:/workspace/
Copy file FROM pod:
scp -P 32061 root@38.80.152.73:/workspace/yourfile.txt .
Copy entire folder TO pod:
scp -P 32061 -r yourfolder root@38.80.152.73:/workspace/
```
## Video tutorial (Windows)
# Connection options
Source: https://docs.runpod.io/pods/connect-to-a-pod
Explore Pod connection options, including the web terminal, SSH, JupyterLab, and VSCode/Cursor. See setup and usage details for Runpod Pods.
Choose a connection method based on your workflow:
| Method | Best for | Persistence | Setup |
| ------------------ | --------------------------------------- | ------------- | ------------------ |
| **Web terminal** | Quick commands, debugging | Session-based | None |
| **SSH** | Long-running processes, reliable access | Persistent | SSH client |
| **JupyterLab** | Data science, notebooks | Session-based | Template-dependent |
| **VS Code/Cursor** | Full development environment | Persistent | Extension |
## Web terminal
Browser-based terminal for quick access. Not recommended for long-running processes (use [SSH](#ssh) instead).
1. Navigate to the [Pods page](https://console.runpod.io/pods).
2. Expand your Pod and click **Connect**.
3. Click **Start** if the terminal is stopped, then **Open Web Terminal**.
If **Start** doesn't respond, refresh the page.
## JupyterLab
Interactive web environment for code, files, and data analysis. Available on templates with JupyterLab pre-configured (e.g., "Runpod Pytorch").
1. Deploy a Pod with a JupyterLab-compatible template (all official Runpod PyTorch templates have JupyterLab pre-configured).
2. Navigate to the [Pods page](https://console.runpod.io/pods) and click **Connect**.
3. Under **HTTP Services**, click the **Jupyter Lab** link (usually port 8888).
Create a Pod with JupyterLab access using the CLI:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
runpodctl pod create \
--name my-jupyter-pod \
--gpu-id "NVIDIA GeForce RTX 4090" \
--image "runpod/pytorch:2.1.0-py3.10-cuda11.8.0-devel-ubuntu22.04" \
--container-disk-in-gb 20 \
--volume-in-gb 50 \
--ports "8888/http" \
--env '{"JUPYTER_PASSWORD":"your_secure_password"}'
```
After the Pod starts, access JupyterLab at `https://[POD_ID]-8888.proxy.runpod.net`.
Set the `JUPYTER_PASSWORD` environment variable to configure JupyterLab authentication. If not set, some templates use a default password shown in the Pod logs.
If the JupyterLab tab displays a blank page for more than a minute or two, try restarting the Pod and opening it again.
## SSH
Secure, reliable command-line access for long-running processes and development.
See [Connect with SSH](/pods/configuration/use-ssh) for setup instructions.
## VS Code / Cursor
Connect your local IDE directly to your Pod for a full development experience.
See [Connect to VS Code or Cursor](/pods/configuration/connect-to-ide) for setup instructions.
# Maintenance, outages, and data safety
Source: https://docs.runpod.io/pods/maintenance-and-outages
Understand how Runpod handles planned and unplanned maintenance and how to protect your data. See setup and usage details for Runpod Pods.
> Learn what to expect during planned and unplanned maintenance events, and how to keep your data safe.
Runpod operates on shared infrastructure. Like any cloud platform, maintenance and unexpected outages can occur. This page explains how these situations are handled and what you can do to protect your work.
## Planned maintenance
When scheduled maintenance is required on a machine hosting your pod, Runpod notifies you in advance. Notifications are sent via email before the maintenance window begins so you have time to save your work, back up data, or migrate to another pod.
During a maintenance window, Runpod does not charge you for the time your pod is unavailable. If you cannot wait for maintenance to complete, you can deploy another resource in the meantime.
If you have questions about a maintenance window or believe your pod was impacted, [contact Runpod Support](https://www.runpod.io/contact).
## Unplanned outages
Hardware failures and sudden crashes can happen without warning. In these cases:
* Runpod may only be able to notify you **after** the outage has begun, not before.
* You will be notified as soon as the issue is identified.
If you believe your workload was impacted by an unplanned outage, [contact Runpod Support](https://www.runpod.io/contact) with your pod ID to understand the impact, timeline, and current status of the incident.
## Data safety
Pods use temporary container storage by default. If your pod is interrupted, restarted, stopped, or terminated, any data that is only stored on container storage will be lost. To protect your work, always store important data on a network volume or an external backup.
### Use a network volume
Attach a [network volume](/storage/network-volumes) to your pod to persist data across restarts and pod deletions. This is the most reliable way to ensure your data survives unexpected outages.
### Set up checkpointing
For long-running jobs, implement checkpointing to save progress periodically (every hour to every few hours depending on job length). This limits the amount of work lost if a pod restarts unexpectedly.
Most machine learning frameworks include built-in checkpointing support. See your framework's documentation to get started:
* PyTorch: [Saving and Loading Models](https://docs.pytorch.org/tutorials/beginner/saving_loading_models.html).
* Hugging Face Transformers: [Checkpointing with Trainer](https://huggingface.co/docs/transformers/main_classes/trainer).
* Hugging Face Accelerate: [Checkpointing guide](https://huggingface.co/docs/accelerate/usage_guides/checkpoint).
* PyTorch Lightning: [Checkpointing](https://lightning.ai/docs/pytorch/stable/common/checkpointing.html).
### Maintain backups
The industry standard for data protection is the **3-2-1 rule**:
* **3** copies of your data
* **2** different storage types (for example, a network volume and an external object store)
* **1** copy stored offsite or in a separate location
Use runpodctl or cloud syncs to automate backups. Runpod cannot guarantee recovery of data stored only on ephemeral container disk.
Set up persistent, portable storage that survives pod restarts and deletions.
Compare container disk, volume disk, and network volume storage types.
# Manage Pods
Source: https://docs.runpod.io/pods/manage-pods
Create, start, stop, and terminate Pods using the Runpod console or CLI. Review setup, configuration, and operations guidance for Runpod Pods.
This page covers the core Pod management operations. For CLI usage, first [install the Runpod CLI](/runpodctl/overview) and configure your API key:
```sh theme={"theme":{"light":"github-light","dark":"github-dark"}}
runpodctl config --apiKey RUNPOD_API_KEY
```
## Quick reference
| Action | Web UI | CLI |
| ------------- | -------------------------------------------------------- | ----------------------------------------------------------------- |
| **Deploy** | [Pods page](https://www.console.runpod.io/pods) → Deploy | `runpodctl pod create --name NAME --gpu-id "GPU" --image "IMAGE"` |
| **Start** | Expand Pod → Play icon | `runpodctl pod start POD_ID` |
| **Stop** | Expand Pod → Stop icon | `runpodctl pod stop POD_ID` |
| **Update** | Three-dot menu → Edit Pod | `runpodctl pod update POD_ID` |
| **Terminate** | Expand Pod → Trash icon | `runpodctl pod delete POD_ID` |
| **List** | [Pods page](https://www.console.runpod.io/pods) | `runpodctl pod list` |
## Deploy a Pod
Deploy preconfigured Pods from the [Runpod Hub](/hub/overview#deploy-as-a-pod) for quick setup.
1. Open the [Pods page](https://www.console.runpod.io/pods) and click **Deploy**.
2. (Optional) Attach a [network volume](/storage/network-volumes) for persistent storage.
3. Select **GPU** or **CPU**, then configure:
**GPU**: Select GPU type → Name your Pod → (Optional) Choose a template → Set GPU count → Click **Deploy On-Demand**
**CPU**: Select CPU type → Choose instance configuration → Name your Pod → Click **Deploy On-Demand**
**CUDA compatibility**: Ensure the host CUDA version matches your requirements. If you see "OCI runtime create failed" errors, use **Additional filters → CUDA Versions** to select compatible machines.
```sh theme={"theme":{"light":"github-light","dark":"github-dark"}}
runpodctl pod create \
--name hello-world \
--gpu-id "NVIDIA A40" \
--image "runpod/pytorch:3.10-2.0.0-117" \
--container-disk-in-gb 10 \
--volume-in-gb 100
```
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
curl --request POST \
--url https://rest.runpod.io/v1/pods \
--header 'Authorization: Bearer RUNPOD_API_KEY' \
--header 'Content-Type: application/json' \
--data '{
"name": "my-pod",
"imageName": "runpod/pytorch:2.1.0-py3.10-cuda11.8.0-devel-ubuntu22.04",
"gpuTypeIds": ["NVIDIA GeForce RTX 4090"],
"gpuCount": 1,
"containerDiskInGb": 50,
"volumeInGb": 20
}'
```
To deploy a Pod from an existing template, use the `templateId` parameter instead of specifying individual configuration options:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
curl --request POST \
--url https://rest.runpod.io/v1/pods \
--header 'Authorization: Bearer RUNPOD_API_KEY' \
--header 'Content-Type: application/json' \
--data '{
"name": "my-pod-from-template",
"templateId": "YOUR_TEMPLATE_ID",
"gpuTypeIds": ["NVIDIA GeForce RTX 4090"],
"gpuCount": 1
}'
```
See the [Pod API reference](/api-reference/pods/POST/pods) for all parameters.
## Stop a Pod
Stopping a Pod releases the GPU and preserves data in `/workspace` (volume disk). Container disk data is cleared.
You'll still be charged for [volume disk storage](/pods/storage/types#volume-disk) while stopped. Terminate the Pod if you don't need to retain your environment.
When using a [network volume](/storage/network-volumes), your `/workspace` data is preserved whether you stop or terminate the Pod.
1. Open the [Pods page](https://www.console.runpod.io/pods) and expand your Pod.
2. Click the **Stop** button (square icon) and confirm.
```sh theme={"theme":{"light":"github-light","dark":"github-dark"}}
runpodctl pod stop $RUNPOD_POD_ID
```
**Schedule a stop** (e.g., after 2 hours):
```sh theme={"theme":{"light":"github-light","dark":"github-dark"}}
sleep 2h; runpodctl pod stop $RUNPOD_POD_ID &
```
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
curl --request POST \
--url "https://rest.runpod.io/v1/pods/$RUNPOD_POD_ID/stop" \
--header 'Authorization: Bearer RUNPOD_API_KEY'
```
## Start a Pod
Resume a stopped Pod. Note: You may be allocated [zero GPUs](/pods/troubleshooting/zero-gpus) if capacity has changed.
1. Open the [Pods page](https://www.console.runpod.io/pods) and expand your Pod.
2. Click the **Start** button (play icon).
```sh theme={"theme":{"light":"github-light","dark":"github-dark"}}
runpodctl pod start $RUNPOD_POD_ID
```
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
curl --request POST \
--url "https://rest.runpod.io/v1/pods/$RUNPOD_POD_ID/start" \
--header 'Authorization: Bearer RUNPOD_API_KEY'
```
## Update a Pod
Modify an existing Pod's configuration, such as storage size, image, ports, or environment variables.
Editing a running Pod resets it completely, erasing all data not stored in `/workspace` or a network volume.
1. Open the [Pods page](https://www.console.runpod.io/pods).
2. Click the three-dot menu next to the Pod you want to update.
3. Click **Edit Pod** and modify your configuration.
4. Click **Save** to apply changes.
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
curl --request PATCH \
--url "https://rest.runpod.io/v1/pods/$RUNPOD_POD_ID" \
--header 'Authorization: Bearer RUNPOD_API_KEY' \
--header 'Content-Type: application/json' \
--data '{
"containerDiskInGb": 100,
"volumeInGb": 200
}'
```
See the [Pod API reference](/api-reference/pods/PATCH/pods/podId) for all editable fields.
## Terminate a Pod
Terminating permanently deletes all data not stored in a [network volume](/storage/network-volumes). Export important data first.
1. Open the [Pods page](https://www.console.runpod.io/pods) and expand your Pod.
2. Stop the Pod if running, then click **Terminate** (trash icon) and confirm.
```sh theme={"theme":{"light":"github-light","dark":"github-dark"}}
runpodctl pod delete $RUNPOD_POD_ID
```
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
curl --request DELETE \
--url "https://rest.runpod.io/v1/pods/$RUNPOD_POD_ID" \
--header 'Authorization: Bearer RUNPOD_API_KEY'
```
## View logs
Pods provide two log types:
* **Container logs**: Application output (stdout)
* **System logs**: Pod lifecycle events (startup, shutdown, errors)
Access logs from the [Pods page](https://www.console.runpod.io/pods) by expanding your Pod and clicking **Logs**.
## Troubleshooting
| Issue | Solution |
| ------------------------------ | ------------------------------------------------------------------------------------------------- |
| **Zero GPUs on restart** | See [Zero GPU Pods](/pods/troubleshooting/zero-gpus) |
| **Pod stuck initializing** | Check logs for command errors; ensure you have an idle job (e.g., `sleep infinity`) if using SSH |
| **Docker Compose not working** | Not supported. Use a [custom template](/pods/templates/overview) with your dependencies baked in. |
Need help? [Contact support](https://www.runpod.io/contact).
# Global networking
Source: https://docs.runpod.io/pods/networking
Connect your Pods through a secure private network for internal communication. Review setup, configuration, and operations guidance for Runpod Pods.
Global networking creates a secure, private network that connects all your Pods within your Runpod account. This feature enables Pod-to-Pod communication as if they were on the same local network, regardless of their physical location across different data centers.
Global networking is currently only available for NVIDIA GPU Pods.
## How global networking works
Global networking provides each Pod with a private IP address accessible only to other Pods in your account. This creates an isolated network layer separate from the public internet, which can be used for:
* Distributed computing workloads.
* Microservice architectures.
* Secure database connections.
* Internal API communication.
* Multi-Pod machine learning pipelines.
The network operates at 100 Mbps between Pods, providing reliable connectivity for most inter-Pod communication needs while maintaining security through complete isolation from external networks.
## Enable global networking
To enable global networking for your Pod:
1. Navigate to the [Pods](https://www.console.runpod.io/pods) section and click **Deploy**.
2. At the top of the page, toggle **Global Networking** to filter and show only Pods with networking support.
3. Select your desired GPU configuration and complete the deployment process.
Once deployed, your Pod receives a private IP address and DNS name visible in the Pod details card.
## Connect to other Pods
Each Pod with global networking enabled can be accessed by other Pods using its internal DNS name:
```
POD_ID.runpod.internal
```
Replace `POD_ID` with the target Pod's ID. For example, if your Pod ID is `abc123xyz`, other Pods can reach it at `abc123xyz.runpod.internal`.
### Test connectivity
Verify network connectivity between Pods by opening a web terminal in one Pod and running:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
# To install ping on your Pod, run: apt-get install -y iputils-ping
ping POD_ID.runpod.internal
```
This confirms the private network connection is working correctly.
### Run internal services
Services running on networked Pods are automatically accessible to other Pods without exposing ports publicly. Simply bind your service to all interfaces (`0.0.0.0`) and connect using the internal DNS name.
For example, a database on Pod `abc123xyz` listening on port 5432 would be accessible to other Pods at:
```
abc123xyz.runpod.internal:5432
```
Each service communicates privately through the internal network, reducing attack surface and improving security.
## Security best practices
Global networking provides network isolation, but proper security practices remain essential. Never expose ports on Pods running sensitive services like databases, cache servers, or internal APIs; instead, use global networking for these components. Even within your private Pod network, you should implement authentication between services.
## Supported data centers
Global networking is available in these 17 data centers worldwide:
Global networking is only available in the data centers listed below. When deploying Pods that require global networking, select a supported data center.
| Region ID | Geographic location |
| --------- | ------------------- |
| CA-MTL-3 | Canada |
| EU-CZ-1 | Czech Republic |
| EU-FR-1 | France |
| EU-NL-1 | Netherlands |
| EU-RO-1 | Romania |
| EU-SE-1 | Sweden |
| EUR-IS-2 | Iceland |
| OC-AU-1 | Australia |
| US-CA-2 | California |
| US-GA-1 | Georgia |
| US-GA-2 | Georgia |
| US-IL-1 | Illinois |
| US-KS-2 | Kansas |
| US-NC-1 | North Carolina |
| US-TX-3 | Texas |
| US-TX-4 | Texas |
| US-WA-1 | Washington |
Choose data centers strategically based on:
* Geographic proximity for lower latency
* Compliance requirements for data residency
* Availability of specific GPU types
## Next steps
With global networking configured, explore these related features:
* [Expose ports](/pods/configuration/expose-ports) to make specific services publicly accessible
* Set up [network volumes](/storage/network-volumes) for shared persistent storage.
* Set up [SSH access](/pods/configuration/use-ssh) for secure Pod management.
For additional support or enterprise networking requirements, [contact our customer service team](https://contact.runpod.io/hc/en-us/requests/new).
# Overview
Source: https://docs.runpod.io/pods/overview
Get on-demand access to powerful computing resources. Review setup, configuration, storage, networking, and operations guidance for Runpod Pods.
Pods provide instant access to powerful GPU and CPU resources for AI , , rendering, and other compute-intensive workloads. You have full control over your computing environment, allowing you to customize software, storage, and networking to match your exact requirements.
## Get started
Create an account and deploy your first Pod.
Select the right GPU type and configuration for your workload.
Access your Pod via SSH, JupyterLab, or VS Code.
## Concepts
### [Templates](/pods/templates/overview)
Pre-configured [Docker image](/tutorials/introduction/containers#what-are-images) setups that let you quickly spin up Pods without manual environment configuration. Instead of installing PyTorch, configuring JupyterLab, and setting up all dependencies yourself, you can select an official Runpod PyTorch template and have everything ready to go instantly.
### [Storage](/pods/storage/types)
Pods offer three types of storage: for temporary files, for persistent storage throughout the Pod's lease, and optional s for permanent storage that can be transferred between Pods.
### [Connection](/pods/connect-to-a-pod)
Once deployed, you can connect to your Pod through SSH for command-line access, web proxy for [exposed web services](/pods/configuration/expose-ports), JupyterLab for data science workflows, or [VS Code/Cursor](/pods/configuration/connect-to-ide) for local IDE integration.
## Deployment options
You can deploy Pods in several ways:
* [From a template](/pods/templates/overview): Pre-configured environments for quick setup of common workflows.
* **Custom containers**: Pull from any compatible container registry such as Docker Hub, GitHub Container Registry, or Amazon ECR. Learn more about [creating your own container images](/tutorials/introduction/containers/create-dockerfiles).
* [From Serverless repos](/hub/overview#deploy-as-a-pod): Deploy any -compatible repository from the directly as a Pod.
## Pod types
Runpod offers two cloud options:
* **Secure Cloud:** Operates in T3/T4 data centers, providing high reliability and security for enterprise and production workloads.
* **Community Cloud:** Connects individual compute providers to users through a vetted, secure peer-to-peer system, with competitive pricing options.
## Pricing
Pods are billed by the minute with no fees for ingress/egress. Runpod also offers long-term [savings plans](/pods/pricing#savings-plans) for extended usage patterns. See [Pod pricing](/pods/pricing) for details.
## Limitations
* **Docker Compose is not supported:** Runpod runs Docker for you, so you cannot spin up your own Docker instance or use Docker Compose on Pods.
* **UDP connections are not supported:** Pods only support TCP and HTTP connections.
* **Windows is not supported:** Pods do not currently support Windows.
## Tutorials
Run LLM inference with HTTP API access.
Emulate a Docker-in-Docker workflow.
Build your own reusable Pod template.
# Pricing
Source: https://docs.runpod.io/pods/pricing
Explore pricing options for Pods, including on-demand and savings plans. Review setup, configuration, and operations guidance for Runpod Pods.
Runpod offers custom pricing plans for large scale and enterprise workloads. [Contact our sales team](https://ecykq.share.hsforms.com/2MZdZATC3Rb62Dgci7knjbA) to learn more.
Pods are billed by the second for compute and storage, with no fees for data ingress or egress. Find the latest GPU pricing on the [Runpod console](https://www.console.runpod.io/pods) during Pod deployment.
## Pricing options
| | On-demand | Savings plan |
| -------------- | ---------------------------------------- | --------------------------------- |
| **Pricing** | Standard hourly rate | Discounted (prepaid) |
| **Commitment** | None | 3 or 6 months upfront |
| **Best for** | Development, testing, variable workloads | Long-running production workloads |
### On-demand
Pay-as-you-go pricing. Resources are dedicated to your Pod and cannot be displaced by other users.
You must have at least one hour's worth of credits for your selected configuration to deploy an on-demand instance.
### Savings plans
Commit to a 3-month or 6-month term upfront for significant discounts on compute costs. When you stop a Pod, the savings plan automatically applies to your next deployment of the same GPU type.
Savings plans only cover GPU compute costs—[storage costs](/pods/storage/types) are billed at standard rates. Storage charges continue to accrue on stopped Pods. If your balance reaches \$0, your Pods stop: those with a network volume are preserved, while those without one are terminated and their data cannot be recovered. Plans are non-refundable and have fixed expiration dates.
## Storage pricing
| Storage type | Running Pod | Stopped Pod | Notes |
| ------------------ | ------------------------ | --------------- | -------------------------------------- |
| **Container disk** | \$0.10/GB/month | Not charged | Temporary; erased when Pod stops |
| **Volume disk** | \$0.10/GB/month | \$0.20/GB/month | Persistent; retained until Pod deleted |
| **Network volume** | \$0.07/GB/month (\< 1TB) | \$0.07/GB/month | Permanent; portable between Pods |
| | \$0.05/GB/month (> 1TB) | \$0.05/GB/month | |
Storage is billed per-second for container and volume disks, and hourly for network volumes. You are not charged if the host is unavailable.
Runpod is not designed for long-term cloud storage. Back up critical data to your local machine or a dedicated storage provider.
## Account limits
* **Minimum balance**: Pods are automatically stopped when your account balance reaches \$0. Pods with a network volume are stopped and their data is preserved, while Pods without a network volume are terminated and their data cannot be recovered. To avoid interruptions, enable [low balance notifications](/accounts-billing/billing#low-balance-notifications) or set up [auto-pay](https://www.console.runpod.io/user/billing).
* **Spend limit**: Default limit of \$80/hour across all resources. [Contact support](https://www.runpod.io/contact) to increase.
## Track your costs
* **Savings plans**: Monitor active plans, commitment periods, and expiration dates in the [Savings plans](https://www.console.runpod.io/savings-plans) section.
* **Billing**: Track usage and charges in the [Billing section](https://www.console.runpod.io/user/billing).
# Sync Pod data with cloud storage providers
Source: https://docs.runpod.io/pods/storage/cloud-sync
Learn how to sync your Pod data with popular cloud storage providers. Review setup, configuration, and operations guidance for Runpod Pods.
Cloud Sync uploads and downloads data between your Pod and external cloud storage providers.
## Supported providers
| Provider | Auth method | Setup complexity |
| ---------------------------------------------------------- | -------------------- | ---------------- |
| [**Amazon S3**](#amazon-s3) | Access Key + Secret | Low |
| [**Google Cloud Storage**](#google-cloud-platform-storage) | Service Account JSON | Medium |
| [**Microsoft Azure**](#microsoft-azure-blob-storage) | Account Name + Key | Medium |
| [**Backblaze B2**](#backblaze-b2-cloud-storage) | Application Key | Low |
| [**Dropbox**](#dropbox) | OAuth Access Token | Medium |
Cloud Sync works with Google Cloud Storage, not Google Drive. For Drive transfers, see [file transfer methods](/pods/storage/transfer-files#transfer-with-google-drive).
## Security best practices
When using Cloud Sync, follow these security guidelines to protect your data and credentials:
* Keep all access keys, tokens, and credentials confidential.
* Use dedicated service accounts or application-specific credentials when possible.
* Grant only the minimum permissions required for data transfer.
* Regularly rotate your access credentials.
* Monitor your cloud storage logs for unauthorized access.
## Amazon S3
Amazon S3 provides scalable object storage that integrates seamlessly with Runpod through Cloud Sync.
Follow the steps below to sync your data with Amazon S3:
Navigate to the [Amazon S3 bucket creation form](https://s3.console.aws.amazon.com/s3/bucket/create?region=us-east-1) in your AWS console.
Provide a descriptive name for your bucket and select your preferred AWS Region (this affects data storage location and access speeds).
If you need your bucket to be publicly accessible, uncheck the **Block public access** option at the bottom of the form. For most use cases, keeping this checked provides better security.
Go to **Security credentials** in your AWS account settings. Create a new Access Key on the Security credentials page.
Your Secret Access Key will be displayed only once during creation, so make sure to save it securely.
In the Runpod console, navigate to the [Pods page](https://console.runpod.io/pods) and select the Pod containing your data. Click **Cloud Sync**, then select **AWS S3** from the available providers.
Enter your **AWS Access Key ID** and **Secret Access Key** in the provided fields. Specify the **AWS Region** where your bucket is located and provide the complete bucket path where you want to store your data.
Click **Copy to/from AWS S3** to initiate the transfer. The transfer progress will be displayed in the Runpod interface. Large datasets may take time depending on your Pod's network connection.
## Google Cloud Platform Storage
Google Cloud Storage offers high-performance object storage with global availability and strong consistency.
Follow the steps below to sync your data with Google Cloud Storage:
Access the Google Cloud Storage dashboard and click **Buckets → Create** to start the bucket creation process.
Choose a globally unique name for your bucket. Leave most configuration options at their default settings unless you have specific requirements.
To allow public access to your bucket contents, uncheck **Enforce Public Access Prevention On This Bucket**. Keep this checked for better security unless public access is required.
Create a service account specifically for Runpod access. This provides better security than using your primary account credentials.
Follow [Google's guide on creating service account keys](https://docs.cloud.google.com/iam/docs/keys-create-delete) to generate a JSON key file. This key contains all necessary authentication information.
In the Runpod console, navigate to the [Pods page](https://console.runpod.io/pods) and select the Pod containing your data. Click **Cloud Sync**, then select **Google Cloud Storage** from the available providers.
Paste the entire contents of your Service Account JSON key into the provided field. Specify the source/destination path within your bucket and select which folders from your Pod to transfer.
Click **Copy to/from Google Cloud Storage** to initiate the transfer. The transfer progress will be displayed in the Runpod interface. Large datasets may take time depending on your Pod's network connection.
## Microsoft Azure Blob Storage
Azure Blob Storage provides massively scalable object storage for unstructured data, with seamless integration into the Azure ecosystem.
Follow the steps below to sync your data with Microsoft Azure Blob Storage:
Start by creating a Resource Group to organize your Azure resources. Navigate to [Resource Groups](https://portal.azure.com/#view/HubsExtension/BrowseResourceGroups) in the Azure portal and click **Create**.
Next, set up a Storage Account under [Storage Accounts](https://portal.azure.com/#view/HubsExtension/BrowseResource/resourceType/Microsoft.Storage%2FStorageAccounts). Click **Create** and assign it to your newly created Resource Group.
Navigate to **Security + Networking → Access Keys** in your storage account to retrieve the authentication key.
Create a Blob Container by going to **Storage Browser → Blob Containers** and clicking **Add Container**. Consider creating folders within the container for better organization if you plan to sync data to/from multiple Pods.
In the Runpod console, navigate to the [Pods page](https://console.runpod.io/pods) and select the Pod containing your data. Click **Cloud Sync**, then select **Azure Blob Storage** from the available providers.
Enter your **Azure Account Name** and **Account Key** in the provided fields. Specify the source/destination path in your blob storage where you want to store your data.
Click **Copy to/from Azure Blob Storage** to initiate the transfer. The transfer progress will be displayed in the Runpod interface. Large datasets may take time depending on your Pod's network connection.
## Backblaze B2 Cloud Storage
Backblaze B2 offers affordable cloud storage with S3-compatible APIs and straightforward pricing.
Follow the steps below to sync your data with Backblaze B2 Cloud Storage:
Navigate to [B2 Cloud Storage Buckets](https://secure.backblaze.com/b2_buckets.htm) and click **Create a Bucket**.
Set the bucket visibility to **Public** to allow Runpod access. You can restrict access later using application keys if needed.
Visit [App Keys](https://secure.backblaze.com/app_keys.htm) to create a new application key. This key provides authenticated access to your bucket.
Save both the KeyID and applicationKey securely—the applicationKey cannot be retrieved after creation.
In the Runpod console, navigate to the [Pods page](https://console.runpod.io/pods) and select the Pod containing your data. Click **Cloud Sync**, then select **Backblaze B2** from the available providers.
Enter your **Backblaze B2 Account ID**, **Application Key**, and **bucket path** as shown in the Backblaze interface.
Click **Copy to/from Backblaze B2** to initiate the transfer. The transfer progress will be displayed in the Runpod interface.
## Dropbox
Dropbox integration allows you to sync your Pod data with your Dropbox account using OAuth authentication.
Follow the steps below to sync your data with Dropbox:
Go to the [Dropbox App Console](https://www.dropbox.com/developers/apps/create) to create a new app.
Select **Scoped Access** for API options and **Full Dropbox** for access type. Choose a descriptive name for your app.
In the Dropbox App Console, navigate to the **Permissions** tab. Enable all required checkboxes for read and write access to ensure Cloud Sync can transfer files properly.
Return to the **Settings** tab of your app. In the OAuth2 section, click **Generate** under Generated Access Token.
Save this token immediately—it won't be shown again after you leave the page. This token authenticates Runpod's access to your Dropbox.
In the Runpod console, navigate to the [Pods page](https://console.runpod.io/pods) and select the Pod containing your data. Click **Cloud Sync**, then select **Dropbox** from the available providers.
Paste your **Dropbox Access Token** and specify the remote path where you want to store the data. Creating a dedicated folder in Dropbox beforehand helps with organization.
Click **Copy to/from Dropbox** to initiate the transfer. The transfer progress will be displayed in the Runpod interface.
## Alternative transfer methods
While Cloud Sync provides the easiest way to sync data with cloud providers, you can also transfer files between your Pod and other destinations using:
* **runpodctl**: A built-in CLI tool for peer-to-peer transfers using one-time codes.
* **SSH-based tools**: Use SCP or rsync for direct transfers to your local machine.
* **Network volumes**: For persistent storage across multiple Pods.
For detailed instructions on these methods, see our [file transfer guide](/pods/storage/transfer-files).
## Troubleshooting
If you encounter issues during syncing:
* **Transfer fails immediately**: Verify your credentials are correct and have the necessary permissions.
* **Slow transfer speeds**: Large datasets take time to transfer. Consider compressing data before syncing or using incremental transfers.
* **Permission denied errors**: Ensure your bucket or container has the correct access policies. Some providers require specific permission configurations for external access.
* **Connection timeouts**: Check that your Pod has stable network connectivity. You may need to retry the transfer.
For additional support, consult your cloud provider's documentation or contact Runpod support.
# Transfer files
Source: https://docs.runpod.io/pods/storage/transfer-files
Move files between your local machine and Pods with a variety of secure transfer methods. See setup and usage details for Runpod Pods.
## Choose your transfer method
Runpod supports four different file transfer methods, each optimized for specific use cases:
| Method | Best for | Setup required | File size limit |
| -------------- | --------------------------------- | ------------------------ | --------------------- |
| **runpodctl** | Quick, occasional transfers | Preinstalled on Pods | Small to medium files |
| **SCP** | Standard file operations | SSH configuration | Any size |
| **rsync** | Large datasets, syncing | SSH + rsync installation | Any size |
| **Cloud sync** | Backup, multi-environment sharing | Cloud provider setup | Provider dependent |
### runpodctl
The simplest option for occasional transfers. Uses secure one-time codes and requires no setup since it's pre-installed on all Pods. Perfect for quick file exchanges.
To install `runpodctl` on your local machine, see the [installation guide](/runpodctl/overview).
### SCP
A reliable, standard method that works over SSH. Ideal for users comfortable with command-line tools who need to transfer both individual files and directories.
To configure your Pod for SSH access, see the ([SSH setup guide](/pods/configuration/use-ssh)).
### rsync
The most powerful option, featuring incremental transfers, compression, and detailed progress reporting. Essential for large datasets, regular synchronization, and preserving file attributes.
To set up `rsync`:
* Configure SSH access (same as for SCP).
* Install rsync on both machines: `apt install rsync`
* Ensure your local machine is running a Linux or WSL environment.
### Cloud sync
Direct synchronization with cloud storage providers like AWS S3, Google Cloud Storage, or Dropbox. Best for creating backups or sharing files across multiple environments.
To learn more, see the [cloud sync configuration guide](/pods/storage/cloud-sync).
## Transfer with runpodctl
The [Runpod CLI](/runpodctl/overview) offers the most straightforward approach to file transfer using secure one-time codes. This method works great for occasional transfers but consider other options for large files.
### Send a file
From the source machine (your local computer or a Pod), run:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
runpodctl send YOUR_FILE
```
You'll see output like this:
```text theme={"theme":{"light":"github-light","dark":"github-dark"}}
Sending 'YOUR_FILE' (5 B)
Code is: 8338-galileo-collect-fidel
On the other computer run
runpodctl receive 8338-galileo-collect-fidel
```
The code `8338-galileo-collect-fidel` is your unique, one-time transfer code.
### Receive a file
On the destination machine, use the code provided by the send command:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
runpodctl receive 8338-galileo-collect-fidel
```
You'll see confirmation of the transfer:
```text theme={"theme":{"light":"github-light","dark":"github-dark"}}
Receiving 'YOUR_FILE' (5 B)
Receiving (<-149.36.0.243:8692)
data.txt 100% |████████████████████| ( 5/ 5B, 0.040 kB/s)
```
## Transfer with SCP
SCP provides reliable file transfer over SSH connections. Use this method when you need standard command-line file operations.
### Basic syntax
The general format for SCP commands (replace `43201` and `194.26.196.6` with your Pod's port and IP):
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
scp -P 43201 -i ~/.ssh/id_ed25519 /local/file/path root@194.26.196.6:/destination/file/path
```
If your private key is stored elsewhere or you're using Windows Command Prompt, update the key path accordingly. For quick one-time setups, consider [password-based SSH](/pods/configuration/use-ssh#password-based-ssh).
### Send files to your Pod
Transfer a single file:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
scp -P 43201 -i ~/.ssh/id_ed25519 ~/documents/example.txt root@194.26.196.6:/root/example.txt
```
Transfer a directory (use `-r` for recursive copying):
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
scp -r -P 43201 -i ~/.ssh/id_ed25519 ~/documents/example_dir root@194.26.196.6:/root/example_dir
```
### Download files from your Pod
Simply reverse the source and destination:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
scp -P 43201 -i ~/.ssh/id_ed25519 root@194.26.196.6:/root/example.txt ~/documents/example.txt
```
## Transfer with rsync
`rsync` offers advanced synchronization features and is the best choice for large datasets or regular backup operations.
`rsync` requires a Linux environment or WSL on Windows.
### Basic syntax
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
rsync -e "ssh -p 43201" /source/file/path root@194.26.196.6:/destination/file/path
```
### Essential flags
* `-a` (archive) - Preserves permissions, timestamps, and attributes (essential for directories)
* `-v` (verbose) - Shows detailed transfer information
* `-z` (compress) - Compresses data during transfer (saves bandwidth, uses more CPU)
* `-p` (progress) - Displays transfer progress
* `-d` (delete) - Removes files from destination that don't exist in source
### Send files to your Pod
Transfer with progress and compression:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
rsync -avz -e "ssh -p 43201" ~/documents/example.txt root@194.26.196.6:/root/example.txt
```
### Download from your Pod
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
rsync -avz -e "ssh -p 43201" root@194.26.196.6:/root/example.txt ~/documents/example.txt
```
### Directory synchronization
Transfer directory contents only (note the trailing slash):
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
rsync -avz -e "ssh -p 43201" ~/documents/example_dir/ root@194.26.196.6:/root/example_dir/
```
Transfer the directory itself (no trailing slash):
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
rsync -avz -e "ssh -p 43201" ~/documents/example_dir root@194.26.196.6:/root/
```
### Incremental transfers
rsync's key advantage is intelligent synchronization. Files that already exist at the destination aren't transferred again:
First transfer (full copy):
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
rsync -avz -e "ssh -p 43201" ~/documents/example.txt root@194.26.196.6:/root/example.txt
sending incremental file list
example.txt
119 100% 0.00kB/s 0:00:00 (xfr#1, to-chk=0/1)
sent 243 bytes received 35 bytes 185.33 bytes/sec
total size is 119 speedup is 0.43
```
Second transfer (minimal data):
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
rsync -avz -e "ssh -p 43201" ~/documents/example.txt root@194.26.196.6:/root/example.txt
sending incremental file list
sent 120 bytes received 12 bytes 88.00 bytes/sec
total size is 119 speedup is 0.90
```
## Sync with cloud storage
Connect your Pod storage directly to cloud providers for seamless backup and synchronization.
To set up cloud sync:
1. Navigate to your **My Pods** page
2. Click the **Cloud Sync** option for your Pod
3. Follow the provider-specific configuration steps
For detailed setup instructions with AWS S3, Google Cloud Storage, Azure, Backblaze, and Dropbox, see the [cloud sync configuration guide](/pods/storage/cloud-sync).
## Transfer with Google Drive
You can also use these Colab notebooks to transfer files between Pods and Google Drive:
* [Send files](https://colab.research.google.com/drive/1UaODD9iGswnKF7SZfsvwHDGWWwLziOsr#scrollTo=2nlcIAY3gGLt)
* [Receive files](https://colab.research.google.com/drive/1ot8pODgystx1D6_zvsALDSvjACBF1cj6#scrollTo=RF1bMqhBOpSZ)
## Troubleshooting
Here are some common issues and possible fixes:
**Connection refused errors:**
* Verify SSH is properly configured on your Pod.
* Check that the correct port and IP address are being used.
* Ensure port 22 is exposed in your Pod configuration.
**Permission denied:**
* Confirm your SSH key is correctly specified with `-i`
* Verify the key has appropriate permissions (`chmod 600 ~/.ssh/id_ed25519`)
* Try password-based authentication for quick tests.
**Large file transfers timing out:**
* Use `rsync` instead of SCP for better reliability.
* Add the `-z` flag to compress data during transfer.
* Consider splitting very large files before transfer.
# Storage options
Source: https://docs.runpod.io/pods/storage/types
Choose the right type of storage for your Pods. Review setup, configuration, storage, networking, and operations guidance for Runpod Pods.
Pods offer three storage types optimized for different use cases. Choose based on your data persistence, performance, and sharing needs.
## Comparison
| | Container disk | Volume disk | Network volume |
| --------------- | --------------------- | ----------------------------- | ----------------------------------- |
| **Persistence** | Lost on stop/restart | Retained until Pod deleted | Retained independently |
| **Mount path** | System-managed | `/workspace` (default) | `/workspace` (replaces volume disk) |
| **Performance** | Fastest (local) | Fast (local) | Variable (network) |
| **Shareable** | No | No | Yes (across Pods) |
| **Resizable** | Yes | Increase only | Yes |
| **Cost** | \$0.10/GB/month | \$0.10/GB/month (running) | \$0.07/GB/month |
| | | \$0.20/GB/month (stopped) | |
| **Best for** | OS, temp files, cache | Models, datasets, checkpoints | Shared data, portable storage |
## Container disk
The container disk provides temporary storage for the operating system and session data. It's created when a launches and is cleared when the Pod stops. Use it for temporary files, caches, and data that doesn't need to persist between sessions.
## Volume disk
The volume disk provides persistent storage that is retained throughout the Pod's lease. Data stored in the `/workspace` directory survives Pod stops and restarts, but is deleted when the Pod is terminated. This is ideal for storing models, datasets, and checkpoints that you need to access across multiple sessions.
### Encrypted volumes
You can encrypt your volume disk to protect sensitive data. When encryption is enabled, the volume is encrypted at rest on the host machine, and only your Pod can access the data.
To enable encryption when creating a Pod, select the **Encrypt volume** checkbox in the Pod creation flow.
Your encryption key cannot be retrieved, and bring your own key is not supported. Runpod securely stores your key and passes it only to your container image at runtime.
Encryption applies only to volume disk. Container disk and network volumes cannot be encrypted.
## Network volume
Network volumes provide permanent storage that exists independently from any Pod. You can attach a network volume to multiple Pods, transfer it between machines, and retain your data even after deleting a Pod. This makes network volumes ideal for shared datasets, collaborative workflows, and portable storage.
Network volumes are available in two tiers:
* **Standard storage**: Cost-effective for general-purpose work.
* **[High-performance storage](/storage/high-performance-storage)**: Premium tier with up to 3x throughput and 4x IOPS for demanding workloads.
[Learn more about network volumes](/storage/network-volumes).
Network volumes must be attached during Pod creation and cannot be detached later. When attached, the network volume replaces the volume disk at `/workspace`.
## Modify storage capacity
You can adjust your Pod's storage capacity at any time:
1. Navigate to the [Pods page](https://console.runpod.io/pods).
2. Click the three dots next to your Pod and select **Edit Pod**.
3. Adjust the container or volume disk size. Note that volume disk size can only be increased, not decreased.
4. Click **Save** to apply your changes.
Editing a running Pod resets it completely, erasing all data that isn't stored in your `/workspace` directory.
## Transfer data
You can export data from your Pod to external cloud providers including AWS S3, Google Cloud Storage, Azure, and Dropbox. Click the **Cloud Sync** button on the Pod page to get started. For detailed instructions, see [Export data](/pods/storage/cloud-sync).
Runpod is not designed for long-term cloud storage. We recommend backing up critical data to your local machine or a dedicated cloud storage provider.
## Next steps
Learn how to set up portable, persistent storage for your Pods.
Learn how to move data to and from your Pod.
# Build a custom Pod template
Source: https://docs.runpod.io/pods/templates/create-custom-template
A step-by-step guide to extending Runpod's official templates. Review setup, configuration, and operations guidance for Runpod Pods.
You can find the complete code for this tutorial, including automated build options with GitHub Actions, in the [runpod-workers/pod-template](https://github.com/runpod-workers/pod-template) repository.
This tutorial shows how to build a custom from the ground up. You'll extend an official Runpod template, add your own dependencies, configure how your container starts, and pre-load machine learning models. This approach saves time during Pod initialization and ensures consistent environments across deployments.
By creating custom templates, you can package everything your project needs into a reusable Docker image. Once built, you can deploy your workload in seconds instead of reinstalling dependencies every time you start a new Pod. You can also share your template with members of your team and the wider Runpod community.
## Requirements
Before you begin, you'll need:
* A [Runpod account](/accounts-billing/manage-accounts).
* Docker installed on your local machine or a remote server.
* A Docker Hub account (or access to another container registry).
* Basic familiarity with Docker and Python.
New to Docker? Learn the fundamentals with the [introduction to containers](/tutorials/introduction/containers) tutorial series, which covers [creating Dockerfiles](/tutorials/introduction/containers/create-dockerfiles), [Docker commands](/tutorials/introduction/containers/docker-commands), and [data persistence](/tutorials/introduction/containers/persist-data).
## Step 1: Set up your project structure
First, create a directory for your custom template and the necessary files.
Create a new directory for your template project:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
mkdir my-custom-pod-template
cd my-custom-pod-template
```
Create the following files in your project directory:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
touch Dockerfile requirements.txt main.py
```
Your project structure should now look like this:
## Step 2: Choose a base image and create your Dockerfile
Runpod offers base images with , , and common dependencies pre-installed. You'll extend one of these images to build your custom template.
Runpod offers several base images. You can explore available base images on [Docker Hub](https://hub.docker.com/u/runpod).
For this tutorial, we'll use the PyTorch image, `runpod/pytorch:1.0.2-cu1281-torch280-ubuntu2404` which includes PyTorch 2.8.0, CUDA 12.8.1, and Ubuntu 24.04.
Open `Dockerfile` and add the following content:
```dockerfile Dockerfile theme={"theme":{"light":"github-light","dark":"github-dark"}}
# Use Runpod PyTorch base image
FROM runpod/pytorch:1.0.2-cu1281-torch280-ubuntu2404
# Set environment variables
# This ensures Python output is immediately visible in logs
ENV PYTHONUNBUFFERED=1
# Set the working directory
WORKDIR /app
# Install system dependencies if needed
RUN apt-get update --yes && \
DEBIAN_FRONTEND=noninteractive apt-get install --yes --no-install-recommends \
wget \
curl \
&& rm -rf /var/lib/apt/lists/*
# Copy requirements file
COPY requirements.txt /app/
# Install Python dependencies
RUN pip install --no-cache-dir --upgrade pip && \
pip install --no-cache-dir -r requirements.txt
# Copy application files
COPY . /app
```
This basic Dockerfile:
* Extends the Runpod PyTorch base image.
* Installs system packages (`wget`, `curl`).
* Installs Python dependencies from `requirements.txt`.
* Copies your application code to `/app`.
## Step 3: Add Python dependencies
Now define the Python packages your application needs.
Open `requirements.txt` and add your Python dependencies:
```txt requirements.txt theme={"theme":{"light":"github-light","dark":"github-dark"}}
# Python dependencies
# Add your packages here
numpy>=1.24.0
requests>=2.31.0
transformers>=4.40.0
```
These packages will be installed when you build your Docker image. Add any additional libraries your application requires.
## Step 4: Configure container startup behavior
Runpod base images come with built-in services like Jupyter and SSH. You can choose how your container starts: whether to keep all the base image services running, run your application alongside those services, or run only your application.
There are three ways to configure how your container starts:
**Option 1: Keep all base image services (default)**
The base image automatically starts Jupyter and SSH based on your template settings. This is the default behavior and is ideal for interactive development and remote access.
**Option 2: Run your application after services start**
This option starts Jupyter/SSH in the background, then runs your application. You'll use a startup script for this.
**Option 3: Application only (no Jupyter or SSH)**
This runs only your application with minimal overhead, which is ideal for production deployments where you don't need interactive access.
### Option 1: Keep all base image services (no changes needed)
If you want the default behavior with Jupyter and SSH services, you don't need to modify the Dockerfile. The base image's `/start.sh` script handles everything automatically.
This is already configured in the Dockerfile from Step 2.
### Option 2: Automatically run the application after services start
If you want to run your application alongside Jupyter/SSH services, add these lines to the end of your Dockerfile:
```dockerfile Dockerfile theme={"theme":{"light":"github-light","dark":"github-dark"}}
# Run application after services start
COPY run.sh /app/run.sh
RUN chmod +x /app/run.sh
CMD ["/app/run.sh"]
```
Create a new file named `run.sh` in the same directory as your `Dockerfile`:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
touch run.sh
```
Then add the following content to it:
```bash run.sh theme={"theme":{"light":"github-light","dark":"github-dark"}}
#!/bin/bash
# Start base image services (Jupyter/SSH) in background
/start.sh &
# Wait for services to start
sleep 2
# Run your application
python /app/main.py
# Wait for background processes
wait
```
This script starts the base services in the background, then runs your application.
### Option 3: Configure application-only mode
For production deployments where you don't need Jupyter or SSH, add these lines to the end of your Dockerfile:
```dockerfile Dockerfile theme={"theme":{"light":"github-light","dark":"github-dark"}}
# Clear entrypoint and run application only
ENTRYPOINT []
CMD ["python", "/app/main.py"]
```
This overrides the base image entrypoint and runs only your Python application.
***
For this tutorial, we'll use option 1 (default behavior for the base image services) so we can test out the various connection options.
## Step 5: Pre-load a model into your template
Pre-loading models into your Docker image means that you won't need to re-download a model every time you start up a new Pod, enabling you to create easily reusable and shareable environments for ML .
There are two ways to pre-load models:
* **Option 1: Automatic download from Hugging Face (recommended)**: This is the simplest approach. During the Docker build, Python downloads and caches the model using the transformers library.
* **Option 2: Manual download with wget**: This gives you explicit control and works with custom or hosted models.
For this tutorial, we'll use Option 1 (automatic download from Hugging Face) for ease of setup and testing, but you can use Option 2 if you need more control.
### Option 1: Pre-load models from Hugging Face
Add these lines to your Dockerfile before the `COPY . /app` line:
```dockerfile Dockerfile theme={"theme":{"light":"github-light","dark":"github-dark"}}
# Set Hugging Face cache directory
ENV HF_HOME=/app/models
ENV HF_HUB_ENABLE_HF_TRANSFER=0
# Pre-download model during build
RUN python -c "from transformers import pipeline; pipeline('sentiment-analysis', model='distilbert-base-uncased-finetuned-sst-2-english')"
```
During the build, Python will download the model and cache it in `/app/models`. When you deploy Pods with this template, the model loads instantly from the cache.
### Option 2: Pre-load models with wget
For more control or to use models from custom sources, you can manually download model files during the build.
Add these lines to your Dockerfile before the `COPY . /app` line:
```dockerfile Dockerfile theme={"theme":{"light":"github-light","dark":"github-dark"}}
# Create model directory and download files
RUN mkdir -p /app/models/distilbert-model && \
cd /app/models/distilbert-model && \
wget -q https://huggingface.co/distilbert-base-uncased-finetuned-sst-2-english/resolve/main/config.json && \
wget -q https://huggingface.co/distilbert-base-uncased-finetuned-sst-2-english/resolve/main/model.safetensors && \
wget -q https://huggingface.co/distilbert-base-uncased-finetuned-sst-2-english/resolve/main/tokenizer_config.json && \
wget -q https://huggingface.co/distilbert-base-uncased-finetuned-sst-2-english/resolve/main/vocab.txt
```
***
For this tutorial, we'll use option 1 (automatic download from Hugging Face).
## Step 6: Create your application
Next we'll create the Python application that will run in your Pod. Open `main.py` and add your application code.
Here's an example app that loads a machine learning model and performs inference on sample texts. (You can also replace this with your own application logic.)
```python main.py theme={"theme":{"light":"github-light","dark":"github-dark"}}
"""
Example Pod template application with sentiment analysis.
"""
import sys
import torch
import time
import signal
from transformers import pipeline
def main():
print("Hello from your custom Runpod template!")
print(f"Python version: {sys.version.split()[0]}")
print(f"PyTorch version: {torch.__version__}")
print(f"CUDA available: {torch.cuda.is_available()}")
if torch.cuda.is_available():
print(f"CUDA version: {torch.version.cuda}")
print(f"GPU device: {torch.cuda.get_device_name(0)}")
# Initialize model
print("\nLoading sentiment analysis model...")
device = 0 if torch.cuda.is_available() else -1
# MODEL LOADING OPTIONS:
# OPTION 1: From Hugging Face Hub cache (default)
# Bakes the model into the container image using transformers pipeline
# Behavior: Loads model from the cache, requires local_files_only=True
classifier = pipeline(
"sentiment-analysis",
model="distilbert-base-uncased-finetuned-sst-2-english",
device=device,
model_kwargs={"local_files_only": True},
)
# OPTION 2: From a local directory
# Download the model files using wget, loads them from the local directory
# Behavior: Loads directly from /app/models/distilbert-model
# To use: Uncomment the pipeline object below, comment OPTION 1 above
# classifier = pipeline('sentiment-analysis',
# model='/app/models/distilbert-model',
# device=device)
print("Model loaded successfully!")
# Example inference
test_texts = [
"This is a wonderful experience!",
"I really don't like this at all.",
"The weather is nice today.",
]
print("\n--- Running sentiment analysis ---")
for text in test_texts:
result = classifier(text)
print(f"Text: {text}")
print(f"Result: {result[0]['label']} (confidence: {result[0]['score']:.4f})\n")
print("Container is running. Press Ctrl+C to stop.")
# Keep container running
def signal_handler(sig, frame):
print("\nShutting down...")
sys.exit(0)
signal.signal(signal.SIGINT, signal_handler)
signal.signal(signal.SIGTERM, signal_handler)
try:
while True:
time.sleep(60)
except KeyboardInterrupt:
signal_handler(None, None)
if __name__ == "__main__":
main()
```
If you're pre-loading a model with `wget` (option 2 from step 5), make sure to uncomment the `classifier = pipeline()` object in `main.py` and comment out the `classifier = pipeline()` object for option 1.
## Step 7: Build and test your Docker image
Now that your template is configured, you can build and test your Docker image locally to make sure it works correctly:
Run the Docker build command from your project directory:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
docker build --platform linux/amd64 -t my-custom-template:v1.0 .
```
The `--platform linux/amd64` flag ensures compatibility with Runpod's infrastructure, and is required if you're building on a Mac or ARM system.
The build process will:
* Download the base image.
* Install system dependencies.
* Install Python packages.
* Download and cache models (if configured).
* Copy your application files.
This may take 5-15 minutes depending on your dependencies and model sizes.
**Use versioned tags instead of `:latest`.** The `:latest` tag is mutable and changes every time you push a new image. This can cause unpredictable deployments, make debugging difficult, and conflict with Runpod's image caching. Use semantic versioning (e.g., `:v1.0`, `:v1.1`) to ensure predictable deployments and easy rollbacks.
Check that your image was created successfully:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
docker images | grep my-custom-template
```
You should see your image listed with the `v1.0` tag, similar to this:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
my-custom-template latest 54c3d1f97912 10 seconds ago 10.9GB
```
To test the container locally, run the following command:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
docker run --rm -it --platform linux/amd64 my-custom-template:v1.0 /bin/bash
```
This starts the container and connects you to a shell inside it, exactly like the Runpod web terminal but running locally on your machine.
You can use this shell to test your application and verify that your dependencies are installed correctly. (Press `Ctrl+D` when you want to return to your local terminal.)
When you connect to the container shell, you'll be taken directly to the `/app` directory, which contains your application code (`main.py`) and `requirements.txt`. Your models can be found in `/app/models`.
Try running the sample application (or any custom code you added):
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
python main.py
```
You should see output from the application in your terminal, including the model loading and inference results.
Press `Ctrl+C` to stop the application and `Ctrl+D` when you're ready to exit the container.
## Step 8: Push to Docker Hub
To use your template with Runpod, push to Docker Hub (or another container registry).
Tag your image with your Docker Hub username:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
docker tag my-custom-template:v1.0 YOUR_DOCKER_USERNAME/my-custom-template:v1.0
```
Replace `YOUR_DOCKER_USERNAME` with your actual Docker Hub username.
Authenticate with Docker Hub:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
docker login
```
If you aren't already logged in to Docker Hub, you'll be prompted to enter your Docker Hub username and password.
Push your image to Docker Hub:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
docker push YOUR_DOCKER_USERNAME/my-custom-template:v1.0
```
This uploads your image to Docker Hub, making it accessible to Runpod. Large images may take several minutes to upload.
## Step 9: Create a Pod template in the Runpod console
Next, create a Pod template using your custom Docker image:
Navigate to the [Templates page](https://console.runpod.io/user/templates) in the Runpod console and click **New Template**.
Configure your template with these settings:
* **Name**: Give your template a descriptive name (e.g., "my-custom-template").
* **Container Image**: Enter the Docker Hub image name and tag: `YOUR_DOCKER_USERNAME/my-custom-template:v1.0`.
* **Container Disk**: Set to at least 15 GB.
* **HTTP Ports**: Expand the section, click **Add port**, then enter **JupyterLab** as the port label and **8888** as the port number.
* **TCP Ports**: Expand the section, click **Add port**, then enter **SSH** as the port label and **22** as the port number.
Leave all other settings on their defaults and click **Save Template**.
## Step 10: Deploy and test your template
Now you can deploy and test your template on a Pod:
Go to the [Pods page](https://console.runpod.io/pods) in the Runpod console and click **Deploy**.
Configure your Pod with these settings:
* **GPU**: The Distilbert model used in this tutorial is very small, so you can **select any available GPU**. If you're using a different model, you'll need to [select a GPU](/pods/choose-a-pod) that matches its requirements.
* **Pod Template**: Click **Change Template**. You should see your custom template ("my-custom-template") in the list. Click it to select it.
Leave all other settings on their defaults and click **Deploy On-Demand**.
Your Pod will start with all your pre-installed dependencies and models. The first deployment may take a few minutes as Runpod downloads your image.
Once your Pod is running, click on your Pod to open the connection options panel.
Try one or more connection options:
* **Web Terminal**: Click **Enable Web Terminal** and then **Open Web Terminal** to access it.
* **JupyterLab**: It may take a few minutes for JupyterLab to start. Once it's labeled as **Ready**, click the **JupyterLab** link to access it.
* **SSH**: Copy the SSH command and run it in your local terminal to access it. (See [Connect to a Pod with SSH](/pods/configuration/use-ssh) for details on how to use SSH.)
After you've connected, try running the sample application (or any custom code you added):
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
python main.py
```
You should see output from the application in your terminal, including the model loading and inference results.
To avoid incurring unnecessary charges, make sure to stop and then terminate your Pod when you're finished. (See [Manage Pods](/pods/manage-pods) for detailed instructions.)
## Next steps
Congratulations! You've built a custom Pod template and deployed it to Runpod.
You can use this as a jumping off point to build your own custom templates with your own applications, dependencies, and models.
For example, you can try:
* Adding more dependencies and models to your template.
* Creating different template versions for different use cases.
* Automating builds using GitHub Actions or other CI/CD tools.
* Using [Runpod secrets](/pods/templates/secrets) to manage sensitive information.
For more information on working with templates, see the [Manage Pod templates](/pods/templates/manage-templates) guide.
For more advanced template management, you can use the [Runpod REST API](/api-reference/templates/POST/templates) to programmatically create and update templates.
# Environment variables
Source: https://docs.runpod.io/pods/templates/environment-variables
Configure Pods with environment variables for settings, secrets, and runtime information. See setup and usage details for Runpod Pods.
Environment variables are key-value pairs accessible within your container. Use them to pass configuration settings, secrets, and runtime information without hardcoding values into your code or container image.
## Set environment variables
You can configure up to 50 environment variables per Pod.
**During Pod creation:**
1. Click **Edit Template** and expand **Environment Variables**.
2. Click **Add Environment Variable** and enter the key-value pair.
**In Pod templates:**
1. Navigate to [My Templates](https://www.console.runpod.io/user/templates).
2. Create or edit a template and add variables in the **Environment Variables** section.
**Using secrets:**
Reference [Runpod secrets](/pods/templates/secrets) for sensitive data:
```
API_KEY={{ RUNPOD_SECRET_my_api_key }}
DATABASE_PASSWORD={{ RUNPOD_SECRET_db_password }}
```
## Update environment variables
1. Go to [Pods](https://www.console.runpod.io/user/pods) and click the three dots next to your Pod.
2. Select **Edit Pod** and expand **Environment Variables**.
3. Add or update variables and click **Save**.
Updating environment variables restarts your Pod, clearing all data outside your volume mount path (`/workspace` by default).
## Access environment variables
**In your Pod's terminal:**
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
echo $VARIABLE_NAME # View specific variable
env | grep RUNPOD # List Runpod variables
```
**In your code:**
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
import os
model_name = os.environ.get('MODEL_NAME', 'default-model')
api_key = os.environ['API_KEY'] # Raises error if not set
```
```javascript theme={"theme":{"light":"github-light","dark":"github-dark"}}
const modelName = process.env.MODEL_NAME || 'default-model';
const apiKey = process.env.API_KEY;
```
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
MODEL_NAME=${MODEL_NAME:-"default-model"}
echo "Using model: $MODEL_NAME"
```
## Runpod-provided variables
Runpod automatically sets these environment variables:
| Variable | Description |
| --------------------- | --------------------------------- |
| `RUNPOD_POD_ID` | Unique Pod identifier. |
| `RUNPOD_DC_ID` | Data center identifier. |
| `RUNPOD_POD_HOSTNAME` | Server hostname. |
| `RUNPOD_GPU_COUNT` | Number of GPUs available. |
| `RUNPOD_CPU_COUNT` | Number of CPUs available. |
| `RUNPOD_PUBLIC_IP` | Public IP address (if available). |
| `RUNPOD_TCP_PORT_22` | Public port mapped to SSH. |
| `RUNPOD_VOLUME_ID` | Attached network volume ID. |
| `RUNPOD_API_KEY` | Pod-scoped API key. |
| `PUBLIC_KEY` | Authorized SSH public keys. |
| `CUDA_VERSION` | Installed CUDA version. |
| `PYTORCH_VERSION` | Installed PyTorch version. |
## Best practices
* **Use secrets for sensitive data**: Never hardcode API keys or passwords. Use [Runpod secrets](/pods/templates/secrets).
* **Validate required variables**: Check that critical variables are set before your application starts.
* **Provide defaults**: Use fallback values for non-critical configuration.
* **Use descriptive names**: Prefer `DATABASE_PASSWORD` over `DB_PASS`.
* **Group related variables**: Use consistent prefixes like `DB_HOST`, `DB_PORT`, `DB_NAME`.
# Manage Pod templates
Source: https://docs.runpod.io/pods/templates/manage-templates
Learn how to create, and manage custom Pod templates. Review setup, configuration, storage, networking, and operations guidance for Runpod Pods.
Creating a custom template allows you to package your specific configuration for reuse and sharing. Templates define all the necessary components to launch a with your desired setup.
## Template configuration options
When creating a template, you'll configure several key components:
**Name:** The display name for your template that will appear in the template browser. Choose a descriptive name that clearly indicates the template's purpose and contents.
**Container image:** The path to the Docker image that forms the foundation of your template. This is where the core functionality of your template resides, including all software packages, dependencies, and files needed for your workload. You can import the image from:
* A public registry like Docker Hub (e.g., `ubuntu:latest`, `pytorch/pytorch:latest`).
* Your own private registry (requires registry credentials).
**Template visibility:** Choose whether your template is available to others. Public templates are available to all Runpod users in the Explore section of the console, while private template are only accessible to you or your team members.
**Compute type:** Templates are restricted to specific compute types and can only be used with matching hardware:
* **NVIDIA GPU:** For GPU-accelerated workloads requiring CUDA support
* **AMD GPU:** For workloads optimized for AMD graphics processors
* **CPU:** For CPU-only workloads that don't require GPU acceleration
**Container start command:** Customize the command that runs when your Pod starts. This overrides the default CMD instruction in your Docker container. You can specify:
* Simple bash commands: `bash -c 'mkdir /workspace && /start.sh'`
* JSON format with entrypoint and cmd: `{"cmd": ["python", "app.py"], "entrypoint": ["bash", "-c"]}`
Most Docker images have built in start commands, so you can usually leave this blank. When customizing your start command, make sure you're not overriding existing commands that are critical for the image to run.
**Registry credentials:** If using a private container image, provide authentication credentials to access your private registry. This ensures Runpod can pull your image during Pod deployment.
**Storage configuration:** Define the storage requirements for your template, including:
* **Container disk size:** The amount of storage allocated for the container's filesystem, including the operating system and installed packages.
* **Volume disk size:** Additional persistent storage that will be mounted to your Pod. This storage persists between Pod restarts and can be used for data, models, and other files you want to preserve.
* **Volume mount path:** The directory path where the persistent volume will be mounted inside the container (commonly `/workspace`).
**Network configuration:** Configure network access for your template:
* **HTTP ports:** Ports that will be accessible via Runpod's HTTP proxy for web interfaces and APIs. These are automatically secured with HTTPS and accessible through Runpod's proxy URLs.
* **TCP ports:** Direct TCP port access for services that require raw TCP connections, such as SSH, databases, or custom protocols.
**Environment variables:** Define key-value pairs that will be available as environment variables inside your Pod. These are useful for:
* Configuration settings specific to your application.
* API keys and credentials (consider using [Secrets](/pods/templates/secrets) for sensitive data).
* Runtime parameters that customize your template's behavior.
## Creating templates
To learn how to create your own custom templates, see [Build a custom Pod template](/pods/templates/create-custom-template).
The Runpod console provides an intuitive interface for template creation:
1. Navigate to the **[Templates](https://www.console.runpod.io/user/templates)** section.
2. Click **New Template**.
3. Fill in the configuration options described above.
4. Click **Save Template** to save it to your account.
5. Test your template by deploying a Pod to ensure it works as expected.
You can also create templates programmatically using the Runpod REST API. For example:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
curl --request POST \
--url https://rest.runpod.io/v1/templates \
--header 'Authorization: Bearer YOUR_API_KEY' \
--header 'Content-Type: application/json' \
--data '{
"category": "NVIDIA",
"containerDiskInGb": 50,
"dockerEntrypoint": [],
"dockerStartCmd": [],
"env": {
"ENV_VAR": "value"
},
"imageName": "CONTAINER_IMAGE",
"isPublic": false,
"isServerless": false,
"name": "TEMPLATE_NAME",
"ports": [
"8888/http",
"22/tcp"
],
"readme": "",
"volumeInGb": 20,
"volumeMountPath": "/workspace"
}'
```
For more details, see the [API reference](/api-reference/templates/POST/templates).
## Using environment variables in templates
Environment variables provide a flexible way to configure your Pod's runtime behavior without modifying the container image.
### Defining environment variables
Environment variables are key-value pairs that become available inside your Pod's container. When creating a template, you can define variables by specifying:
* **Key**: The environment variable name (e.g., `DATABASE_URL`, `API_KEY`).
* **Value**: The value assigned to that variable.
### Use cases for environment variables
Environment variables are particularly useful for:
* **Configuration settings**: Database connections, API endpoints, feature flags.
* **Runtime parameters**: Model paths, batch sizes, processing options.
* **Integration credentials**: API keys, authentication tokens (consider using [Secrets](/pods/templates/secrets) for sensitive data).
* **Application behavior**: Debug modes, logging levels, output formats.
### Runpod system environment variables
Runpod automatically provides several [predefined environment variables](/pods/templates/environment-variables) in every Pod, for example:
* `RUNPOD_POD_ID`: Unique identifier for your Pod.
* `RUNPOD_API_KEY`: API key for making Runpod API calls from within the Pod.
* `RUNPOD_POD_HOSTNAME`: Hostname of the server running your Pod.
### Using secrets in templates
For sensitive information like passwords and API keys, use [Runpod secrets](/pods/templates/secrets) instead of plain environment variables. Secrets are encrypted and can be referenced in your templates using the format:
```
{{ RUNPOD_SECRET_secret_name }}
```
This approach ensures sensitive data is properly protected while still being accessible to your Pod.
# Overview
Source: https://docs.runpod.io/pods/templates/overview
Streamline your Pod deployments with templates, bundling prebuilt container images with hardware specs and network settings.
templates are pre-configured [Docker image](/tutorials/introduction/containers#what-are-images) setups that let you quickly spin up Pods without manual environment configuration. Instead of installing PyTorch, configuring JupyterLab, and setting up dependencies yourself, you can select a template and have everything ready instantly.
Browse official and community templates.
Build your own reusable Pod configuration.
Edit, share, and organize your templates.
Configure template behavior with variables.
## Template types
| Type | Description | Support |
| ------------- | --------------------------------------------------------------------------------------------------- | ----------------------------------------------------------- |
| **Official** | Curated by Runpod with proven demand and maintained quality. Regularly tested and updated. | Full Runpod support |
| **Community** | Created by users and promoted based on community usage. Wide variety of specialized configurations. | [Community Discord](https://discord.com/invite/runpod) only |
| **Custom** | Created by you for specialized workloads. Can be private or shared publicly. | Self-supported |
Runpod does not maintain or provide customer support for community templates. If you encounter issues, contact the template creator directly or seek help on the [community Discord](https://discord.com/invite/runpod).
## What templates include
Templates contain all components needed to launch a fully configured Pod:
* **Container image**: The Docker image with all software packages and dependencies.
* **Hardware specifications**: Container disk size, volume size, and mount paths.
* **Network settings**: Exposed HTTP and TCP ports for web UIs or APIs.
* **Environment variables**: Pre-configured settings that customize application behavior.
* **Startup commands**: Instructions that run when the Pod launches.
# Manage secrets
Source: https://docs.runpod.io/pods/templates/secrets
Securely store and manage sensitive information like API keys, passwords, and tokens with Runpod secrets.
This guide shows how to create, view, edit, delete, and use secrets in your to protect sensitive data and improve security.
## What are Runpod secrets
Secrets are encrypted strings that store sensitive information separately from your template configuration, providing a secure way to store and manage sensitive information such as API keys, passwords, and authentication tokens in your Pod templates.
After creating a secret, you can safely reference it in your templates without exposing any sensitive data in plain text:
```
{{ RUNPOD_SECRET_secret_name }}
```
This approach offers several advantages:
* **Security**: Sensitive data is encrypted and never displayed in plain text once created, protecting against accidental exposure.
* **Reusability**: The same secret can be referenced across multiple templates and Pods without duplication.
* **Access control**: Secrets are tied to your account or team, ensuring only authorized users can access them.
This makes secrets particularly useful for:
* **API authentication**: Store API keys for services like OpenAI, Hugging Face, or cloud providers without hardcoding them in your templates.
* **Database credentials**: Securely provide database connection strings and passwords to your applications.
* **Model access tokens**: Store authentication tokens required to download gated models or datasets.
* **Service integration**: Keep webhook URLs, service account keys, and other integration credentials secure.
## Create a secret
You can create secrets through the Runpod web interface to securely store sensitive information:
1. Go to the [Secrets](https://www.console.runpod.io/user/secrets) section in the Runpod console.
2. Click **Create Secret** to open the creation form.
3. Provide the required information:
* **Secret Name**: A unique identifier for your secret (e.g., `huggingface_token`, `database_password`).
* **Secret Value**: The actual sensitive data you want to store.
* **Description** (optional): A helpful description of what this secret contains or how it's used.
4. Click **Create Secret** to save your encrypted secret.
Once a secret is created, its value cannot be viewed through the interface. This is a security feature that prevents accidental exposure of sensitive data. If you need to verify or change the value, you must modify the secret or create a new one.
## View secret details
You can view metadata about your secrets without exposing the sensitive values:
1. Go to the [Secrets](https://www.console.runpod.io/user/secrets) section.
2. Click on the secret name you want to inspect.
3. Click the configuration icon and select **View Secret**.
This shows you the secret name, description, and creation date, but never the actual secret value.
## Modify a secret value
To update the value of an existing secret:
1. Go to the [Secrets](https://www.console.runpod.io/user/secrets) section.
2. Click on the name of the secret you want to modify.
3. Click **Manage** and select **Edit Secret Value**.
4. Enter the new secret value.
5. Click **Save Changes** to update the encrypted value.
## Delete a secret
To permanently remove a secret:
1. Go to the [Secrets](https://www.console.runpod.io/user/secrets) section.
2. Click on the secret you want to delete.
3. Click **Manage** and select **Delete Secret**.
4. Type or copy/paste the secret name to confirm deletion.
5. Click **Confirm Delete** to permanently remove the secret.
Deleting a secret is permanent and cannot be undone. Make sure no active templates or Pods are using the secret before deletion, as this will cause those deployments to fail.
## Using secrets in Pod templates
Once you've created secrets, you can reference them in your Pod templates to provide secure access to sensitive data.
### Direct reference method
Reference your secrets directly in the [environment variables section](/pods/templates/environment-variables) of your Pod template using the `RUNPOD_SECRET_` prefix followed by your secret name:
```
{{ RUNPOD_SECRET_secret_name }}
```
For example, if you created a secret named `huggingface_token`, you would reference it as:
```
{{ RUNPOD_SECRET_huggingface_token }}
```
This syntax tells Runpod to substitute the encrypted secret value when the Pod starts, making it available as an environment variable inside your container.
### Web interface selection
When creating or editing a Pod template through the web interface, you can also:
1. Navigate to the environment variables section of your template.
2. Use the secret selector (click the key icon) to choose from your available secrets.
3. The interface will automatically format the reference syntax for you.
## Best practices for using secrets
* **Naming conventions**: Use descriptive names that clearly indicate the secret's purpose (e.g., `openai_api_key`, `database_password`, `github_token`).
* **Environment variable mapping**: Map secrets to appropriately named environment variables in your templates:
```
API_KEY={{ RUNPOD_SECRET_openai_key }}
DATABASE_URL={{ RUNPOD_SECRET_db_connection }}
```
* **Minimal exposure**: Only include secrets in templates that actually need them to reduce the attack surface.
* **Regular rotation**: Periodically update secret values, especially for long-lived credentials like API keys.
# JupyterLab blank page issue
Source: https://docs.runpod.io/pods/troubleshooting/jupyterlab-blank-page
What to do when you open JupyterLab on a Pod and see a blank or non-responsive page. See setup and usage details for Runpod Pods.
When opening JupyterLab on your Pod, you may see a blank white page, even when the JupyterLab link in the Pod connection panel says it's "Ready." This page provides guidance to help you troubleshoot this issue.
## Understanding JupyterLab readiness
When you start a Pod with JupyterLab configured, it periodically pings the JupyterLab server to check its status. When the console displays a "Ready" status, it only means that the Jupyter server [`/api/status`](https://jupyter-server.readthedocs.io/en/latest/developers/rest-api.html#get--api-status) endpoint is responding to HTTP requests, not that JupyterLab has successfully started. It may take additional time to fully load and become usable.
## Common causes
A blank JupyterLab page can occur for several reasons, and sometimes multiple issues occur at once:
* **Pod still starting up**: You might see a blank white screen for some time after starting your Pod. This happens because Pod resources or services aren't fully initialized yet (even though it may be listed as "Running" in the console).
The best way to check if your Pod is ready to use is by checking the **Telemetry** tab in the Pod details page in the Runpod console. If a Pod is receiving telemetry, it should be ready to use, but individual services like JupyterLab may take a few minutes to start up.
* **Jupyter service still loading**: The Jupyter service may still be spinning up or performing initial setup work.
* **Browser cache issues**: If you keep seeing the same blank screen after closing and reopening the JupyterLab link, your browser (or an intermediate layer) might be serving a cached broken response.
* **Network problems**: If JupyterLab fails to load or stalls and other sites are also slow, you may have local network or connectivity issues between your machine and the Pod.
* **Misconfigured Pod or template**: If JupyterLab never loads despite multiple restarts and long waits, the image or template might not be starting Jupyter on the expected port or path.
## Troubleshooting steps
Follow the steps below based on what you're currently seeing.
### Status shows "Initializing"
Wait at least 30 to 60 seconds after starting your Pod before opening JupyterLab. The Pod needs time to fully initialize all services.
### Status shows "Ready" but page is blank
1. Wait on the blank screen for at least 60 seconds to allow JupyterLab to finish starting.
2. Try a hard refresh:
* Windows/Linux: Press Ctrl + Shift + R
* Mac: Press Cmd + Shift + R
3. Open the JupyterLab link in a private or incognito browser window to rule out browser caching.
4. If still blank, check your Pod logs in the Runpod console for JupyterLab-related errors or confirmation that Jupyter has started. Look for messages indicating that Jupyter is running on port 8888 (or your configured port). If you don't see any messages, check the Pod logs for errors.
Runpod only performs the JupyterLab health check on port 8888. If you're using a different port to expose the service, we recommend changing the port in your template configuration to 8888.
5. If you see errors or Jupyter never appears to start, restart the Pod and repeat the steps above.
### Repeated restarts
If you restart the same Pod more than three times and JupyterLab never loads, treat this as a configuration or template issue rather than a transient startup delay.
Verify the following:
* You're using a template that supports JupyterLab on the documented port (typically 8888).
* Any required environment variables or startup commands for Jupyter are correctly set in your template configuration. If you're not sure, check the template documentation.
If still stuck, share your Pod logs and template configuration with support or in the [Runpod Discord](https://discord.com/invite/runpod).
Runpod does not maintain or provide customer support for community templates. If you encounter issues, contact the template creator directly or seek help on the [community Discord](https://discord.com/invite/runpod).
# JupyterLab checkpoints folder access
Source: https://docs.runpod.io/pods/troubleshooting/jupyterlab-checkpoints-folder
Access folders named checkpoints in JupyterLab by renaming them, moving files through the interface, or using terminal commands.
If you're unable to open a folder named "checkpoints" in JupyterLab, this is a known issue where JupyterLab treats "checkpoints" as a reserved keyword.
## The issue
JupyterLab cannot open any directory named exactly "checkpoints". When you try to click on it, nothing happens because the system triggers its internal `listCheckpoints` function instead of opening the directory. This commonly affects ML model directories and ComfyUI installations.
## Solutions
### Option 1: Temporarily rename via terminal
Access the terminal in JupyterLab or through SSH and rename the folder:
```bash Rename to access theme={"theme":{"light":"github-light","dark":"github-dark"}}
# Rename to make it accessible
mv checkpoints checkpoint
# After working with the files, rename back
mv checkpoint checkpoints
```
The `mv` command only renames the folder, it doesn't delete any data.
### Option 2: Drag and drop method
Download your files to a different directory, then use JupyterLab's interface to drag and drop files into the checkpoints folder. The folder won't open, but it will accept dropped files.
### Option 3: Use the terminal for file operations
Work with the checkpoints folder entirely through the terminal:
```bash Access checkpoints via terminal theme={"theme":{"light":"github-light","dark":"github-dark"}}
# List contents
ls -la checkpoints/
# Copy files in
cp /path/to/model.safetensors checkpoints/
# Move files in
mv ~/downloaded-model.ckpt checkpoints/
```
## Related information
This issue has been reported in multiple GitHub issues and affects both JupyterLab and classic Jupyter Notebook. The folder name "checkpoints" conflicts with Jupyter's internal checkpoint system for notebook files.
# Pod migration
Source: https://docs.runpod.io/pods/troubleshooting/pod-migration
Automatically migrate your Pod to a new machine when your GPU is unavailable. Review setup, configuration, and operations guidance for Runpod Pods.
Pod migration is currently in beta. [Join our Discord](https://discord.com/invite/runpod) if you'd like to provide feedback.
When you start a Pod, it's assigned to a specific physical with 4-8 GPUs. This creates a link between your Pod and that particular machine. As long as your Pod is running, that GPU is exclusively reserved for you, which ensures stable pricing and prevents your work from being interrupted.
When you stop a Pod, you release that specific GPU, allowing other users to rent it. If another user rents the GPU while your Pod is stopped, the GPU will be occupied when you try to restart. Because your Pod is still tied to that original machine, you'll see message asking you to migrate your Pod. This doesn't mean there are no GPUs of that type available on Runpod, just that none are available on the specific physical machine where your Pod's data is stored.
## Your options when GPUs are unavailable
When prompted to migrate your Pod, you have three options:
1. **Do nothing**: If you don't want to migrate your data, you can wait and try again later. The GPU will become available once another user stops their Pod on that machine.
2. **Start Pod with CPUs**: If you don't need GPU access immediately, you can start your Pod with CPUs only. This lets you access your data and manually migrate files if needed, but the Pod will have limited CPU resources and is not suitable for compute-intensive tasks.
3. **Automatically migrate Pod data**: This option spins up a new Pod with the same specifications as your current one and automatically migrates your data to a machine with available GPUs. The migration process finds a new machine with your requested GPU type, provisions the instance, and transfers your network volume data from the old Pod to the new one.
## Important considerations for migration
When you trigger an automatic Pod migration, you'll receive a new Pod with a new ID and IP address. This is because Pod IDs are architecturally tied to specific physical machines.
This may impact your workload if you have:
* A Pod ID hardcoded in an API call.
* A proxy URL hardcoded (e.g., `http://b63b243b47bd340becc72fbe9b3e642c.proxy.runpod.net`).
* A firewall or VPN configured with a specific Pod ID.
* A firewall or VPN configured with a specific Pod IP address.
* A specific URL for your server (when you start a new Pod, you'll get a new URL for any UI or server you've set up).
## Preventing Pod migration scenarios
The most effective way to avoid the need for Pod migrations is to use [network volumes](/storage/network-volumes). Network volumes decouple your data from specific physical machines, storing your `/workspace` data on a separate, persistent volume that can be attached to any Pod. If you need to terminate a Pod, you can deploy a new one and attach the same network volume, giving you immediate access to your data on any machine with an available GPU.
# Storage full
Source: https://docs.runpod.io/pods/troubleshooting/storage-full
Diagnose and resolve storage-full errors on Runpod Pods by checking disk usage, locating large files, and managing container or volume storage.
Storage full errors can occur when users generate many files, transfer files, or perform other storage-intensive tasks. This document provides guidance to help you troubleshoot this.
## Check disk usage
When encountering a storage full, the first step is to check your container’s disk usage. You can use the `df -h` command to display a summary of disk usage.
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
df -h
```
Example output:
```sh theme={"theme":{"light":"github-light","dark":"github-dark"}}
root@9b8e325167b2:/# df -h
Filesystem Size Used Avail Use% Mounted on
overlay 20G 16M 20G 1% /
tmpfs 64M 0 64M 0% /dev
tmpfs 252G 0 252G 0% /sys/fs/cgroup
shm 24G 0 24G 0% /dev/shm
/dev/sda2 457G 12G 423G 3% /usr/bin/nvidia-smi
tmpfs 252G 12K 252G 1% /proc/driver/nvidia
tmpfs 252G 4.0K 252G 1% /etc/nvidia/nvidia-application-profiles-rc.d
tmpfs 51G 4.4M 51G 1% /run/nvidia-persistenced/socket
tmpfs 252G 0 252G 0% /proc/asound
tmpfs 252G 0 252G 0% /proc/acpi
tmpfs 252G 0 252G 0% /proc/scsi
tmpfs 252G 0 252G 0% /sys/firmware
tmpfs 252G 0 252G 0% /sys/devices/virtual/powercap
```
## Key areas to check
**Container Disk Usage**: The primary storage area for your container is mounted on the `overlay` filesystem. This indicates the container's root directory.
```sh theme={"theme":{"light":"github-light","dark":"github-dark"}}
Filesystem Size Used Avail Use% Mounted on
overlay 20G 16M 20G 1% /
```
You can use the command `du -sh .` to check the space usage of the current directory.
By default, the volume disk or network volume is mounted at `/workspace`, You can check the usage with the following example::
```sh theme={"theme":{"light":"github-light","dark":"github-dark"}}
root@9b8e325167b2:/# cd workspace/
root@9b8e325167b2:/workspace# du -sh .
194M .
```
**Identifying Large Files**: To identify the top 10 largest files in your `/workspace`, you can run the following command:
```sh theme={"theme":{"light":"github-light","dark":"github-dark"}}
root@9b8e325167b2:/# find /workspace -type f -exec du -h {} + | sort -rh | head -n 10
96M /workspace/f.txt
96M /workspace/e.txt
1.0K /workspace/c.txt
512 /workspace/b.txt
512 /workspace/a.txt
```
## Removing files and directories
Once you've identified large files or directories that are no longer needed, you can remove them to free up space.
This will permanently delete the file, folder. Use with caution.
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
# To delete a specific file, use the rm command:
rm /path/to/file
# To remove an entire directory and its contents, use the rm -r command:
rm -r /path/to/directory
```
## JupyterLab hidden trash bin
If you deleted files through JupyterLab's UI but storage isn't freed, they may be in a hidden trash directory.
### Quick check and fix
Check if trash directories exist:
```bash Check for trash directories theme={"theme":{"light":"github-light","dark":"github-dark"}}
{ ls -lah $HOME/.local/share/Trash/ 2>/dev/null && echo "✓ Found: $HOME/.local/share/Trash/"; ls -lah /workspace/.Trash* 2>/dev/null && echo "✓ Found: /workspace/.Trash*"; } || echo "✗ No trash directories found"
```
If trash was found above, clear it:
```bash Clear trash directories theme={"theme":{"light":"github-light","dark":"github-dark"}}
[ -d "$HOME/.local/share/Trash" ] && rm -rf $HOME/.local/share/Trash/* && echo "✓ Cleared home trash" || echo "✗ No home trash to clear"
ls -d /workspace/.Trash* 2>/dev/null && rm -rf /workspace/.Trash* && echo "✓ Cleared workspace trash" || echo "✗ No workspace trash to clear"
```
## Additional storage
If your Pod needs more than 20GB of storage, consider using a network volume. For more information, see [Network volumes](/storage/network-volumes), or refer to [this blog post](https://www.runpod.io/blog/network-volumes-on-runpod-secure-cloud).
# JupyterLab server token authentication
Source: https://docs.runpod.io/pods/troubleshooting/token-authentication-enabled
Recover the JupyterLab server token from a Runpod Pod terminal, then use it to authenticate when the JupyterLab login screen appears.
If you see a "Token authentication is enabled" screen when trying to access your Pod's JupyterLab server, follow the steps below to log in.
1. Go to the Pod page in the Runpod console and click the **Connect** button for the Pod you want to access.
2. Look for the **Web Terminal** start button.
3. Click **Start**, then open the web terminal.
4. In the terminal, run the following command to get the JupyterLab server token:
```
jupyter server list
```
You should see output similar to this:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
root@2779b5db68b8:/# jupyter server list
Currently running servers:
http://localhost:8888/?token=ua5nw5fwkdzseqpp5apj :: /
root@2779b5db68b8:/#
```
The token you need is the string of characters that appears after the `=` sign, such as `ua5nw5fwkdzseqpp5apj` in the example above.
Copy this token, return to your JupyterLab login page, and paste it into the **Token** field to sign in.
# 502 errors
Source: https://docs.runpod.io/pods/troubleshooting/troubleshooting-502-errors
Troubleshoot 502 errors on Runpod Pods by checking GPU attachment, application status, port configuration, logs, and template requirements.
502 errors can occur when users attempt to access a program running on a specific port of a deployed Pod and the program isn't running or has encountered an error. This document provides guidance to help you troubleshoot this error.
### Check your Pod's GPU
The first step to troubleshooting a 502 error is to check whether your Pod has a GPU attached.
1. **Access your Pod's settings**: Click on your Pod's settings in the user interface to access detailed information about your Pod.
2. **Verify GPU attachment**: Here, you should be able to see if your Pod has a GPU attached. If it does not, you will need to attach a GPU.
If a GPU is attached, you will see it under the Pods screen (e.g. 1 x A6000). If a GPU is not attached, this number will be 0. Runpod does allow you to spin up a Pod with 0 GPUs so that you can connect to it via a Terminal or CloudSync to access data. However, the options to connect to Runpod via the web interface will be nonfunctional, even if they are lit up.
### Check your Pod's logs
After confirming that your Pod has a GPU attached, the next step is to check your Pod's logs for any errors.
1. **Access your Pod's logs**: You can view the logs from the Pod's settings in the user interface.
2.
**Look for errors**: Browse through the logs to find any error messages that may provide clues about why you're experiencing a 502 error.
### Verify additional steps for official templates
In some cases, for our official templates, the user interface does not work right away and may require additional steps to be performed by the user.
1. **Access the template's ReadMe**: Navigate to the template's page and open the ReadMe file.
2. **Follow additional steps**: The ReadMe file should provide instructions on any additional steps you need to perform to get the UI functioning properly. Make sure to follow these instructions closely.
Remember, each template may have unique requirements or steps for setup. It is always recommended to thoroughly review the documentation associated with each template.
If you continue to experience 502 errors after following these steps, please contact our support team. We're here to help ensure that your experience on our platform is as seamless as possible.
# Zero GPU Pods on restart
Source: https://docs.runpod.io/pods/troubleshooting/zero-gpus
What to do when your Pod machine has zero GPUs. Review setup, configuration, storage, networking, and operations guidance for Runpod Pods.
When you restart a stopped Pod, you might see a message telling you that there are "Zero GPU Pods." This is because there are no GPUs available on the where your Pod was running.
## Why does this happen?
When you deploy a Pod, it's assigned to a GPU on a specific physical machine. This creates a link between your Pod and that particular piece of hardware. As long as your Pod is running, that GPU is exclusively reserved for you. When you stop your Pod, you release that specific GPU, allowing other users to rent it. Your Pod's [volume storage](/pods/storage/types) remains on the physical machine, but the GPU slot becomes available.
If another user rents that GPU while your Pod is stopped, the GPU will be occupied when you try to restart. Because your Pod is still tied to that original machine, it cannot start with a GPU.
When this happens, Runpod gives you the option to start the Pod with zero GPUs. This is primarily a data recovery feature, allowing you to access your Pod's volume disk without access to the GPU.
## What are my options?
If you encounter this situation, you have three choices:
1. **Start with zero GPUs for data access**: Start the Pod without a GPU to access its local storage. This is useful for retrieving files, but the Pod will have limited CPU resources and is not suitable for compute tasks. You should use this option to back up or transfer your data before terminating the Pod.
2. **Wait and retry**: You can wait and try to restart the Pod again later. The GPU may become available if the other user stops their Pod, but there is no guarantee of when that will happen.
3. **Terminate and redeploy**: If you need a GPU immediately, terminate the current Pod and deploy a new one with the same configuration. The new Pod will be scheduled on any machine in the Runpod network with an available GPU of your chosen type.
## How do I prevent this?
The most effective way to avoid this issue is to use **[network volumes](/storage/network-volumes)**.
Network volumes decouple your data from a specific physical machine. Your `/workspace` data is stored on a separate, persistent volume that can be attached to any Pod. If you need to terminate a Pod, you can simply deploy a new one and attach the same network volume, giving you immediate access to your data on a new machine with an available GPU.
# Connect AI coding tools with Public Endpoints
Source: https://docs.runpod.io/public-endpoints/ai-coding-tools
Configure AI coding tools like OpenCode, Cursor, and Cline with Runpod Public Endpoints. Review setup and request guidance for Runpod Public Endpoints.
Runpod's [Public Endpoints](/public-endpoints/overview) provide OpenAI-compatible APIs that are compatible with most AI coding assistants. This page shows you how to configure OpenCode, Cursor, and Cline to use Runpod's Public Endpoints as a model provider.
## Requirements
Before you start, you'll need:
* A [Runpod account](/accounts-billing/manage-accounts) with an [API key](/get-started/api-keys), and at least \$5 in Runpod credits.
* One or more of the following AI coding tools installed on your local machine:
* [OpenCode](https://opencode.ai/): Terminal-based AI coding assistant.
* [Cursor](https://cursor.com/): AI-powered code editor.
* [Cline](https://marketplace.visualstudio.com/items?itemName=saoudrizwan.claude-dev): VS Code extension for AI-assisted coding.
## Available endpoints
Runpod provides two Public Endpoints that can be used to power AI coding tools:
| Model | Base URL | Model ID | Context window |
| ----------------------------------------------------- | -------------------------------------------------- | --------------------- | -------------- |
| [GPT OSS 120B](/public-endpoints/models/gpt-oss-120b) | `https://api.runpod.ai/v2/gpt-oss-120b/openai/v1` | `openai/gpt-oss-120b` | 131,072 tokens |
| [Qwen3 32B AWQ](/public-endpoints/models/qwen3-32b) | `https://api.runpod.ai/v2/qwen3-32b-awq/openai/v1` | `Qwen/Qwen3-32B-AWQ` | 32,768 tokens |
Both endpoints follow the OpenAI API specification, so they work with any tool that supports custom OpenAI-compatible providers.
Many AI coding tools can be configured to use any [OpenAI-compatible](/serverless/vllm/openai-compatibility) model API. You can build your own OpenAI-compatible endpoint with [Runpod Serverless](/serverless/overview).
## Configure OpenCode
OpenCode supports multiple provider configurations, so you can set up both Runpod endpoints and switch between them.
OpenCode looks for its config at `~/.config/opencode/opencode.json`. Run this command to create the directory (if it doesn't exist) and generate the config file:
```json theme={"theme":{"light":"github-light","dark":"github-dark"}}
mkdir -p ~/.config/opencode && cat << 'EOF' > ~/.config/opencode/opencode.json
{
"$schema": "https://opencode.ai/config.json",
"provider": {
"runpod-gpt": {
"npm": "@ai-sdk/openai-compatible",
"name": "Runpod GPT OSS 120B",
"options": {
"baseURL": "https://api.runpod.ai/v2/gpt-oss-120b/openai/v1",
"apiKey": "{env:RUNPOD_API_KEY}"
},
"models": {
"gpt-oss-120b": {
"id": "openai/gpt-oss-120b",
"name": "GPT OSS 120B (Runpod)",
"limit": { "context": 131072, "output": 4096 }
}
}
},
"runpod-qwen": {
"npm": "@ai-sdk/openai-compatible",
"name": "Runpod Qwen3",
"options": {
"baseURL": "https://api.runpod.ai/v2/qwen3-32b-awq/openai/v1",
"apiKey": "{env:RUNPOD_API_KEY}"
},
"models": {
"qwen3-32b": {
"id": "Qwen/Qwen3-32B-AWQ",
"name": "Qwen3 32B AWQ (Runpod)",
"limit": { "context": 32768, "output": 4096 }
}
}
}
}
}
EOF
```
The `{env:RUNPOD_API_KEY}` syntax in the config file tells OpenCode to read your [API key](/get-started/api-keys) from the `RUNPOD_API_KEY` environment variable.
Run this command to set the environment variable, replacing `rpa_YOUR_API_KEY` with your actual API key:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
export RUNPOD_API_KEY="rpa_YOUR_API_KEY"
```
This only sets the environment variable for your current shell session. You can add the export command to your shell profile (`~/.bashrc`, `~/.zshrc`, etc.) so you don't need to set the environment variable every time you open a new shell.
Run this command to check that the configuration is working:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
opencode models
```
You should see output similar to this:
```text highlight={7-8} theme={"theme":{"light":"github-light","dark":"github-dark"}}
opencode/big-pickle
opencode/glm-4.7-free
opencode/gpt-5-nano
opencode/kimi-k2.5-free
opencode/minimax-m2.1-free
opencode/trinity-large-preview-free
runpod-gpt/gpt-oss-120b
runpod-qwen/qwen3-32b
```
After confirming that the Runpod endpoints are listed, you can start OpenCode and try out the Runpod endpoints.
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
opencode
```
Press Ctrl + p to open the command palette and select **Switch model** to select a Runpod endpoint.
## Configure Cursor
Cursor supports a single global OpenAI-compatible endpoint override, so you can only use one Runpod endpoint at a time.
The Qwen3 32B AWQ endpoint is not compatible with Cursor.
Launch Cursor and press Shift + Cmd + J (macOS) or Shift + Ctrl + J (Windows/Linux) to open Settings.
Go to **Cursor Settings > Models** and expand the **API Keys** section.
Find the **OpenAI API Key** field. Enable it, then enter your Runpod API key (`rpa_...`).
Enable **Override OpenAI Base URL** and enter:
```text theme={"theme":{"light":"github-light","dark":"github-dark"}}
https://api.runpod.ai/v2/gpt-oss-120b/openai/v1
```
Scroll up and click **View All Models** to see the list of available models.
Click **Add Custom Model** and enter the model ID exactly as shown (case-sensitive):
```text theme={"theme":{"light":"github-light","dark":"github-dark"}}
openai/gpt-oss-120b
```
Select `openai/gpt-oss-120b` from the model list when using Cursor's AI features.
## Configure Cline
Cline is a VS Code extension with its own settings panel. Unlike Cursor, Cline supports multiple provider profiles, so you can configure both Runpod endpoints and switch between them.
Click the Cline icon in the sidebar to open the Cline panel, then click the gear icon to open Settings.
Set **API Provider** to **OpenAI Compatible**.
Fill in the following fields:
| Setting | Value |
| -------- | ------------------------------------------------- |
| Base URL | `https://api.runpod.ai/v2/gpt-oss-120b/openai/v1` |
| API Key | `rpa_YOUR_API_KEY` |
| Model ID | `openai/gpt-oss-120b` |
Click **Save** to apply your settings.
To use Qwen3 instead, use these values:
| Setting | Value |
| -------- | -------------------------------------------------- |
| Base URL | `https://api.runpod.ai/v2/qwen3-32b-awq/openai/v1` |
| Model ID | `Qwen/Qwen3-32B-AWQ` |
# Vercel AI SDK
Source: https://docs.runpod.io/public-endpoints/ai-sdk
Use the @runpod/ai-sdk-provider package to integrate Public Endpoints with the Vercel AI SDK. Review setup and request guidance for Runpod Public Endpoints.
The `@runpod/ai-sdk-provider` package integrates Runpod Public Endpoints with the [Vercel AI SDK](https://ai-sdk.dev/docs/introduction). This gives you a streamlined, type-safe interface for text generation, streaming, image generation, and video generation in JavaScript and TypeScript projects.
The Vercel AI SDK is a popular open-source library for building AI applications. By using the Runpod provider, you can access Runpod's Public Endpoints using the same patterns and APIs you'd use with other AI providers like OpenAI or Anthropic.
## Why use the Vercel AI SDK?
* **Unified interface**: Use the same `generateText`, `streamText`, `generateImage`, and `generateVideo` functions regardless of which AI provider you're using.
* **Type safety**: Full TypeScript support with typed responses and parameters.
* **Streaming built-in**: First-class support for streaming text responses.
* **Framework integrations**: Works seamlessly with Next.js, React, Svelte, and other frameworks.
* **Provider switching**: Easily switch between Runpod and other providers without rewriting your code.
## Installation
Install the Runpod provider alongside the Vercel AI SDK:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
npm install @runpod/ai-sdk-provider ai
```
## Configuration
### Default configuration
The provider reads your API key from the `RUNPOD_API_KEY` environment variable by default. Import the `runpod` instance and start using it immediately:
```typescript theme={"theme":{"light":"github-light","dark":"github-dark"}}
import { runpod } from "@runpod/ai-sdk-provider";
```
Set the environment variable in your shell or `.env` file:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
export RUNPOD_API_KEY="YOUR_API_KEY"
```
### Custom configuration
For more control, use `createRunpod` to create a custom provider instance:
```typescript theme={"theme":{"light":"github-light","dark":"github-dark"}}
import { createRunpod } from "@runpod/ai-sdk-provider";
const runpod = createRunpod({
apiKey: "YOUR_API_KEY",
baseURL: "https://api.runpod.ai/v2",
headers: {
"X-Custom-Header": "value",
},
});
```
| Option | Description | Default |
| --------- | -------------------------------------------- | -------------------------- |
| `apiKey` | Your Runpod API key | `RUNPOD_API_KEY` env var |
| `baseURL` | Base URL for API requests | `https://api.runpod.ai/v2` |
| `headers` | Custom HTTP headers to include with requests | `{}` |
## Using custom endpoints
You can use your own [Serverless endpoints](/serverless/overview) with the AI SDK. This is useful when you've deployed a custom model or want to use a specific endpoint you've created.
### Using endpoint IDs
Pass your Serverless endpoint ID directly as the model identifier:
```typescript theme={"theme":{"light":"github-light","dark":"github-dark"}}
import { runpod } from "@runpod/ai-sdk-provider";
import { generateText, experimental_generateImage as generateImage } from "ai";
// Use a custom chat endpoint
const { text } = await generateText({
model: runpod("your-endpoint-id"),
prompt: "Hello, how are you?",
});
// Use a custom image endpoint
const { image } = await generateImage({
model: runpod.image("your-image-endpoint-id"),
prompt: "A beautiful sunset",
});
```
The SDK resolves your endpoint ID to `https://api.runpod.ai/v2/{endpointId}` automatically.
### Using Console URLs
Copy an endpoint URL directly from the Runpod Console and use it as the model identifier:
```typescript theme={"theme":{"light":"github-light","dark":"github-dark"}}
import { runpod } from "@runpod/ai-sdk-provider";
import { experimental_generateImage as generateImage } from "ai";
const { image } = await generateImage({
model: runpod.image("https://console.runpod.io/serverless/user/endpoint/abc123xyz"),
prompt: "A serene mountain landscape",
});
```
The SDK extracts the endpoint ID from the Console URL and routes requests to your endpoint.
## Text generation
### Basic text generation
Use `generateText` to generate text from a prompt:
```typescript theme={"theme":{"light":"github-light","dark":"github-dark"}}
import { runpod } from "@runpod/ai-sdk-provider";
import { generateText } from "ai";
const { text, finishReason, usage } = await generateText({
model: runpod("qwen3-32b-awq"),
prompt: "Write a Python function that checks if a number is prime:",
});
console.log(text);
console.log(`Tokens used: ${usage.totalTokens}`);
```
The response includes:
* `text`: The generated text
* `finishReason`: Why generation stopped (`stop`, `length`, etc.)
* `usage`: Token counts (`promptTokens`, `completionTokens`, `totalTokens`)
### Chat conversations
For multi-turn conversations, pass a `messages` array instead of a `prompt`:
```typescript theme={"theme":{"light":"github-light","dark":"github-dark"}}
import { runpod } from "@runpod/ai-sdk-provider";
import { generateText } from "ai";
const { text } = await generateText({
model: runpod("qwen3-32b-awq"),
messages: [
{
role: "system",
content: "You are a helpful coding assistant. Be concise.",
},
{
role: "user",
content: "How do I read a JSON file in Python?",
},
],
});
console.log(text);
```
### Generation parameters
Control the generation behavior with additional parameters:
```typescript theme={"theme":{"light":"github-light","dark":"github-dark"}}
const { text } = await generateText({
model: runpod("qwen3-32b-awq"),
prompt: "Write a creative story about a robot:",
temperature: 0.8, // Higher = more creative (0-1)
maxTokens: 500, // Maximum tokens to generate
topP: 0.9, // Nucleus sampling threshold
});
```
## Streaming
For real-time output (useful for chat interfaces), use `streamText`:
```typescript theme={"theme":{"light":"github-light","dark":"github-dark"}}
import { runpod } from "@runpod/ai-sdk-provider";
import { streamText } from "ai";
const { textStream } = await streamText({
model: runpod("qwen3-32b-awq"),
prompt: "Explain quantum computing in simple terms:",
temperature: 0.7,
});
for await (const chunk of textStream) {
process.stdout.write(chunk);
}
```
### Streaming with callbacks
You can also use callbacks to handle streaming events:
```typescript theme={"theme":{"light":"github-light","dark":"github-dark"}}
import { runpod } from "@runpod/ai-sdk-provider";
import { streamText } from "ai";
const result = await streamText({
model: runpod("qwen3-32b-awq"),
prompt: "Write a poem about the ocean:",
onChunk: ({ chunk }) => {
if (chunk.type === "text-delta") {
process.stdout.write(chunk.textDelta);
}
},
onFinish: ({ text, usage }) => {
console.log(`\n\nTotal tokens: ${usage.totalTokens}`);
},
});
```
## Image generation
### Text-to-image
Generate images using models like [Flux](/public-endpoints/models/flux-dev):
```typescript theme={"theme":{"light":"github-light","dark":"github-dark"}}
import { runpod } from "@runpod/ai-sdk-provider";
import { experimental_generateImage as generateImage } from "ai";
import { writeFileSync } from "fs";
const { image } = await generateImage({
model: runpod.image("black-forest-labs-flux-1-dev"),
prompt: "A serene mountain landscape at sunset, photorealistic",
aspectRatio: "16:9",
});
// Save the image to a file
writeFileSync("output.png", image.uint8Array);
// Or access as base64
console.log(image.base64);
```
The response includes:
* `image.uint8Array`: Binary image data
* `image.base64`: Base64-encoded image
* `image.mimeType`: Image MIME type (e.g., `image/png`)
### Image editing
Edit existing images by providing reference images:
```typescript theme={"theme":{"light":"github-light","dark":"github-dark"}}
import { runpod } from "@runpod/ai-sdk-provider";
import { experimental_generateImage as generateImage } from "ai";
const { image } = await generateImage({
model: runpod.image("google-nano-banana-edit"),
prompt: {
text: "Add modern Scandinavian furniture to this room",
images: ["https://example.com/empty-room.png"],
},
aspectRatio: "16:9",
});
```
For models that support multiple reference images:
```typescript theme={"theme":{"light":"github-light","dark":"github-dark"}}
const { image } = await generateImage({
model: runpod.image("google-nano-banana-edit"),
prompt: {
text: "Combine these into an epic band photo",
images: [
"https://example.com/drummer.png",
"https://example.com/guitarist.png",
"https://example.com/bassist.png",
"https://example.com/singer.png",
],
},
});
```
### Provider options
Pass model-specific parameters using `providerOptions`:
```typescript theme={"theme":{"light":"github-light","dark":"github-dark"}}
const { image } = await generateImage({
model: runpod.image("black-forest-labs-flux-1-dev"),
prompt: "A sunset over the ocean",
providerOptions: {
runpod: {
negative_prompt: "blurry, low quality, distorted",
num_inference_steps: 30,
guidance: 7.5,
seed: 42,
enable_safety_checker: true,
},
},
});
```
| Option | Description |
| ----------------------- | ------------------------------------------------ |
| `negative_prompt` | Elements to exclude from the image |
| `num_inference_steps` | Number of denoising steps (higher = more detail) |
| `guidance` | How closely to follow the prompt (0-10) |
| `seed` | Seed for reproducible results (-1 for random) |
| `enable_safety_checker` | Enable content safety filtering |
| `maxPollAttempts` | Max polling attempts for async generation |
| `pollIntervalMillis` | Milliseconds between status polls |
## Video generation
Use `experimental_generateVideo` to generate videos from text prompts or images. The Runpod provider supports 15 video models, including Sora, Wan, Seedance, and Kling.
Video generation is asynchronous—the SDK submits a job, polls for completion, and returns the video URL when ready.
### Text-to-video
Generate videos from text prompts:
```typescript theme={"theme":{"light":"github-light","dark":"github-dark"}}
import { runpod } from "@runpod/ai-sdk-provider";
import { experimental_generateVideo as generateVideo } from "ai";
const { video } = await generateVideo({
model: runpod.video("alibaba/wan-2.6-t2v"),
prompt: "A golden retriever running on a sunny beach, cinematic, 4k",
});
console.log(video.url);
```
The response includes:
* `video.url`: URL to the generated video
* `video.mediaType`: Video MIME type (`video/mp4`)
### Image-to-video
Animate an existing image:
```typescript theme={"theme":{"light":"github-light","dark":"github-dark"}}
import { runpod } from "@runpod/ai-sdk-provider";
import { experimental_generateVideo as generateVideo } from "ai";
const { video } = await generateVideo({
model: runpod.video("alibaba/wan-2.6-i2v"),
prompt: "Animate this scene with gentle camera movement",
image: new URL("https://example.com/image.png"),
});
console.log(video.url);
```
### Video generation parameters
Control the video generation with additional parameters:
```typescript theme={"theme":{"light":"github-light","dark":"github-dark"}}
const { video } = await generateVideo({
model: runpod.video("alibaba/wan-2.6-t2v"),
prompt: "A serene mountain landscape with flowing water",
duration: 5,
aspectRatio: "16:9",
seed: 42,
});
```
### Video provider options
Pass model-specific parameters using `providerOptions`:
```typescript theme={"theme":{"light":"github-light","dark":"github-dark"}}
const { video } = await generateVideo({
model: runpod.video("alibaba/wan-2.6-t2v"),
prompt: "A serene mountain landscape with flowing water",
duration: 5,
aspectRatio: "16:9",
providerOptions: {
runpod: {
negative_prompt: "blurry, low quality",
guidance_scale: 7.5,
},
},
});
```
| Option | Description |
| --------------------- | ------------------------------------------------- |
| `negative_prompt` | Elements to exclude from the video |
| `guidance_scale` | How closely to follow the prompt |
| `num_inference_steps` | Number of inference steps |
| `style` | Style preset (model-specific) |
| `maxPollAttempts` | Max polling attempts (default: 120) |
| `pollIntervalMillis` | Milliseconds between status polls (default: 5000) |
## Supported models
### Text models
| Model ID | Description |
| --------------- | ----------------------------------------------------------------------------------------------------------------- |
| `qwen3-32b-awq` | [Qwen3 32B](/public-endpoints/models/qwen3-32b) with AWQ quantization. Good for general text and code generation. |
| `gpt-oss-120b` | [GPT OSS 120B](/public-endpoints/models/gpt-oss-120b). Supports tool calling. |
### Image models
| Model ID | Description |
| ---------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------- |
| `black-forest-labs-flux-1-dev` | [Flux Dev](/public-endpoints/models/flux-dev). High quality, detailed images. |
| `black-forest-labs-flux-1-schnell` | [Flux Schnell](/public-endpoints/models/flux-schnell). Fast generation, good for prototyping. |
| `google-nano-banana-edit` | [Nano Banana Edit](/public-endpoints/models/nano-banana-edit). Supports multiple reference images. |
| `google/nano-banana-2-edit` | [Nano Banana 2 Edit](/public-endpoints/models/nano-banana-2-edit). Image editing with 14 aspect ratios and resolution options (1k/2k/4k). |
| `bytedance-seedream-4-0-t2i` | [Seedream 4.0](/public-endpoints/models/seedream-4-t2i). Text-to-image with good prompt adherence. |
| `tongyi-mai/z-image-turbo` | [Z-Image Turbo](/public-endpoints/models/z-image-turbo). Fast 6B parameter model with text-to-image support. |
### Video models
| Model ID | Type | Resolution | Aspect Ratios | Duration |
| --------------------------------------- | ----------- | ----------------- | ------------------------------- | ---------- |
| `pruna/p-video` | t2v | 720p, 1080p | 16:9, 9:16 | 5s |
| `vidu/q3-t2v` | t2v | 720p, 1080p | 16:9, 9:16, 1:1 | 5, 10s |
| `vidu/q3-i2v` | i2v | 720p, 1080p | 16:9, 9:16, 1:1 | 5, 10s |
| `kwaivgi/kling-v2.6-std-motion-control` | i2v + video | 720p | 16:9, 9:16, 1:1 | 5, 10s |
| `kwaivgi/kling-video-o1-r2v` | i2v | 720p | 16:9, 9:16, 1:1 | 3–10s |
| `kwaivgi/kling-v2.1-i2v-pro` | i2v | 720p | 16:9, 9:16, 1:1 | 5, 10s |
| `alibaba/wan-2.6-t2v` | t2v | 720p, 1080p | 16:9, 9:16 | 5, 10, 15s |
| `alibaba/wan-2.6-i2v` | i2v | 720p, 1080p | 16:9, 9:16 | 5, 10, 15s |
| `alibaba/wan-2.5` | i2v | 480p, 720p, 1080p | 16:9, 9:16 | 5, 10s |
| `alibaba/wan-2.2-t2v-720-lora` | i2v | 720p | 16:9 | 5, 8s |
| `alibaba/wan-2.2-i2v-720` | i2v | 720p | 16:9 | 5, 8s |
| `alibaba/wan-2.1-i2v-720` | i2v | 720p | 16:9 | 5s |
| `bytedance/seedance-v1.5-pro-i2v` | i2v | 480p, 720p | 21:9, 16:9, 9:16, 1:1, 4:3, 3:4 | 4–12s |
| `openai/sora-2-pro-i2v` | i2v | 720p, 1080p | 16:9, 9:16, 1:1 | 4, 8, 12s |
| `openai/sora-2-i2v` | i2v | 720p, 1080p | 16:9, 9:16, 1:1 | 4, 8, 12s |
For a complete list of available models and their parameters, see the [model reference](/public-endpoints/reference).
## Example: Chat application
Here's a complete example of a simple chat application using streaming:
```typescript theme={"theme":{"light":"github-light","dark":"github-dark"}}
import { runpod } from "@runpod/ai-sdk-provider";
import { streamText } from "ai";
import * as readline from "readline";
const rl = readline.createInterface({
input: process.stdin,
output: process.stdout,
});
const messages: { role: "user" | "assistant"; content: string }[] = [];
async function chat(userMessage: string) {
messages.push({ role: "user", content: userMessage });
const { textStream } = await streamText({
model: runpod("qwen3-32b-awq"),
system: "You are a helpful assistant.",
messages,
});
let assistantMessage = "";
process.stdout.write("\nAssistant: ");
for await (const chunk of textStream) {
process.stdout.write(chunk);
assistantMessage += chunk;
}
messages.push({ role: "assistant", content: assistantMessage });
console.log("\n");
}
function prompt() {
rl.question("You: ", async (input) => {
if (input.toLowerCase() === "exit") {
rl.close();
return;
}
await chat(input);
prompt();
});
}
console.log('Chat started. Type "exit" to quit.\n');
prompt();
```
## Next steps
* [Model reference](/public-endpoints/reference): View all available models and their parameters.
* [Make API requests](/public-endpoints/requests): Learn about the REST API for lower-level control.
* [@runpod/ai-sdk-provider on GitHub](https://github.com/runpod/ai-sdk-provider): View the source code and contribute.
* [Vercel AI SDK documentation](https://ai-sdk.dev/docs/introduction): Learn more about the AI SDK.
# Chatterbox Turbo
Source: https://docs.runpod.io/public-endpoints/models/chatterbox-turbo
Fast open-source text-to-speech with expressive voice cloning and paralinguistic tags. See model inputs and outputs on Runpod Public Endpoints.
Chatterbox Turbo is Resemble AI's fastest open-source text-to-speech model with paralinguistic tags for non-speech sounds and expressive voice cloning capabilities. It supports multiple preset voices and custom voice cloning via audio URL.
Test Chatterbox Turbo in the Runpod Hub playground.
| | |
| ------------ | --------------------------------------------------- |
| **Endpoint** | `https://api.runpod.ai/v2/chatterbox-turbo/runsync` |
| **Pricing** | \$0.001 per second |
| **Type** | Text-to-speech |
## Request
All parameters are passed within the `input` object in the request body.
Text to convert to speech.
Preset voice ID. Options: `aaron`, `abigail`, `anaya`, `andy`, `archer`, `brian`, `chloe`, `dylan`, `emmanuel`, `ethan`, `evelyn`, `gavin`, `gordon`, `ivan`, `laura`, `lucy`, `madison`, `marisol`, `meera`, `walter`.
URL of an audio file for voice cloning. Overrides the `voice` parameter.
Output audio format. Options: `wav`, `flac`, `ogg`.
```bash cURL theme={"theme":{"light":"github-light","dark":"github-dark"}}
curl -X POST "https://api.runpod.ai/v2/chatterbox-turbo/runsync" \
-H "Authorization: Bearer $RUNPOD_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"input": {
"prompt": "Hello! Welcome to our text-to-speech demonstration.",
"voice": "lucy",
"format": "wav"
}
}'
```
```python Python theme={"theme":{"light":"github-light","dark":"github-dark"}}
import requests
response = requests.post(
"https://api.runpod.ai/v2/chatterbox-turbo/runsync",
headers={
"Authorization": f"Bearer {RUNPOD_API_KEY}",
"Content-Type": "application/json",
},
json={
"input": {
"prompt": "Hello! Welcome to our text-to-speech demonstration.",
"voice": "lucy",
"format": "wav",
}
},
)
result = response.json()
print(result["output"]["audio_url"])
```
```javascript JavaScript theme={"theme":{"light":"github-light","dark":"github-dark"}}
const response = await fetch(
"https://api.runpod.ai/v2/chatterbox-turbo/runsync",
{
method: "POST",
headers: {
Authorization: `Bearer ${RUNPOD_API_KEY}`,
"Content-Type": "application/json",
},
body: JSON.stringify({
input: {
prompt: "Hello! Welcome to our text-to-speech demonstration.",
voice: "lucy",
format: "wav",
},
}),
}
);
const result = await response.json();
console.log(result.output.audio_url);
```
## Response
Unique identifier for the request.
Request status. Returns `COMPLETED` on success, `FAILED` on error.
The generation result containing the audio URL.
URL of the generated audio file. This URL expires after 7 days.
```json 200 theme={"theme":{"light":"github-light","dark":"github-dark"}}
{
"id": "sync-a1b2c3d4-e5f6-7890-abcd-ef1234567890-u1",
"status": "COMPLETED",
"delayTime": 10,
"executionTime": 1856,
"output": {
"audio_url": "https://audio.runpod.ai/abc123/output.wav"
}
}
```
Audio URLs expire after 7 days. Download and store generated audio files immediately if you need to keep them.
## Available voices
Chatterbox Turbo includes 20 preset voices:
| Voice | Voice | Voice | Voice |
| -------- | ------- | ------ | ------ |
| aaron | abigail | anaya | andy |
| archer | brian | chloe | dylan |
| emmanuel | ethan | evelyn | gavin |
| gordon | ivan | laura | lucy |
| madison | marisol | meera | walter |
## Cost calculation
Chatterbox Turbo charges \$0.001 per second of generated audio.
# Cogito 671B v2.1
Source: https://docs.runpod.io/public-endpoints/models/cogito-671b
Deep Cogito's 671B parameter Mixture-of-Experts model with FP8 dynamic quantization for efficient inference.
Cogito 671B v2.1 is Deep Cogito's massive 671B parameter Mixture-of-Experts (MoE) language model. It uses FP8 dynamic quantization for efficient inference while maintaining high-quality outputs across reasoning, coding, and general knowledge tasks.
Test Cogito 671B v2.1 in the Runpod Hub playground.
| | |
| ------------ | --------------------------------------------------------------- |
| **Endpoint** | `https://api.runpod.ai/v2/cogito-671b-v2-1-fp8-dynamic/runsync` |
| **Pricing** | \$0.50 per 1M tokens |
| **Type** | Text generation |
This endpoint is fully compatible with the OpenAI API. See the [OpenAI compatibility examples](#openai-api-compatibility) below.
## Request
All parameters are passed within the `input` object in the request body.
Prompt for text generation.
Maximum number of tokens to output.
Randomness of the output. Lower values make output more predictable and deterministic. Range: 0.0-1.0.
Nucleus sampling threshold. Samples from the smallest set of words whose cumulative probability exceeds this threshold.
Restricts sampling to the top K most probable words.
Stops generation if the given string is encountered.
```bash cURL theme={"theme":{"light":"github-light","dark":"github-dark"}}
curl -X POST "https://api.runpod.ai/v2/cogito-671b-v2-1-fp8-dynamic/runsync" \
-H "Authorization: Bearer $RUNPOD_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"input": {
"prompt": "Write a detailed analysis of the economic impacts of renewable energy adoption:",
"max_tokens": 1024,
"temperature": 0.7
}
}'
```
```python Python theme={"theme":{"light":"github-light","dark":"github-dark"}}
import requests
response = requests.post(
"https://api.runpod.ai/v2/cogito-671b-v2-1-fp8-dynamic/runsync",
headers={
"Authorization": f"Bearer {RUNPOD_API_KEY}",
"Content-Type": "application/json",
},
json={
"input": {
"prompt": "Write a detailed analysis of the economic impacts of renewable energy adoption:",
"max_tokens": 1024,
"temperature": 0.7,
}
},
)
result = response.json()
print(result["output"])
```
```javascript JavaScript theme={"theme":{"light":"github-light","dark":"github-dark"}}
const response = await fetch(
"https://api.runpod.ai/v2/cogito-671b-v2-1-fp8-dynamic/runsync",
{
method: "POST",
headers: {
Authorization: `Bearer ${RUNPOD_API_KEY}`,
"Content-Type": "application/json",
},
body: JSON.stringify({
input: {
prompt: "Write a detailed analysis of the economic impacts of renewable energy adoption:",
max_tokens: 1024,
temperature: 0.7,
},
}),
}
);
const result = await response.json();
console.log(result.output);
```
## Response
Unique identifier for the request.
Request status. Returns `COMPLETED` on success, `FAILED` on error.
Time in milliseconds the request spent in queue before processing began.
Time in milliseconds the model took to generate the response.
The generation result containing the text and usage information.
Array containing the generated text.
Cost of the generation in USD.
Token usage information with `input` and `output` counts.
```json 200 theme={"theme":{"light":"github-light","dark":"github-dark"}}
{
"delayTime": 45,
"executionTime": 8234,
"id": "sync-a1b2c3d4-e5f6-7890-abcd-ef1234567890-u1",
"output": [
{
"choices": [
{
"tokens": [
"The economic impacts of renewable energy adoption are multifaceted and far-reaching. Here's a comprehensive analysis:\n\n1. Job Creation and Labor Markets..."
]
}
],
"cost": 0.0005,
"usage": {
"input": 20,
"output": 980
}
}
],
"status": "COMPLETED"
}
```
## OpenAI API compatibility
Cogito 671B v2.1 is fully compatible with the OpenAI API format. You can use the OpenAI Python client to interact with this endpoint.
```python Python (OpenAI SDK) theme={"theme":{"light":"github-light","dark":"github-dark"}}
from openai import OpenAI
client = OpenAI(
api_key=RUNPOD_API_KEY,
base_url="https://api.runpod.ai/v2/cogito-671b-v2-1-fp8-dynamic/openai/v1",
)
response = client.chat.completions.create(
model="cogito-671b-v2-1-fp8-dynamic",
messages=[
{
"role": "system",
"content": "You are a helpful assistant with expertise in economics and analysis.",
},
{
"role": "user",
"content": "Analyze the economic impacts of renewable energy adoption.",
},
],
max_tokens=1024,
)
print(response.choices[0].message.content)
```
For streaming responses, add `stream=True`:
```python Python (Streaming) theme={"theme":{"light":"github-light","dark":"github-dark"}}
response = client.chat.completions.create(
model="cogito-671b-v2-1-fp8-dynamic",
messages=[
{"role": "user", "content": "Explain the principles of machine learning."}
],
max_tokens=1024,
stream=True,
)
for chunk in response:
if chunk.choices[0].delta.content:
print(chunk.choices[0].delta.content, end="")
```
For more details, see [Send vLLM requests](/serverless/vllm/vllm-requests) and the [OpenAI API compatibility guide](/serverless/vllm/openai-compatibility).
## Cost calculation
Cogito 671B v2.1 charges \$0.50 per 1M tokens. Example costs:
| Tokens | Cost |
| ---------------- | -------- |
| 1,000 tokens | \$0.0005 |
| 10,000 tokens | \$0.005 |
| 100,000 tokens | \$0.05 |
| 1,000,000 tokens | \$0.50 |
# Flux Dev
Source: https://docs.runpod.io/public-endpoints/models/flux-dev
High-quality image generation with exceptional prompt adherence and rich detail. See model inputs and outputs on Runpod Public Endpoints.
Flux Dev is Black Forest Labs' flagship image generation model, optimized for high visual fidelity and detailed outputs. It offers exceptional prompt adherence and produces images with rich detail, making it ideal for production use cases.
Test Flux Dev in the Runpod Hub playground.
| | |
| ------------ | --------------------------------------------------------------- |
| **Endpoint** | `https://api.runpod.ai/v2/black-forest-labs-flux-1-dev/runsync` |
| **Pricing** | \$0.02 per megapixel |
| **Type** | Image generation |
For faster, cheaper generation during development, consider [Flux Schnell](/public-endpoints/models/flux-schnell) which is optimized for speed.
## Request
All parameters are passed within the `input` object in the request body.
Text description of the desired image. Be specific and detailed for best results.
Elements to exclude from the generated image. Use this to prevent unwanted features or styles.
Image width in pixels. Must be divisible by 64. Range: 256-1536.
Image height in pixels. Must be divisible by 64. Range: 256-1536.
Number of denoising steps. Higher values produce more detailed images but take longer. Range: 1-50.
How closely to follow the prompt. Higher values produce images more faithful to the prompt but may reduce creativity. Range: 0.0-10.0.
Seed for reproducible results. Use the same seed with identical parameters to generate the same image. Set to -1 for random.
Output image format. Accepts `png` or `jpeg`.
```bash cURL theme={"theme":{"light":"github-light","dark":"github-dark"}}
curl -X POST "https://api.runpod.ai/v2/black-forest-labs-flux-1-dev/runsync" \
-H "Authorization: Bearer $RUNPOD_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"input": {
"prompt": "A serene mountain landscape at sunset, golden light filtering through clouds, photorealistic",
"negative_prompt": "blurry, low quality",
"width": 1024,
"height": 1024,
"num_inference_steps": 28,
"guidance": 7.5,
"seed": 42,
"image_format": "png"
}
}'
```
```python Python theme={"theme":{"light":"github-light","dark":"github-dark"}}
import requests
response = requests.post(
"https://api.runpod.ai/v2/black-forest-labs-flux-1-dev/runsync",
headers={
"Authorization": f"Bearer {RUNPOD_API_KEY}",
"Content-Type": "application/json",
},
json={
"input": {
"prompt": "A serene mountain landscape at sunset, golden light filtering through clouds, photorealistic",
"negative_prompt": "blurry, low quality",
"width": 1024,
"height": 1024,
"num_inference_steps": 28,
"guidance": 7.5,
"seed": 42,
"image_format": "png",
}
},
)
result = response.json()
print(result["output"]["image_url"])
```
```javascript JavaScript theme={"theme":{"light":"github-light","dark":"github-dark"}}
const response = await fetch(
"https://api.runpod.ai/v2/black-forest-labs-flux-1-dev/runsync",
{
method: "POST",
headers: {
"Authorization": `Bearer ${RUNPOD_API_KEY}`,
"Content-Type": "application/json",
},
body: JSON.stringify({
input: {
prompt: "A serene mountain landscape at sunset, golden light filtering through clouds, photorealistic",
negative_prompt: "blurry, low quality",
width: 1024,
height: 1024,
num_inference_steps: 28,
guidance: 7.5,
seed: 42,
image_format: "png",
},
}),
}
);
const result = await response.json();
console.log(result.output.image_url);
```
## Response
Unique identifier for the request.
Request status. Returns `COMPLETED` on success, `FAILED` on error.
Time in milliseconds the request spent in queue before processing began.
Time in milliseconds the model took to generate the image.
Identifier of the worker that processed the request.
The generation result containing the image URL and cost.
URL of the generated image. This URL expires after 7 days.
Cost of the generation in USD, calculated based on the output megapixels.
```json 200 theme={"theme":{"light":"github-light","dark":"github-dark"}}
{
"id": "sync-a1b2c3d4-e5f6-7890-abcd-ef1234567890-u1",
"status": "COMPLETED",
"delayTime": 17,
"executionTime": 3986,
"workerId": "oqk7ao1uomckye",
"output": {
"image_url": "https://image.runpod.ai/abc123/output.png",
"cost": 0.02097152
}
}
```
```json 400 theme={"theme":{"light":"github-light","dark":"github-dark"}}
{
"id": "sync-a1b2c3d4-e5f6-7890-abcd-ef1234567890-u1",
"status": "FAILED",
"error": "Invalid parameter: width must be divisible by 64"
}
```
Image URLs expire after 7 days. Download and store generated images immediately if you need to keep them.
## Cost calculation
Flux Dev charges \$0.02 per megapixel. Cost is calculated as: `(width × height / 1,000,000) × \$0.02`.
| Image size | Megapixels | Cost |
| ---------- | ---------- | -------- |
| 512×512 | 0.26 MP | \$0.0052 |
| 1024×1024 | 1.05 MP | \$0.021 |
| 1536×1536 | 2.36 MP | \$0.047 |
# Flux Kontext Dev
Source: https://docs.runpod.io/public-endpoints/models/flux-kontext-dev
12 billion parameter model for editing images based on text instructions. Explore this model's inputs and outputs on Runpod Public Endpoints.
Flux Kontext Dev is a 12 billion parameter rectified flow transformer capable of editing images based on text instructions. It excels at making targeted edits to existing images while preserving the overall context and style.
Test Flux Kontext Dev in the Runpod Hub playground.
| | |
| ------------ | ----------------------------------------------------------------------- |
| **Endpoint** | `https://api.runpod.ai/v2/black-forest-labs-flux-1-kontext-dev/runsync` |
| **Pricing** | \$0.025 per image |
| **Type** | Image editing |
## Request
All parameters are passed within the `input` object in the request body.
Text instructions describing the desired edits to the image.
URL of the input image to edit.
Elements to exclude from the edited image.
Output image size in format "width\*height".
Number of denoising steps. Higher values produce more detailed edits. Range: 1-50.
How closely to follow the prompt. Range: 0.0-10.0.
Seed for reproducible results. Set to -1 for random.
Output image format. Accepts `png` or `jpeg`.
Whether to run safety checks on the output.
```bash cURL theme={"theme":{"light":"github-light","dark":"github-dark"}}
curl -X POST "https://api.runpod.ai/v2/black-forest-labs-flux-1-kontext-dev/runsync" \
-H "Authorization: Bearer $RUNPOD_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"input": {
"prompt": "Exact same bluebird, same angle and posture, now perched on a cloud in outer space",
"image": "https://example.com/bluebird.png",
"negative_prompt": "",
"seed": -1,
"num_inference_steps": 28,
"guidance": 2,
"size": "1024*1024",
"output_format": "png",
"enable_safety_checker": true
}
}'
```
```python Python theme={"theme":{"light":"github-light","dark":"github-dark"}}
import requests
response = requests.post(
"https://api.runpod.ai/v2/black-forest-labs-flux-1-kontext-dev/runsync",
headers={
"Authorization": f"Bearer {RUNPOD_API_KEY}",
"Content-Type": "application/json",
},
json={
"input": {
"prompt": "Exact same bluebird, same angle and posture, now perched on a cloud in outer space",
"image": "https://example.com/bluebird.png",
"negative_prompt": "",
"seed": -1,
"num_inference_steps": 28,
"guidance": 2,
"size": "1024*1024",
"output_format": "png",
"enable_safety_checker": True,
}
},
)
result = response.json()
print(result["output"]["image_url"])
```
```javascript JavaScript theme={"theme":{"light":"github-light","dark":"github-dark"}}
const response = await fetch(
"https://api.runpod.ai/v2/black-forest-labs-flux-1-kontext-dev/runsync",
{
method: "POST",
headers: {
Authorization: `Bearer ${RUNPOD_API_KEY}`,
"Content-Type": "application/json",
},
body: JSON.stringify({
input: {
prompt:
"Exact same bluebird, same angle and posture, now perched on a cloud in outer space",
image: "https://example.com/bluebird.png",
negative_prompt: "",
seed: -1,
num_inference_steps: 28,
guidance: 2,
size: "1024*1024",
output_format: "png",
enable_safety_checker: true,
},
}),
}
);
const result = await response.json();
console.log(result.output.image_url);
```
## Response
Unique identifier for the request.
Request status. Returns `COMPLETED` on success, `FAILED` on error.
Time in milliseconds the request spent in queue before processing began.
Time in milliseconds the model took to edit the image.
Identifier of the worker that processed the request.
The generation result containing the image URL and cost.
URL of the edited image. This URL expires after 7 days.
Cost of the generation in USD, calculated based on the output megapixels.
```json 200 theme={"theme":{"light":"github-light","dark":"github-dark"}}
{
"id": "sync-a1b2c3d4-e5f6-7890-abcd-ef1234567890-u1",
"status": "COMPLETED",
"delayTime": 18,
"executionTime": 4521,
"workerId": "oqk7ao1uomckye",
"output": {
"image_url": "https://image.runpod.ai/abc123/output.png",
"cost": 0.031457
}
}
```
```json 400 theme={"theme":{"light":"github-light","dark":"github-dark"}}
{
"id": "sync-a1b2c3d4-e5f6-7890-abcd-ef1234567890-u1",
"status": "FAILED",
"error": "Invalid image URL"
}
```
Image URLs expire after 7 days. Download and store edited images immediately if you need to keep them.
## Cost calculation
Flux Kontext Dev charges \$0.025 per image edited.
# Flux Schnell
Source: https://docs.runpod.io/public-endpoints/models/flux-schnell
Fast, lightweight image generation optimized for speed and prototyping. Explore this model's inputs and outputs on Runpod Public Endpoints.
Flux Schnell is Black Forest Labs' fastest and most lightweight FLUX model, ideal for local development, prototyping, and personal use. It generates images quickly with lower step counts while maintaining good quality.
Test Flux Schnell in the Runpod Hub playground.
| | |
| ------------ | ------------------------------------------------------------------- |
| **Endpoint** | `https://api.runpod.ai/v2/black-forest-labs-flux-1-schnell/runsync` |
| **Pricing** | \$0.0024 per megapixel |
| **Type** | Image generation |
Flux Schnell is optimized for speed and works best with lower step counts. For higher fidelity production images, consider [Flux Dev](/public-endpoints/models/flux-dev).
## Request
All parameters are passed within the `input` object in the request body.
Text description of the desired image. Be specific and detailed for best results.
Elements to exclude from the generated image. Use this to prevent unwanted features or styles.
Image width in pixels. Must be divisible by 64. Range: 256-1536.
Image height in pixels. Must be divisible by 64. Range: 256-1536.
Number of denoising steps. Schnell is optimized for low step counts. Range: 1-8.
How closely to follow the prompt. Higher values produce images more faithful to the prompt. Range: 0.0-10.0.
Seed for reproducible results. Use the same seed with identical parameters to generate the same image. Set to -1 for random.
Output image format. Accepts `png` or `jpeg`.
```bash cURL theme={"theme":{"light":"github-light","dark":"github-dark"}}
curl -X POST "https://api.runpod.ai/v2/black-forest-labs-flux-1-schnell/runsync" \
-H "Authorization: Bearer $RUNPOD_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"input": {
"prompt": "A quick sketch of a mountain landscape",
"width": 1024,
"height": 1024,
"num_inference_steps": 4,
"guidance": 1.0,
"seed": 123
}
}'
```
```python Python theme={"theme":{"light":"github-light","dark":"github-dark"}}
import requests
response = requests.post(
"https://api.runpod.ai/v2/black-forest-labs-flux-1-schnell/runsync",
headers={
"Authorization": f"Bearer {RUNPOD_API_KEY}",
"Content-Type": "application/json",
},
json={
"input": {
"prompt": "A quick sketch of a mountain landscape",
"width": 1024,
"height": 1024,
"num_inference_steps": 4,
"guidance": 1.0,
"seed": 123,
}
},
)
result = response.json()
print(result["output"]["image_url"])
```
```javascript JavaScript theme={"theme":{"light":"github-light","dark":"github-dark"}}
const response = await fetch(
"https://api.runpod.ai/v2/black-forest-labs-flux-1-schnell/runsync",
{
method: "POST",
headers: {
Authorization: `Bearer ${RUNPOD_API_KEY}`,
"Content-Type": "application/json",
},
body: JSON.stringify({
input: {
prompt: "A quick sketch of a mountain landscape",
width: 1024,
height: 1024,
num_inference_steps: 4,
guidance: 1.0,
seed: 123,
},
}),
}
);
const result = await response.json();
console.log(result.output.image_url);
```
## Response
Unique identifier for the request.
Request status. Returns `COMPLETED` on success, `FAILED` on error.
Time in milliseconds the request spent in queue before processing began.
Time in milliseconds the model took to generate the image.
Identifier of the worker that processed the request.
The generation result containing the image URL and cost.
URL of the generated image. This URL expires after 7 days.
Cost of the generation in USD, calculated based on the output megapixels.
```json 200 theme={"theme":{"light":"github-light","dark":"github-dark"}}
{
"id": "sync-a1b2c3d4-e5f6-7890-abcd-ef1234567890-u1",
"status": "COMPLETED",
"delayTime": 12,
"executionTime": 856,
"workerId": "oqk7ao1uomckye",
"output": {
"image_url": "https://image.runpod.ai/abc123/output.png",
"cost": 0.00251658
}
}
```
```json 400 theme={"theme":{"light":"github-light","dark":"github-dark"}}
{
"id": "sync-a1b2c3d4-e5f6-7890-abcd-ef1234567890-u1",
"status": "FAILED",
"error": "Invalid parameter: width must be divisible by 64"
}
```
Image URLs expire after 7 days. Download and store generated images immediately if you need to keep them.
## Cost calculation
Flux Schnell charges \$0.0024 per megapixel. Cost is calculated as: `(width × height / 1,000,000) × \$0.0024`.
| Image size | Megapixels | Cost |
| ---------- | ---------- | -------- |
| 512×512 | 0.26 MP | \$0.0006 |
| 1024×1024 | 1.05 MP | \$0.0025 |
| 1536×1536 | 2.36 MP | \$0.0057 |
# GPT-OSS 120B
Source: https://docs.runpod.io/public-endpoints/models/gpt-oss-120b
OpenAI's open-weight language model with 120 billion parameters for text generation. See model inputs and outputs on Runpod Public Endpoints.
GPT-OSS 120B is OpenAI's open-weight 120B parameter language model, offering powerful text generation capabilities with advanced reasoning and instruction-following abilities.
Test GPT-OSS 120B in the Runpod Hub playground.
| | |
| ------------ | ----------------------------------------------- |
| **Endpoint** | `https://api.runpod.ai/v2/gpt-oss-120b/runsync` |
| **Pricing** | \$10.00 per 1M tokens |
| **Type** | Text generation |
This endpoint is fully compatible with the OpenAI API. See the [OpenAI compatibility examples](#openai-api-compatibility) below.
## Request
All parameters are passed within the `input` object in the request body.
Prompt for text generation.
Maximum number of tokens to output.
Randomness of the output. Lower values make output more predictable and deterministic. Range: 0.0-1.0.
Nucleus sampling threshold. Samples from the smallest set of words whose cumulative probability exceeds this threshold.
Restricts sampling to the top K most probable words.
Stops generation if the given string is encountered.
```bash cURL theme={"theme":{"light":"github-light","dark":"github-dark"}}
curl -X POST "https://api.runpod.ai/v2/gpt-oss-120b/runsync" \
-H "Authorization: Bearer $RUNPOD_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"input": {
"prompt": "Explain the concept of quantum entanglement in simple terms:",
"max_tokens": 512,
"temperature": 0.7
}
}'
```
```python Python theme={"theme":{"light":"github-light","dark":"github-dark"}}
import requests
response = requests.post(
"https://api.runpod.ai/v2/gpt-oss-120b/runsync",
headers={
"Authorization": f"Bearer {RUNPOD_API_KEY}",
"Content-Type": "application/json",
},
json={
"input": {
"prompt": "Explain the concept of quantum entanglement in simple terms:",
"max_tokens": 512,
"temperature": 0.7,
}
},
)
result = response.json()
print(result["output"])
```
```javascript JavaScript theme={"theme":{"light":"github-light","dark":"github-dark"}}
const response = await fetch(
"https://api.runpod.ai/v2/gpt-oss-120b/runsync",
{
method: "POST",
headers: {
Authorization: `Bearer ${RUNPOD_API_KEY}`,
"Content-Type": "application/json",
},
body: JSON.stringify({
input: {
prompt: "Explain the concept of quantum entanglement in simple terms:",
max_tokens: 512,
temperature: 0.7,
},
}),
}
);
const result = await response.json();
console.log(result.output);
```
## Response
Unique identifier for the request.
Request status. Returns `COMPLETED` on success, `FAILED` on error.
Time in milliseconds the request spent in queue before processing began.
Time in milliseconds the model took to generate the response.
The generation result containing the text and usage information.
Array containing the generated text.
Cost of the generation in USD.
Token usage information with `input` and `output` counts.
```json 200 theme={"theme":{"light":"github-light","dark":"github-dark"}}
{
"delayTime": 30,
"executionTime": 4521,
"id": "sync-a1b2c3d4-e5f6-7890-abcd-ef1234567890-u1",
"output": [
{
"choices": [
{
"tokens": [
"Quantum entanglement is a phenomenon where two particles become connected in such a way that measuring one particle instantly affects the other, no matter how far apart they are..."
]
}
],
"cost": 0.005,
"usage": {
"input": 15,
"output": 485
}
}
],
"status": "COMPLETED"
}
```
## OpenAI API compatibility
GPT-OSS 120B is fully compatible with the OpenAI API format. You can use the OpenAI Python client to interact with this endpoint.
```python Python (OpenAI SDK) theme={"theme":{"light":"github-light","dark":"github-dark"}}
from openai import OpenAI
client = OpenAI(
api_key=RUNPOD_API_KEY,
base_url="https://api.runpod.ai/v2/gpt-oss-120b/openai/v1",
)
response = client.chat.completions.create(
model="gpt-oss-120b",
messages=[
{
"role": "system",
"content": "You are a helpful assistant.",
},
{
"role": "user",
"content": "Explain the concept of quantum entanglement in simple terms.",
},
],
max_tokens=512,
)
print(response.choices[0].message.content)
```
For streaming responses, add `stream=True`:
```python Python (Streaming) theme={"theme":{"light":"github-light","dark":"github-dark"}}
response = client.chat.completions.create(
model="gpt-oss-120b",
messages=[
{"role": "user", "content": "Write a short story about space exploration."}
],
max_tokens=512,
stream=True,
)
for chunk in response:
if chunk.choices[0].delta.content:
print(chunk.choices[0].delta.content, end="")
```
For more details, see [Send vLLM requests](/serverless/vllm/vllm-requests) and the [OpenAI API compatibility guide](/serverless/vllm/openai-compatibility).
## Cost calculation
GPT-OSS 120B charges \$10.00 per 1M tokens. Example costs:
| Tokens | Cost |
| ---------------- | ------- |
| 1,000 tokens | \$0.01 |
| 10,000 tokens | \$0.10 |
| 100,000 tokens | \$1.00 |
| 1,000,000 tokens | \$10.00 |
# IBM Granite 4.0
Source: https://docs.runpod.io/public-endpoints/models/granite-4
A 32-billion-parameter long-context instruction model for text generation. See model inputs and outputs on Runpod Public Endpoints.
IBM Granite-4.0-H-Small is a 32B parameter long-context instruct model. It excels at general text generation, instruction following, and conversational AI tasks with support for extended context lengths.
Test IBM Granite 4.0 in the Runpod Hub playground.
| | |
| ------------ | ------------------------------------------------------ |
| **Endpoint** | `https://api.runpod.ai/v2/granite-4-0-h-small/runsync` |
| **Pricing** | \$10.00 per 1M tokens |
| **Type** | Text generation |
## Request
All parameters are passed within the `input` object in the request body.
Array of message objects with role and content.
The role of the message author. Use `system`, `user`, or `assistant`.
The content of the message.
Maximum number of tokens to generate.
Controls randomness in generation. Lower values make output more deterministic. Range: 0.0-1.0.
Seed for reproducible results. Set to -1 for random.
Restricts sampling to the top K most probable tokens.
Nucleus sampling threshold. Range: 0.0-1.0.
```bash cURL theme={"theme":{"light":"github-light","dark":"github-dark"}}
curl -X POST "https://api.runpod.ai/v2/granite-4-0-h-small/runsync" \
-H "Authorization: Bearer $RUNPOD_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"input": {
"messages": [
{
"role": "system",
"content": "You are a helpful assistant. Please ensure responses are professional, accurate, and safe."
},
{
"role": "user",
"content": "What is Runpod?"
}
],
"sampling_params": {
"max_tokens": 512,
"temperature": 0.7,
"seed": -1,
"top_k": -1,
"top_p": 1
}
}
}'
```
```python Python theme={"theme":{"light":"github-light","dark":"github-dark"}}
import requests
response = requests.post(
"https://api.runpod.ai/v2/granite-4-0-h-small/runsync",
headers={
"Authorization": f"Bearer {RUNPOD_API_KEY}",
"Content-Type": "application/json",
},
json={
"input": {
"messages": [
{
"role": "system",
"content": "You are a helpful assistant. Please ensure responses are professional, accurate, and safe.",
},
{"role": "user", "content": "What is Runpod?"},
],
"sampling_params": {
"max_tokens": 512,
"temperature": 0.7,
"seed": -1,
"top_k": -1,
"top_p": 1,
},
}
},
)
result = response.json()
print(result["output"])
```
```javascript JavaScript theme={"theme":{"light":"github-light","dark":"github-dark"}}
const response = await fetch(
"https://api.runpod.ai/v2/granite-4-0-h-small/runsync",
{
method: "POST",
headers: {
Authorization: `Bearer ${RUNPOD_API_KEY}`,
"Content-Type": "application/json",
},
body: JSON.stringify({
input: {
messages: [
{
role: "system",
content: "You are a helpful assistant. Please ensure responses are professional, accurate, and safe.",
},
{ role: "user", content: "What is Runpod?" },
],
sampling_params: {
max_tokens: 512,
temperature: 0.7,
seed: -1,
top_k: -1,
top_p: 1,
},
},
}),
}
);
const result = await response.json();
console.log(result.output);
```
## Response
Unique identifier for the request.
Request status. Returns `COMPLETED` on success, `FAILED` on error.
Time in milliseconds the request spent in queue before processing began.
Time in milliseconds the model took to generate the response.
Identifier of the worker that processed the request.
The generation result containing the text and usage information.
Array containing the generated text.
Cost of the generation in USD.
Token usage information.
```json 200 theme={"theme":{"light":"github-light","dark":"github-dark"}}
{
"id": "sync-a1b2c3d4-e5f6-7890-abcd-ef1234567890-u1",
"status": "COMPLETED",
"delayTime": 15,
"executionTime": 2345,
"workerId": "oqk7ao1uomckye",
"output": {
"choices": [
{
"tokens": [
"Runpod is a cloud computing platform that provides GPU resources for AI and machine learning workloads..."
]
}
],
"cost": 0.00185,
"usage": {
"input": 35,
"output": 150
}
}
}
```
```json 400 theme={"theme":{"light":"github-light","dark":"github-dark"}}
{
"id": "sync-a1b2c3d4-e5f6-7890-abcd-ef1234567890-u1",
"status": "FAILED",
"error": "Invalid messages format"
}
```
## Cost calculation
IBM Granite 4.0 charges \$10.00 per 1M tokens. Example costs:
| Tokens | Cost |
| ---------------- | ------- |
| 1,000 tokens | \$0.01 |
| 10,000 tokens | \$0.10 |
| 100,000 tokens | \$1.00 |
| 1,000,000 tokens | \$10.00 |
# InfiniteTalk
Source: https://docs.runpod.io/public-endpoints/models/infinitetalk
Audio-driven video generation that creates talking or singing videos from a single image. See model inputs and outputs on Runpod Public Endpoints.
InfiniteTalk is an audio-driven video generation model that creates talking or singing videos from a single image and audio input. It animates faces and bodies to match the audio, making it ideal for creating talking head videos, virtual presenters, or lip-synced content.
Test InfiniteTalk in the Runpod Hub playground.
| | |
| ------------ | ----------------------------------------------- |
| **Endpoint** | `https://api.runpod.ai/v2/infinitetalk/runsync` |
| **Pricing** | \$0.25–\$0.50 per video |
| **Type** | Video generation |
## Request
All parameters are passed within the `input` object in the request body.
Text description of the desired video.
URL of the source image to animate.
URL of the audio file to drive the animation.
Output video resolution. Valid options: `480p` or `720p`.
Enable content safety checking.
```bash cURL theme={"theme":{"light":"github-light","dark":"github-dark"}}
curl -X POST "https://api.runpod.ai/v2/infinitetalk/runsync" \
-H "Authorization: Bearer $RUNPOD_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"input": {
"prompt": "a cartoon computer talking",
"image": "https://example.com/avatar.jpg",
"audio": "https://example.com/speech.wav",
"size": "480p",
"enable_safety_checker": true
}
}'
```
```python Python theme={"theme":{"light":"github-light","dark":"github-dark"}}
import requests
response = requests.post(
"https://api.runpod.ai/v2/infinitetalk/runsync",
headers={
"Authorization": f"Bearer {RUNPOD_API_KEY}",
"Content-Type": "application/json",
},
json={
"input": {
"prompt": "a cartoon computer talking",
"image": "https://example.com/avatar.jpg",
"audio": "https://example.com/speech.wav",
"size": "480p",
"enable_safety_checker": True,
}
},
)
result = response.json()
print(result["output"]["video_url"])
```
```javascript JavaScript theme={"theme":{"light":"github-light","dark":"github-dark"}}
const response = await fetch(
"https://api.runpod.ai/v2/infinitetalk/runsync",
{
method: "POST",
headers: {
Authorization: `Bearer ${RUNPOD_API_KEY}`,
"Content-Type": "application/json",
},
body: JSON.stringify({
input: {
prompt: "a cartoon computer talking",
image: "https://example.com/avatar.jpg",
audio: "https://example.com/speech.wav",
size: "480p",
enable_safety_checker: true,
},
}),
}
);
const result = await response.json();
console.log(result.output.video_url);
```
## Response
Unique identifier for the request.
Request status. Returns `COMPLETED` on success, `FAILED` on error.
Time in milliseconds the request spent in queue before processing began.
Time in milliseconds the model took to generate the video.
Identifier of the worker that processed the request.
The generation result containing the video URL and cost.
URL of the generated video. This URL expires after 7 days.
Cost of the generation in USD.
```json 200 theme={"theme":{"light":"github-light","dark":"github-dark"}}
{
"id": "sync-a1b2c3d4-e5f6-7890-abcd-ef1234567890-u1",
"status": "COMPLETED",
"delayTime": 25,
"executionTime": 45678,
"workerId": "oqk7ao1uomckye",
"output": {
"video_url": "https://video.runpod.ai/abc123/output.mp4",
"cost": 0.25
}
}
```
```json 400 theme={"theme":{"light":"github-light","dark":"github-dark"}}
{
"id": "sync-a1b2c3d4-e5f6-7890-abcd-ef1234567890-u1",
"status": "FAILED",
"error": "Invalid audio URL"
}
```
Video URLs expire after 7 days. Download and store generated videos immediately if you need to keep them.
## Cost calculation
InfiniteTalk pricing varies by resolution:
| Resolution | Cost |
| ---------- | ------ |
| 480p | \$0.25 |
| 720p | \$0.50 |
# Kling v2.1 I2V Pro
Source: https://docs.runpod.io/public-endpoints/models/kling-v2-1
Professional-grade image-to-video with enhanced visual fidelity. Review inputs and output formats for this model on Runpod Public Endpoints.
Kling v2.1 I2V Pro is a professional-grade image-to-video model with enhanced visual fidelity. It generates high-quality videos from static images with smooth motion and excellent detail preservation.
Test Kling v2.1 I2V Pro in the Runpod Hub playground.
| | |
| ------------ | ----------------------------------------------------- |
| **Endpoint** | `https://api.runpod.ai/v2/kling-v2-1-i2v-pro/runsync` |
| **Pricing** | \$0.45 per 5 seconds |
| **Type** | Video generation |
## Request
All parameters are passed within the `input` object in the request body.
Text description of the desired video motion and content.
URL of the source image to animate.
Elements to exclude from the video.
How closely to follow the prompt.
Video duration in seconds.
Enable content safety checking.
```bash cURL theme={"theme":{"light":"github-light","dark":"github-dark"}}
curl -X POST "https://api.runpod.ai/v2/kling-v2-1-i2v-pro/runsync" \
-H "Authorization: Bearer $RUNPOD_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"input": {
"prompt": "A majestic magic dragon breathing fire over an ancient castle",
"image": "https://example.com/dragon.png",
"negative_prompt": "",
"guidance_scale": 0.5,
"duration": 5,
"enable_safety_checker": true
}
}'
```
```python Python theme={"theme":{"light":"github-light","dark":"github-dark"}}
import requests
response = requests.post(
"https://api.runpod.ai/v2/kling-v2-1-i2v-pro/runsync",
headers={
"Authorization": f"Bearer {RUNPOD_API_KEY}",
"Content-Type": "application/json",
},
json={
"input": {
"prompt": "A majestic magic dragon breathing fire over an ancient castle",
"image": "https://example.com/dragon.png",
"negative_prompt": "",
"guidance_scale": 0.5,
"duration": 5,
"enable_safety_checker": True,
}
},
)
result = response.json()
print(result["output"]["video_url"])
```
```javascript JavaScript theme={"theme":{"light":"github-light","dark":"github-dark"}}
const response = await fetch(
"https://api.runpod.ai/v2/kling-v2-1-i2v-pro/runsync",
{
method: "POST",
headers: {
Authorization: `Bearer ${RUNPOD_API_KEY}`,
"Content-Type": "application/json",
},
body: JSON.stringify({
input: {
prompt: "A majestic magic dragon breathing fire over an ancient castle",
image: "https://example.com/dragon.png",
negative_prompt: "",
guidance_scale: 0.5,
duration: 5,
enable_safety_checker: true,
},
}),
}
);
const result = await response.json();
console.log(result.output.video_url);
```
## Response
Unique identifier for the request.
Request status. Returns `COMPLETED` on success, `FAILED` on error.
Time in milliseconds the request spent in queue before processing began.
Time in milliseconds the model took to generate the video.
Identifier of the worker that processed the request.
The generation result containing the video URL and cost.
URL of the generated video. This URL expires after 7 days.
Cost of the generation in USD.
```json 200 theme={"theme":{"light":"github-light","dark":"github-dark"}}
{
"id": "sync-a1b2c3d4-e5f6-7890-abcd-ef1234567890-u1",
"status": "COMPLETED",
"delayTime": 30,
"executionTime": 67890,
"workerId": "oqk7ao1uomckye",
"output": {
"video_url": "https://video.runpod.ai/abc123/output.mp4",
"cost": 0.36
}
}
```
```json 400 theme={"theme":{"light":"github-light","dark":"github-dark"}}
{
"id": "sync-a1b2c3d4-e5f6-7890-abcd-ef1234567890-u1",
"status": "FAILED",
"error": "Invalid image URL"
}
```
Video URLs expire after 7 days. Download and store generated videos immediately if you need to keep them.
## Cost calculation
Kling v2.1 I2V Pro pricing:
| Duration | Cost |
| ---------- | ------ |
| 5 seconds | \$0.45 |
| 10 seconds | \$0.90 |
# Kling v2.6 Motion Control
Source: https://docs.runpod.io/public-endpoints/models/kling-v2-6-motion-control
Transfer motion from reference videos to animate still images. Review inputs and output formats for this model on Runpod Public Endpoints.
Kling v2.6 Standard Motion Control transfers motion from reference videos to animate still images. Upload a character image and a motion clip, and the model extracts the movement to generate smooth video output.
Test Kling v2.6 Motion Control in the Runpod Hub playground.
| | |
| ------------ | ---------------------------------------------------------------- |
| **Endpoint** | `https://api.runpod.ai/v2/kling-v2-6-std-motion-control/runsync` |
| **Pricing** | \$0.21–\$0.63+ per video |
| **Type** | Video generation |
## Request
All parameters are passed within the `input` object in the request body.
URL of the character image to animate.
URL of the reference video containing the motion to transfer.
Source of character orientation. Options: `video`, `image`.
Text description of the desired motion.
Elements to exclude from the generated video.
Whether to preserve audio from the reference video.
```bash cURL theme={"theme":{"light":"github-light","dark":"github-dark"}}
curl -X POST "https://api.runpod.ai/v2/kling-v2-6-std-motion-control/runsync" \
-H "Authorization: Bearer $RUNPOD_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"input": {
"image": "https://example.com/character.png",
"video": "https://example.com/dance-motion.mp4",
"character_orientation": "video",
"prompt": "Make the character follow the dance movements",
"keep_original_sound": true
}
}'
```
```python Python theme={"theme":{"light":"github-light","dark":"github-dark"}}
import requests
response = requests.post(
"https://api.runpod.ai/v2/kling-v2-6-std-motion-control/runsync",
headers={
"Authorization": f"Bearer {RUNPOD_API_KEY}",
"Content-Type": "application/json",
},
json={
"input": {
"image": "https://example.com/character.png",
"video": "https://example.com/dance-motion.mp4",
"character_orientation": "video",
"prompt": "Make the character follow the dance movements",
"keep_original_sound": True,
}
},
)
result = response.json()
print(result["output"]["video_url"])
```
```javascript JavaScript theme={"theme":{"light":"github-light","dark":"github-dark"}}
const response = await fetch(
"https://api.runpod.ai/v2/kling-v2-6-std-motion-control/runsync",
{
method: "POST",
headers: {
Authorization: `Bearer ${RUNPOD_API_KEY}`,
"Content-Type": "application/json",
},
body: JSON.stringify({
input: {
image: "https://example.com/character.png",
video: "https://example.com/dance-motion.mp4",
character_orientation: "video",
prompt: "Make the character follow the dance movements",
keep_original_sound: true,
},
}),
}
);
const result = await response.json();
console.log(result.output.video_url);
```
## Response
Unique identifier for the request.
Request status. Returns `COMPLETED` on success, `FAILED` on error.
The generation result containing the video URL and cost.
URL of the generated video. This URL expires after 7 days.
Cost of the generation in USD.
```json 200 theme={"theme":{"light":"github-light","dark":"github-dark"}}
{
"id": "sync-a1b2c3d4-e5f6-7890-abcd-ef1234567890-u1",
"status": "COMPLETED",
"delayTime": 25,
"executionTime": 45678,
"output": {
"video_url": "https://video.runpod.ai/abc123/output.mp4",
"cost": 0.42
}
}
```
Video URLs expire after 7 days. Download and store generated videos immediately if you need to keep them.
## Cost calculation
Kling v2.6 Motion Control pricing by duration:
| Duration | Cost |
| ----------- | ------ |
| 1-3 seconds | \$0.21 |
| 4-6 seconds | \$0.42 |
| 7-9 seconds | \$0.63 |
# Kling Video O1 R2V
Source: https://docs.runpod.io/public-endpoints/models/kling-video-o1-r2v
Creative video generation using character, prop, or scene references from multiple viewpoints. See model inputs and outputs on Runpod Public Endpoints.
Kling Video O1 R2V generates creative videos using character, prop, or scene references from multiple viewpoints. It can combine multiple reference images to create coherent video content.
Test Kling Video O1 R2V in the Runpod Hub playground.
| | |
| ------------ | ----------------------------------------------------- |
| **Endpoint** | `https://api.runpod.ai/v2/kling-video-o1-r2v/runsync` |
| **Pricing** | \$0.112 per second |
| **Type** | Video generation |
## Request
All parameters are passed within the `input` object in the request body.
Text description of the desired video action.
Array of 1-10 reference image URLs.
Optional reference video URL.
Elements to exclude from the generated video.
Aspect ratio. Options: `16:9`, `9:16`, `1:1`.
Video duration in seconds. Options: 3-10 seconds.
Seed for reproducible results. Set to -1 for random.
Automatically expand and enhance the prompt.
Enable content safety checking.
```bash cURL theme={"theme":{"light":"github-light","dark":"github-dark"}}
curl -X POST "https://api.runpod.ai/v2/kling-video-o1-r2v/runsync" \
-H "Authorization: Bearer $RUNPOD_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"input": {
"prompt": "The robot is dancing with the teddy bear",
"images": [
"https://example.com/robot.png",
"https://example.com/teddy-bear.png"
],
"aspect_ratio": "16:9",
"duration": 5,
"seed": -1
}
}'
```
```python Python theme={"theme":{"light":"github-light","dark":"github-dark"}}
import requests
response = requests.post(
"https://api.runpod.ai/v2/kling-video-o1-r2v/runsync",
headers={
"Authorization": f"Bearer {RUNPOD_API_KEY}",
"Content-Type": "application/json",
},
json={
"input": {
"prompt": "The robot is dancing with the teddy bear",
"images": [
"https://example.com/robot.png",
"https://example.com/teddy-bear.png",
],
"aspect_ratio": "16:9",
"duration": 5,
"seed": -1,
}
},
)
result = response.json()
print(result["output"]["video_url"])
```
```javascript JavaScript theme={"theme":{"light":"github-light","dark":"github-dark"}}
const response = await fetch(
"https://api.runpod.ai/v2/kling-video-o1-r2v/runsync",
{
method: "POST",
headers: {
Authorization: `Bearer ${RUNPOD_API_KEY}`,
"Content-Type": "application/json",
},
body: JSON.stringify({
input: {
prompt: "The robot is dancing with the teddy bear",
images: [
"https://example.com/robot.png",
"https://example.com/teddy-bear.png",
],
aspect_ratio: "16:9",
duration: 5,
seed: -1,
},
}),
}
);
const result = await response.json();
console.log(result.output.video_url);
```
## Response
Unique identifier for the request.
Request status. Returns `COMPLETED` on success, `FAILED` on error.
The generation result containing the video URL and cost.
URL of the generated video. This URL expires after 7 days.
Cost of the generation in USD.
```json 200 theme={"theme":{"light":"github-light","dark":"github-dark"}}
{
"id": "sync-a1b2c3d4-e5f6-7890-abcd-ef1234567890-u1",
"status": "COMPLETED",
"delayTime": 30,
"executionTime": 75432,
"output": {
"video_url": "https://video.runpod.ai/abc123/output.mp4",
"cost": 0.56
}
}
```
Video URLs expire after 7 days. Download and store generated videos immediately if you need to keep them.
## Cost calculation
Kling Video O1 R2V charges \$0.112 per second of video:
| Duration | Cost |
| ---------- | ------- |
| 3 seconds | \$0.336 |
| 5 seconds | \$0.56 |
| 10 seconds | \$1.12 |
# Minimax Speech 02 HD
Source: https://docs.runpod.io/public-endpoints/models/minimax-speech
High-definition text-to-speech with emotional control and voice customization. See model inputs and outputs on Runpod Public Endpoints.
Minimax Speech 02 HD is a high-definition text-to-speech model with emotional control and voice customization. It produces natural-sounding speech with adjustable speed, pitch, volume, and emotional tone.
Test Minimax Speech 02 HD in the Runpod Hub playground.
| | |
| ------------ | ------------------------------------------------------- |
| **Endpoint** | `https://api.runpod.ai/v2/minimax-speech-02-hd/runsync` |
| **Pricing** | \$0.05 per 1000 characters |
| **Type** | Text-to-speech |
## Request
All parameters are passed within the `input` object in the request body.
Text to convert to speech.
Voice identifier for the desired voice.
Speech speed multiplier.
Volume level.
Pitch adjustment.
Emotion to convey. Options include `happy`, `sad`, `neutral`, `angry`, `fearful`, `surprised`.
Enable English text normalization for better pronunciation of numbers, abbreviations, etc.
Fallback audio URL if generation fails.
```bash cURL theme={"theme":{"light":"github-light","dark":"github-dark"}}
curl -X POST "https://api.runpod.ai/v2/minimax-speech-02-hd/runsync" \
-H "Authorization: Bearer $RUNPOD_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"input": {
"prompt": "Welcome to our advanced text-to-speech system. This is a demonstration of natural speech synthesis.",
"voice_id": "Wise_Woman",
"speed": 1,
"volume": 1,
"pitch": 0,
"emotion": "happy",
"english_normalization": false
}
}'
```
```python Python theme={"theme":{"light":"github-light","dark":"github-dark"}}
import requests
response = requests.post(
"https://api.runpod.ai/v2/minimax-speech-02-hd/runsync",
headers={
"Authorization": f"Bearer {RUNPOD_API_KEY}",
"Content-Type": "application/json",
},
json={
"input": {
"prompt": "Welcome to our advanced text-to-speech system. This is a demonstration of natural speech synthesis.",
"voice_id": "Wise_Woman",
"speed": 1,
"volume": 1,
"pitch": 0,
"emotion": "happy",
"english_normalization": False,
}
},
)
result = response.json()
print(result["output"]["audio_url"])
```
```javascript JavaScript theme={"theme":{"light":"github-light","dark":"github-dark"}}
const response = await fetch(
"https://api.runpod.ai/v2/minimax-speech-02-hd/runsync",
{
method: "POST",
headers: {
Authorization: `Bearer ${RUNPOD_API_KEY}`,
"Content-Type": "application/json",
},
body: JSON.stringify({
input: {
prompt: "Welcome to our advanced text-to-speech system. This is a demonstration of natural speech synthesis.",
voice_id: "Wise_Woman",
speed: 1,
volume: 1,
pitch: 0,
emotion: "happy",
english_normalization: false,
},
}),
}
);
const result = await response.json();
console.log(result.output.audio_url);
```
## Response
Unique identifier for the request.
Request status. Returns `COMPLETED` on success, `FAILED` on error.
Time in milliseconds the request spent in queue before processing began.
Time in milliseconds the model took to generate the audio.
Identifier of the worker that processed the request.
The generation result containing the audio URL and cost.
URL of the generated audio file. This URL expires after 7 days.
Cost of the generation in USD.
```json 200 theme={"theme":{"light":"github-light","dark":"github-dark"}}
{
"id": "sync-a1b2c3d4-e5f6-7890-abcd-ef1234567890-u1",
"status": "COMPLETED",
"delayTime": 14,
"executionTime": 3456,
"workerId": "oqk7ao1uomckye",
"output": {
"audio_url": "https://audio.runpod.ai/abc123/output.mp3",
"cost": 0.0055
}
}
```
```json 400 theme={"theme":{"light":"github-light","dark":"github-dark"}}
{
"id": "sync-a1b2c3d4-e5f6-7890-abcd-ef1234567890-u1",
"status": "FAILED",
"error": "Invalid voice_id"
}
```
Audio URLs expire after 7 days. Download and store generated audio files immediately if you need to keep them.
## Cost calculation
Minimax Speech 02 HD charges \$0.05 per 1000 characters. Example costs:
| Characters | Cost |
| ----------------- | ------- |
| 500 characters | \$0.025 |
| 1,000 characters | \$0.05 |
| 10,000 characters | \$0.50 |
# Moonshot Kimi
Source: https://docs.runpod.io/public-endpoints/models/moonshot-kimi
Use Moonshot Kimi models for reasoning, chat, and coding through Runpod Public Endpoints, with supported inputs, outputs, and request examples.
Moonshot's Kimi family of models handles advanced reasoning and chat. It supports extended thinking with a visible reasoning trace before the final answer. A single endpoint serves three variants, each selected by setting the `model` field in the request body.
Test Moonshot Kimi in the Runpod Hub playground.
| | |
| ------------ | ---------------------------------------------------------------------- |
| **Endpoint** | `https://api.runpod.ai/v2/moonshot-kimi/runsync` |
| **Pricing** | \$4.00–\$15.00 per 1M output tokens, \$0.95–\$3.00 per 1M input tokens |
| **Type** | Text generation |
This endpoint is fully compatible with the OpenAI API. See the [OpenAI compatibility examples](#openai-api-compatibility) below.
## Model variants
Choose a variant by setting the `model` field in the request body (default `kimi-k2.6`). The endpoint slug stays `moonshot-kimi` for every variant.
| `model` value | Description | Context window | Input price | Output price |
| --------------------- | --------------------------------------------------------------------------------------------------------------------- | --------------------- | -------------------- | --------------------- |
| `kimi-k2.6` (default) | General-purpose model with thinking and non-thinking modes, agentic capabilities, and multimodal input. | 256K (262,144 tokens) | \$0.95 per 1M tokens | \$4.00 per 1M tokens |
| `kimi-k2.7-code` | Coding-focused model with thinking mode and agentic capabilities. | 256K (262,144 tokens) | \$0.95 per 1M tokens | \$4.00 per 1M tokens |
| `kimi-k3` | Flagship model with always-on reasoning and configurable reasoning effort for long-horizon coding and knowledge work. | 1M (1,048,576 tokens) | \$3.00 per 1M tokens | \$15.00 per 1M tokens |
To target a variant other than the default, set the `model` field to its ID. For example, to use the flagship model, set `"model": "kimi-k3"` in the request body (or `model="kimi-k3"` with the OpenAI SDK).
## Request
All parameters are passed within the `input` object in the request body.
Array of message objects with role and content.
The role of the message author. Use `system`, `user`, or `assistant`.
The content of the message.
The Kimi variant to use. One of `kimi-k2.6`, `kimi-k2.7-code`, or `kimi-k3`.
Maximum number of tokens to generate.
Controls randomness in generation. Lower values make output more deterministic.
Seed for reproducible results.
Restricts sampling to the top K most probable tokens.
Nucleus sampling threshold. Range: 0.0-1.0.
```bash cURL theme={"theme":{"light":"github-light","dark":"github-dark"}}
curl -X POST "https://api.runpod.ai/v2/moonshot-kimi/runsync" \
-H "Authorization: Bearer $RUNPOD_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"input": {
"messages": [
{
"role": "system",
"content": "You are Kimi."
},
{
"role": "user",
"content": "What is Runpod?"
}
],
"sampling_params": {
"max_tokens": 2048,
"temperature": 1
},
"model": "kimi-k2.6"
}
}'
```
```python Python theme={"theme":{"light":"github-light","dark":"github-dark"}}
import requests
response = requests.post(
"https://api.runpod.ai/v2/moonshot-kimi/runsync",
headers={
"Authorization": f"Bearer {RUNPOD_API_KEY}",
"Content-Type": "application/json",
},
json={
"input": {
"messages": [
{"role": "system", "content": "You are Kimi."},
{"role": "user", "content": "What is Runpod?"},
],
"sampling_params": {
"max_tokens": 2048,
"temperature": 1,
},
"model": "kimi-k2.6",
}
},
)
result = response.json()
print(result["output"])
```
```javascript JavaScript theme={"theme":{"light":"github-light","dark":"github-dark"}}
const response = await fetch(
"https://api.runpod.ai/v2/moonshot-kimi/runsync",
{
method: "POST",
headers: {
Authorization: `Bearer ${RUNPOD_API_KEY}`,
"Content-Type": "application/json",
},
body: JSON.stringify({
input: {
messages: [
{ role: "system", content: "You are Kimi." },
{ role: "user", content: "What is Runpod?" },
],
sampling_params: {
max_tokens: 2048,
temperature: 1,
},
model: "kimi-k2.6",
},
}),
}
);
const result = await response.json();
console.log(result.output);
```
## Response
Unique identifier for the request.
Request status. Returns `COMPLETED` on success, `FAILED` on error.
Time in milliseconds the request spent in queue before processing began.
Time in milliseconds the model took to generate the response.
Identifier of the worker that processed the request.
The generation result containing the text and usage information.
Array containing the generated text.
Cost of the generation in USD.
Token usage information.
```json 200 theme={"theme":{"light":"github-light","dark":"github-dark"}}
{
"id": "sync-a1b2c3d4-e5f6-7890-abcd-ef1234567890-u1",
"status": "COMPLETED",
"delayTime": 15,
"executionTime": 2345,
"workerId": "oqk7ao1uomckye",
"output": {
"choices": [
{
"tokens": [
"Runpod is a cloud computing platform that provides GPU resources for AI and machine learning workloads..."
]
}
],
"cost": 0.00074,
"usage": {
"input": 35,
"output": 150
}
}
}
```
```json 400 theme={"theme":{"light":"github-light","dark":"github-dark"}}
{
"id": "sync-a1b2c3d4-e5f6-7890-abcd-ef1234567890-u1",
"status": "FAILED",
"error": "Invalid messages format"
}
```
## OpenAI API compatibility
Moonshot Kimi is fully compatible with the OpenAI API format. You can use the OpenAI Python client to interact with this endpoint. You can set `model` to any of the three variant IDs.
```python Python (OpenAI SDK) theme={"theme":{"light":"github-light","dark":"github-dark"}}
from openai import OpenAI
client = OpenAI(
api_key=RUNPOD_API_KEY,
base_url="https://api.runpod.ai/v2/moonshot-kimi/openai/v1",
)
response = client.chat.completions.create(
model="kimi-k2.6",
messages=[
{
"role": "system",
"content": "You are Kimi.",
},
{
"role": "user",
"content": "What is Runpod?",
},
],
max_tokens=2048,
temperature=1,
reasoning_effort="max", # set for adding reasoning
top_p=0.9,
)
print(response.choices[0].message.content)
```
For streaming responses, add `stream=True`:
```python Python (Streaming) theme={"theme":{"light":"github-light","dark":"github-dark"}}
response = client.chat.completions.create(
model="kimi-k2.6",
messages=[
{"role": "system", "content": "You are Kimi."},
{"role": "user", "content": "Explain quantum computing in simple terms."}
],
max_tokens=2048,
stream=True,
)
for chunk in response:
if chunk.choices[0].delta.content:
print(chunk.choices[0].delta.content, end="")
```
For more details, see [Send vLLM requests](/serverless/vllm/vllm-requests) and the [OpenAI API compatibility guide](/serverless/vllm/openai-compatibility).
## Cost calculation
Moonshot Kimi charges separate rates for input and output tokens, and the rate depends on the variant. Kimi K2.6 and Kimi K2.7 Code charge \$0.95 per 1M input tokens and \$4.00 per 1M output tokens. Kimi K3 charges \$3.00 per 1M input tokens and \$15.00 per 1M output tokens. Example costs:
| Tokens | K2.6 & K2.7 Code input (\$0.95/1M) | K2.6 & K2.7 Code output (\$4.00/1M) | K3 input (\$3.00/1M) | K3 output (\$15.00/1M) |
| ---------------- | ---------------------------------- | ----------------------------------- | -------------------- | ---------------------- |
| 1,000 tokens | \$0.00095 | \$0.004 | \$0.003 | \$0.015 |
| 10,000 tokens | \$0.0095 | \$0.04 | \$0.03 | \$0.15 |
| 100,000 tokens | \$0.095 | \$0.40 | \$0.30 | \$1.50 |
| 1,000,000 tokens | \$0.95 | \$4.00 | \$3.00 | \$15.00 |
# Nano Banana 2 Edit
Source: https://docs.runpod.io/public-endpoints/models/nano-banana-2-edit
Google's latest multi-image editing model with resolution options up to 4K. Explore this model's inputs and outputs on Runpod Public Endpoints.
Nano Banana 2 Edit is Google's latest image editing model for combining and editing multiple reference images. It supports up to 14 input images and offers resolution options from 1K to 4K output. For best results, use 1-3 reference images.
Test Nano Banana 2 Edit in the Runpod Hub playground.
| | |
| ------------ | ------------------------------------------------------------ |
| **Endpoint** | `https://api.runpod.ai/v2/google-nano-banana-2-edit/runsync` |
| **Pricing** | \$0.0875 (1K), \$0.13 (2K), \$0.175 (4K) |
| **Type** | Image editing |
## Request
All parameters are passed within the `input` object in the request body.
Array of reference image URLs to edit or combine. Supports up to 14 images, but 1-3 images is recommended for best stability.
Text description of the desired edit or how to combine the images.
Output resolution. Options: `1k`, `2k`, `4k`.
Output aspect ratio (e.g., `1:1`, `3:2`, `16:9`, `9:16`, `4:5`).
Output format. Options: `png`, `jpeg`.
Return result as base64 string instead of URL.
```bash cURL theme={"theme":{"light":"github-light","dark":"github-dark"}}
curl -X POST "https://api.runpod.ai/v2/google-nano-banana-2-edit/runsync" \
-H "Authorization: Bearer $RUNPOD_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"input": {
"images": [
"https://example.com/subject.jpg",
"https://example.com/style-reference.jpg"
],
"prompt": "Apply the artistic style from the second image to the subject in the first image",
"resolution": "1k",
"aspect_ratio": "1:1",
"output_format": "png"
}
}'
```
```python Python theme={"theme":{"light":"github-light","dark":"github-dark"}}
import requests
response = requests.post(
"https://api.runpod.ai/v2/google-nano-banana-2-edit/runsync",
headers={
"Authorization": f"Bearer {RUNPOD_API_KEY}",
"Content-Type": "application/json",
},
json={
"input": {
"images": [
"https://example.com/subject.jpg",
"https://example.com/style-reference.jpg",
],
"prompt": "Apply the artistic style from the second image to the subject in the first image",
"resolution": "1k",
"aspect_ratio": "1:1",
"output_format": "png",
}
},
)
result = response.json()
print(result["output"]["image_url"])
```
```javascript JavaScript theme={"theme":{"light":"github-light","dark":"github-dark"}}
const response = await fetch(
"https://api.runpod.ai/v2/google-nano-banana-2-edit/runsync",
{
method: "POST",
headers: {
Authorization: `Bearer ${RUNPOD_API_KEY}`,
"Content-Type": "application/json",
},
body: JSON.stringify({
input: {
images: [
"https://example.com/subject.jpg",
"https://example.com/style-reference.jpg",
],
prompt: "Apply the artistic style from the second image to the subject in the first image",
resolution: "1k",
aspect_ratio: "1:1",
output_format: "png",
},
}),
}
);
const result = await response.json();
console.log(result.output.image_url);
```
## Response
Unique identifier for the request.
Request status. Returns `COMPLETED` on success, `FAILED` on error.
Time in milliseconds the request spent in queue before processing began.
Time in milliseconds the model took to edit the images.
The generation result containing the image URL and cost.
URL of the edited image. This URL expires after 7 days.
Cost of the generation in USD.
```json 200 theme={"theme":{"light":"github-light","dark":"github-dark"}}
{
"id": "sync-a1b2c3d4-e5f6-7890-abcd-ef1234567890-u1",
"status": "COMPLETED",
"delayTime": 21,
"executionTime": 6234,
"output": {
"image_url": "https://image.runpod.ai/abc123/output.png",
"cost": 0.0875
}
}
```
```json 400 theme={"theme":{"light":"github-light","dark":"github-dark"}}
{
"id": "sync-a1b2c3d4-e5f6-7890-abcd-ef1234567890-u1",
"status": "FAILED",
"error": "Invalid image URL in images array"
}
```
Image URLs expire after 7 days. Download and store edited images immediately if you need to keep them.
## Cost calculation
Nano Banana 2 Edit pricing varies by output resolution:
| Resolution | Price per image |
| ---------- | --------------- |
| 1K | \$0.0875 |
| 2K | \$0.13 |
| 4K | \$0.175 |
# Nano Banana Edit
Source: https://docs.runpod.io/public-endpoints/models/nano-banana-edit
Google's state-of-the-art image editing model for combining multiple source images. See model inputs and outputs on Runpod Public Endpoints.
Nano Banana Edit is Google's state-of-the-art image editing model that excels at combining multiple source images into a cohesive output. It can take up to four reference images and merge them based on text instructions.
Test Nano Banana Edit in the Runpod Hub playground.
| | |
| ------------ | --------------------------------------------------- |
| **Endpoint** | `https://api.runpod.ai/v2/nano-banana-edit/runsync` |
| **Pricing** | \$0.038 per image |
| **Type** | Image editing |
## Request
All parameters are passed within the `input` object in the request body.
Editing instructions describing the desired transformation or how to combine the images.
Array of image URLs to edit or combine. Supports up to 4 images.
Enable content safety checking.
```bash cURL theme={"theme":{"light":"github-light","dark":"github-dark"}}
curl -X POST "https://api.runpod.ai/v2/nano-banana-edit/runsync" \
-H "Authorization: Bearer $RUNPOD_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"input": {
"prompt": "Combine these four source images into a single realistic 3D character figure scene",
"images": [
"https://example.com/character1.jpg",
"https://example.com/character2.jpg"
],
"enable_safety_checker": true
}
}'
```
```python Python theme={"theme":{"light":"github-light","dark":"github-dark"}}
import requests
response = requests.post(
"https://api.runpod.ai/v2/nano-banana-edit/runsync",
headers={
"Authorization": f"Bearer {RUNPOD_API_KEY}",
"Content-Type": "application/json",
},
json={
"input": {
"prompt": "Combine these four source images into a single realistic 3D character figure scene",
"images": [
"https://example.com/character1.jpg",
"https://example.com/character2.jpg",
],
"enable_safety_checker": True,
}
},
)
result = response.json()
print(result["output"]["image_url"])
```
```javascript JavaScript theme={"theme":{"light":"github-light","dark":"github-dark"}}
const response = await fetch(
"https://api.runpod.ai/v2/nano-banana-edit/runsync",
{
method: "POST",
headers: {
Authorization: `Bearer ${RUNPOD_API_KEY}`,
"Content-Type": "application/json",
},
body: JSON.stringify({
input: {
prompt: "Combine these four source images into a single realistic 3D character figure scene",
images: [
"https://example.com/character1.jpg",
"https://example.com/character2.jpg",
],
enable_safety_checker: true,
},
}),
}
);
const result = await response.json();
console.log(result.output.image_url);
```
## Response
Unique identifier for the request.
Request status. Returns `COMPLETED` on success, `FAILED` on error.
Time in milliseconds the request spent in queue before processing began.
Time in milliseconds the model took to edit the images.
Identifier of the worker that processed the request.
The generation result containing the image URL and cost.
URL of the edited/combined image. This URL expires after 7 days.
Cost of the generation in USD, calculated based on the output megapixels.
```json 200 theme={"theme":{"light":"github-light","dark":"github-dark"}}
{
"id": "sync-a1b2c3d4-e5f6-7890-abcd-ef1234567890-u1",
"status": "COMPLETED",
"delayTime": 23,
"executionTime": 5432,
"workerId": "oqk7ao1uomckye",
"output": {
"image_url": "https://image.runpod.ai/abc123/output.png",
"cost": 0.028311
}
}
```
```json 400 theme={"theme":{"light":"github-light","dark":"github-dark"}}
{
"id": "sync-a1b2c3d4-e5f6-7890-abcd-ef1234567890-u1",
"status": "FAILED",
"error": "Invalid image URL in images array"
}
```
Image URLs expire after 7 days. Download and store edited images immediately if you need to keep them.
## Cost calculation
Nano Banana Edit charges \$0.038 per image edited.
# Nano Banana Pro Edit
Source: https://docs.runpod.io/public-endpoints/models/nano-banana-pro-edit
Google's advanced image editing model with support for up to 14 reference images and multiple resolutions.
Nano Banana Pro Edit is Google's advanced image editing model that excels at combining multiple source images into a cohesive output. It supports up to 14 reference images and offers multiple resolution options for flexible output quality.
Test Nano Banana Pro Edit in the Runpod Hub playground.
| | |
| ------------ | ------------------------------------------------------- |
| **Endpoint** | `https://api.runpod.ai/v2/nano-banana-pro-edit/runsync` |
| **Pricing** | \$0.14–\$0.24 per image |
| **Type** | Image editing |
## Request
All parameters are passed within the `input` object in the request body.
Editing instructions describing the desired transformation or how to combine the images.
Array of image URLs to edit or combine. Supports up to 14 images.
Output resolution. Options: `1k`, `2k`, `4k`.
Output aspect ratio. Options: `1:1`, `3:2`, `2:3`, `4:3`, `4:5`, `5:4`, `9:16`, `16:9`, `21:9`.
Output image format. Options: `png`, `jpeg`.
Enable content safety checking.
```bash cURL theme={"theme":{"light":"github-light","dark":"github-dark"}}
curl -X POST "https://api.runpod.ai/v2/nano-banana-pro-edit/runsync" \
-H "Authorization: Bearer $RUNPOD_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"input": {
"prompt": "Combine these images into a cohesive scene with consistent lighting",
"images": [
"https://example.com/image1.jpg",
"https://example.com/image2.jpg",
"https://example.com/image3.jpg"
],
"resolution": "2k",
"aspect_ratio": "16:9",
"enable_safety_checker": true
}
}'
```
```python Python theme={"theme":{"light":"github-light","dark":"github-dark"}}
import requests
response = requests.post(
"https://api.runpod.ai/v2/nano-banana-pro-edit/runsync",
headers={
"Authorization": f"Bearer {RUNPOD_API_KEY}",
"Content-Type": "application/json",
},
json={
"input": {
"prompt": "Combine these images into a cohesive scene with consistent lighting",
"images": [
"https://example.com/image1.jpg",
"https://example.com/image2.jpg",
"https://example.com/image3.jpg",
],
"resolution": "2k",
"aspect_ratio": "16:9",
"enable_safety_checker": True,
}
},
)
result = response.json()
print(result["output"]["image_url"])
```
```javascript JavaScript theme={"theme":{"light":"github-light","dark":"github-dark"}}
const response = await fetch(
"https://api.runpod.ai/v2/nano-banana-pro-edit/runsync",
{
method: "POST",
headers: {
Authorization: `Bearer ${RUNPOD_API_KEY}`,
"Content-Type": "application/json",
},
body: JSON.stringify({
input: {
prompt: "Combine these images into a cohesive scene with consistent lighting",
images: [
"https://example.com/image1.jpg",
"https://example.com/image2.jpg",
"https://example.com/image3.jpg",
],
resolution: "2k",
aspect_ratio: "16:9",
enable_safety_checker: true,
},
}),
}
);
const result = await response.json();
console.log(result.output.image_url);
```
## Response
Unique identifier for the request.
Request status. Returns `COMPLETED` on success, `FAILED` on error.
The generation result containing the image URL and cost.
URL of the edited/combined image. This URL expires after 7 days.
Cost of the generation in USD.
```json 200 theme={"theme":{"light":"github-light","dark":"github-dark"}}
{
"id": "sync-a1b2c3d4-e5f6-7890-abcd-ef1234567890-u1",
"status": "COMPLETED",
"delayTime": 25,
"executionTime": 8432,
"output": {
"image_url": "https://image.runpod.ai/abc123/output.png",
"cost": 0.14
}
}
```
Image URLs expire after 7 days. Download and store edited images immediately if you need to keep them.
## Cost calculation
Nano Banana Pro Edit charges based on resolution:
| Resolution | Cost |
| ---------- | ------ |
| 1k | \$0.14 |
| 2k | \$0.14 |
| 4k | \$0.24 |
# P-Image Edit
Source: https://docs.runpod.io/public-endpoints/models/p-image-edit
Premium image editing with complex compositions, style transfers, and targeted edits. See model inputs and outputs on Runpod Public Endpoints.
P-Image Edit is Pruna's premium image editing model that supports complex compositions, style transfers, and targeted edits with text instructions. It can process up to 5 images in a single request.
Test P-Image Edit in the Runpod Hub playground.
| | |
| ------------ | ----------------------------------------------- |
| **Endpoint** | `https://api.runpod.ai/v2/p-image-edit/runsync` |
| **Pricing** | \$0.01 per image |
| **Type** | Image editing |
## Request
All parameters are passed within the `input` object in the request body.
Text instructions for editing the image.
Array of 1-5 image URLs for batch processing.
Output aspect ratio. Options: `match_input_image`, `1:1`, `16:9`, `9:16`, `4:3`, `3:4`, `3:2`, `2:3`.
Seed for reproducible results. Set to -1 for random.
Disable content safety filtering.
```bash cURL theme={"theme":{"light":"github-light","dark":"github-dark"}}
curl -X POST "https://api.runpod.ai/v2/p-image-edit/runsync" \
-H "Authorization: Bearer $RUNPOD_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"input": {
"prompt": "Transform into a watercolor painting style",
"images": ["https://example.com/photo.jpg"],
"aspect_ratio": "match_input_image",
"seed": -1
}
}'
```
```python Python theme={"theme":{"light":"github-light","dark":"github-dark"}}
import requests
response = requests.post(
"https://api.runpod.ai/v2/p-image-edit/runsync",
headers={
"Authorization": f"Bearer {RUNPOD_API_KEY}",
"Content-Type": "application/json",
},
json={
"input": {
"prompt": "Transform into a watercolor painting style",
"images": ["https://example.com/photo.jpg"],
"aspect_ratio": "match_input_image",
"seed": -1,
}
},
)
result = response.json()
print(result["output"]["image_url"])
```
```javascript JavaScript theme={"theme":{"light":"github-light","dark":"github-dark"}}
const response = await fetch(
"https://api.runpod.ai/v2/p-image-edit/runsync",
{
method: "POST",
headers: {
Authorization: `Bearer ${RUNPOD_API_KEY}`,
"Content-Type": "application/json",
},
body: JSON.stringify({
input: {
prompt: "Transform into a watercolor painting style",
images: ["https://example.com/photo.jpg"],
aspect_ratio: "match_input_image",
seed: -1,
},
}),
}
);
const result = await response.json();
console.log(result.output.image_url);
```
## Response
Unique identifier for the request.
Request status. Returns `COMPLETED` on success, `FAILED` on error.
The generation result containing the image URL and cost.
URL of the edited image. This URL expires after 7 days.
Cost of the generation in USD.
```json 200 theme={"theme":{"light":"github-light","dark":"github-dark"}}
{
"id": "sync-a1b2c3d4-e5f6-7890-abcd-ef1234567890-u1",
"status": "COMPLETED",
"delayTime": 12,
"executionTime": 2456,
"output": {
"image_url": "https://image.runpod.ai/abc123/output.png",
"cost": 0.01
}
}
```
Image URLs expire after 7 days. Download and store edited images immediately if you need to keep them.
## Cost calculation
P-Image Edit charges \$0.01 per image edited.
# P-Image T2I
Source: https://docs.runpod.io/public-endpoints/models/p-image-t2i
Ultra-fast text-to-image with automatic prompt enhancement and 2-stage refinement. See model inputs and outputs on Runpod Public Endpoints.
P-Image is Pruna's ultra-fast text-to-image model with automatic prompt enhancement and 2-stage refinement. It generates high-quality images quickly with minimal configuration.
Test P-Image T2I in the Runpod Hub playground.
| | |
| ------------ | ---------------------------------------------- |
| **Endpoint** | `https://api.runpod.ai/v2/p-image-t2i/runsync` |
| **Pricing** | \$0.005 per image |
| **Type** | Image generation |
## Request
All parameters are passed within the `input` object in the request body.
Text description of the desired image.
Output aspect ratio. Options: `1:1`, `16:9`, `9:16`, `4:3`, `3:4`, `3:2`, `2:3`.
Seed for reproducible results.
Disable content safety filtering.
```bash cURL theme={"theme":{"light":"github-light","dark":"github-dark"}}
curl -X POST "https://api.runpod.ai/v2/p-image-t2i/runsync" \
-H "Authorization: Bearer $RUNPOD_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"input": {
"prompt": "A majestic lion standing on a rocky cliff at sunset",
"aspect_ratio": "16:9"
}
}'
```
```python Python theme={"theme":{"light":"github-light","dark":"github-dark"}}
import requests
response = requests.post(
"https://api.runpod.ai/v2/p-image-t2i/runsync",
headers={
"Authorization": f"Bearer {RUNPOD_API_KEY}",
"Content-Type": "application/json",
},
json={
"input": {
"prompt": "A majestic lion standing on a rocky cliff at sunset",
"aspect_ratio": "16:9",
}
},
)
result = response.json()
print(result["output"]["image_url"])
```
```javascript JavaScript theme={"theme":{"light":"github-light","dark":"github-dark"}}
const response = await fetch(
"https://api.runpod.ai/v2/p-image-t2i/runsync",
{
method: "POST",
headers: {
Authorization: `Bearer ${RUNPOD_API_KEY}`,
"Content-Type": "application/json",
},
body: JSON.stringify({
input: {
prompt: "A majestic lion standing on a rocky cliff at sunset",
aspect_ratio: "16:9",
},
}),
}
);
const result = await response.json();
console.log(result.output.image_url);
```
## Response
Unique identifier for the request.
Request status. Returns `COMPLETED` on success, `FAILED` on error.
The generation result containing the image URL and cost.
URL of the generated image. This URL expires after 7 days.
Cost of the generation in USD.
```json 200 theme={"theme":{"light":"github-light","dark":"github-dark"}}
{
"id": "sync-a1b2c3d4-e5f6-7890-abcd-ef1234567890-u1",
"status": "COMPLETED",
"delayTime": 10,
"executionTime": 1245,
"output": {
"image_url": "https://image.runpod.ai/abc123/output.png",
"cost": 0.005
}
}
```
Image URLs expire after 7 days. Download and store generated images immediately if you need to keep them.
## Cost calculation
P-Image T2I charges \$0.005 per image generated.
# Pruna Video
Source: https://docs.runpod.io/public-endpoints/models/p-video
Premium AI video generation from text, images, and audio with fast generation times. See model inputs and outputs on Runpod Public Endpoints.
Pruna Video is a premium AI video generation model that creates videos from text prompts, images, or audio in under 10 seconds. It supports multiple resolutions up to 1080p, various aspect ratios, and optional audio conditioning for synchronized video generation.
Test Pruna Video in the Runpod Hub playground.
| | |
| ------------ | ------------------------------------------ |
| **Endpoint** | `https://api.runpod.ai/v2/p-video/runsync` |
| **Pricing** | \$0.02/s (720p), \$0.04/s (1080p) |
| **Type** | Video generation |
## Request
All parameters are passed within the `input` object in the request body.
Text description of the desired video content.
URL of an input image for image-to-video generation. Supports jpg, jpeg, png, webp. When provided, `aspect_ratio` is ignored.
URL of an audio file for audio-conditioned generation. Supports flac, mp3, wav. When provided, `duration` is ignored and the video matches audio length.
Video duration in seconds (1-10). Ignored when audio is provided.
Video resolution. Options: `720p`, `1080p`.
Frames per second. Options: `24`, `48`.
Output aspect ratio. Options: `16:9`, `9:16`, `4:3`, `3:4`, `3:2`, `2:3`, `1:1`. Ignored when image is provided.
Seed for reproducible generation.
Enable draft mode for faster, lower-quality preview. Reduces cost by 75%.
Include audio in the output video.
Automatically enhance the prompt for better results.
```bash cURL theme={"theme":{"light":"github-light","dark":"github-dark"}}
curl -X POST "https://api.runpod.ai/v2/p-video/runsync" \
-H "Authorization: Bearer $RUNPOD_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"input": {
"prompt": "A timelapse of clouds moving over a mountain range at sunset",
"duration": 5,
"resolution": "720p",
"aspect_ratio": "16:9"
}
}'
```
```python Python theme={"theme":{"light":"github-light","dark":"github-dark"}}
import requests
response = requests.post(
"https://api.runpod.ai/v2/p-video/runsync",
headers={
"Authorization": f"Bearer {RUNPOD_API_KEY}",
"Content-Type": "application/json",
},
json={
"input": {
"prompt": "A timelapse of clouds moving over a mountain range at sunset",
"duration": 5,
"resolution": "720p",
"aspect_ratio": "16:9",
}
},
)
result = response.json()
print(result["output"]["video_url"])
```
```javascript JavaScript theme={"theme":{"light":"github-light","dark":"github-dark"}}
const response = await fetch(
"https://api.runpod.ai/v2/p-video/runsync",
{
method: "POST",
headers: {
Authorization: `Bearer ${RUNPOD_API_KEY}`,
"Content-Type": "application/json",
},
body: JSON.stringify({
input: {
prompt: "A timelapse of clouds moving over a mountain range at sunset",
duration: 5,
resolution: "720p",
aspect_ratio: "16:9",
},
}),
}
);
const result = await response.json();
console.log(result.output.video_url);
```
## Response
Unique identifier for the request.
Request status. Returns `COMPLETED` on success, `FAILED` on error.
Time in milliseconds the request spent in queue before processing began.
Time in milliseconds the model took to generate the video.
The generation result containing the video URL and cost.
URL of the generated video. This URL expires after 7 days.
Cost of the generation in USD.
```json 200 theme={"theme":{"light":"github-light","dark":"github-dark"}}
{
"id": "sync-a1b2c3d4-e5f6-7890-abcd-ef1234567890-u1",
"status": "COMPLETED",
"delayTime": 15,
"executionTime": 8542,
"output": {
"video_url": "https://video.runpod.ai/abc123/output.mp4",
"cost": 0.10
}
}
```
```json 400 theme={"theme":{"light":"github-light","dark":"github-dark"}}
{
"id": "sync-a1b2c3d4-e5f6-7890-abcd-ef1234567890-u1",
"status": "FAILED",
"error": "Invalid audio format"
}
```
Video URLs expire after 7 days. Download and store generated videos immediately if you need to keep them.
## Cost calculation
Pruna Video pricing varies by resolution and draft mode:
| Resolution | Standard | Draft mode |
| ---------- | ----------------- | ------------------ |
| 720p | \$0.02 per second | \$0.005 per second |
| 1080p | \$0.04 per second | \$0.01 per second |
Example costs (standard mode):
| Resolution | 5 seconds | 10 seconds |
| ---------- | --------- | ---------- |
| 720p | \$0.10 | \$0.20 |
| 1080p | \$0.20 | \$0.40 |
# Qwen Image
Source: https://docs.runpod.io/public-endpoints/models/qwen-image
Image generation foundation model with advanced text rendering capabilities. Explore this model's inputs and outputs on Runpod Public Endpoints.
Qwen Image is an image generation foundation model with advanced text rendering capabilities. It excels at generating images that include readable, well-formed text within the image.
Test Qwen Image in the Runpod Hub playground.
| | |
| ------------ | ------------------------------------------------- |
| **Endpoint** | `https://api.runpod.ai/v2/qwen-image-t2i/runsync` |
| **Pricing** | \$0.02 per image |
| **Type** | Image generation |
## Request
All parameters are passed within the `input` object in the request body.
Text description of the desired image.
Elements to exclude from the generated image.
Image dimensions in format "width\*height".
Seed for reproducible results. Set to -1 for random.
Enable content safety checking.
```bash cURL theme={"theme":{"light":"github-light","dark":"github-dark"}}
curl -X POST "https://api.runpod.ai/v2/qwen-image-t2i/runsync" \
-H "Authorization: Bearer $RUNPOD_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"input": {
"prompt": "A fashion-forward woman sitting at cobblestone street in Paris",
"negative_prompt": "",
"size": "1328*1328",
"seed": -1,
"enable_safety_checker": true
}
}'
```
```python Python theme={"theme":{"light":"github-light","dark":"github-dark"}}
import requests
response = requests.post(
"https://api.runpod.ai/v2/qwen-image-t2i/runsync",
headers={
"Authorization": f"Bearer {RUNPOD_API_KEY}",
"Content-Type": "application/json",
},
json={
"input": {
"prompt": "A fashion-forward woman sitting at cobblestone street in Paris",
"negative_prompt": "",
"size": "1328*1328",
"seed": -1,
"enable_safety_checker": True,
}
},
)
result = response.json()
print(result["output"]["image_url"])
```
```javascript JavaScript theme={"theme":{"light":"github-light","dark":"github-dark"}}
const response = await fetch(
"https://api.runpod.ai/v2/qwen-image-t2i/runsync",
{
method: "POST",
headers: {
Authorization: `Bearer ${RUNPOD_API_KEY}`,
"Content-Type": "application/json",
},
body: JSON.stringify({
input: {
prompt: "A fashion-forward woman sitting at cobblestone street in Paris",
negative_prompt: "",
size: "1328*1328",
seed: -1,
enable_safety_checker: true,
},
}),
}
);
const result = await response.json();
console.log(result.output.image_url);
```
## Response
Unique identifier for the request.
Request status. Returns `COMPLETED` on success, `FAILED` on error.
Time in milliseconds the request spent in queue before processing began.
Time in milliseconds the model took to generate the image.
Identifier of the worker that processed the request.
The generation result containing the image URL and cost.
URL of the generated image. This URL expires after 7 days.
Cost of the generation in USD, calculated based on the output megapixels.
```json 200 theme={"theme":{"light":"github-light","dark":"github-dark"}}
{
"id": "sync-a1b2c3d4-e5f6-7890-abcd-ef1234567890-u1",
"status": "COMPLETED",
"delayTime": 15,
"executionTime": 3254,
"workerId": "oqk7ao1uomckye",
"output": {
"image_url": "https://image.runpod.ai/abc123/output.png",
"cost": 0.035267
}
}
```
```json 400 theme={"theme":{"light":"github-light","dark":"github-dark"}}
{
"id": "sync-a1b2c3d4-e5f6-7890-abcd-ef1234567890-u1",
"status": "FAILED",
"error": "Invalid parameter: size format must be width*height"
}
```
Image URLs expire after 7 days. Download and store generated images immediately if you need to keep them.
## Cost calculation
Qwen Image charges \$0.02 per image generated.
# Qwen Image Edit
Source: https://docs.runpod.io/public-endpoints/models/qwen-image-edit
Image editing with unique text rendering capabilities. Review model inputs, output formats, and request guidance for Runpod Public Endpoints.
Qwen Image Edit extends Qwen's advanced text rendering capabilities to image editing tasks. It excels at making precise edits to existing images while preserving quality and adding or modifying text within images.
Test Qwen Image Edit in the Runpod Hub playground.
| | |
| ------------ | -------------------------------------------------- |
| **Endpoint** | `https://api.runpod.ai/v2/qwen-image-edit/runsync` |
| **Pricing** | \$0.02 per image |
| **Type** | Image editing |
## Request
All parameters are passed within the `input` object in the request body.
Editing instructions describing the desired changes.
URL of the image to edit.
Elements to exclude from the edited image.
Seed for reproducible results. Set to -1 for random.
Output format. Accepts `png` or `jpeg`.
Enable content safety checking.
```bash cURL theme={"theme":{"light":"github-light","dark":"github-dark"}}
curl -X POST "https://api.runpod.ai/v2/qwen-image-edit/runsync" \
-H "Authorization: Bearer $RUNPOD_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"input": {
"prompt": "change the trench coat and high heels color to light grey",
"negative_prompt": "",
"seed": -1,
"image": "https://example.com/fashion-photo.png",
"output_format": "png",
"enable_safety_checker": true
}
}'
```
```python Python theme={"theme":{"light":"github-light","dark":"github-dark"}}
import requests
response = requests.post(
"https://api.runpod.ai/v2/qwen-image-edit/runsync",
headers={
"Authorization": f"Bearer {RUNPOD_API_KEY}",
"Content-Type": "application/json",
},
json={
"input": {
"prompt": "change the trench coat and high heels color to light grey",
"negative_prompt": "",
"seed": -1,
"image": "https://example.com/fashion-photo.png",
"output_format": "png",
"enable_safety_checker": True,
}
},
)
result = response.json()
print(result["output"]["image_url"])
```
```javascript JavaScript theme={"theme":{"light":"github-light","dark":"github-dark"}}
const response = await fetch(
"https://api.runpod.ai/v2/qwen-image-edit/runsync",
{
method: "POST",
headers: {
Authorization: `Bearer ${RUNPOD_API_KEY}`,
"Content-Type": "application/json",
},
body: JSON.stringify({
input: {
prompt: "change the trench coat and high heels color to light grey",
negative_prompt: "",
seed: -1,
image: "https://example.com/fashion-photo.png",
output_format: "png",
enable_safety_checker: true,
},
}),
}
);
const result = await response.json();
console.log(result.output.image_url);
```
## Response
Unique identifier for the request.
Request status. Returns `COMPLETED` on success, `FAILED` on error.
Time in milliseconds the request spent in queue before processing began.
Time in milliseconds the model took to edit the image.
Identifier of the worker that processed the request.
The generation result containing the image URL and cost.
URL of the edited image. This URL expires after 7 days.
Cost of the generation in USD, calculated based on the output megapixels.
```json 200 theme={"theme":{"light":"github-light","dark":"github-dark"}}
{
"id": "sync-a1b2c3d4-e5f6-7890-abcd-ef1234567890-u1",
"status": "COMPLETED",
"delayTime": 14,
"executionTime": 3875,
"workerId": "oqk7ao1uomckye",
"output": {
"image_url": "https://image.runpod.ai/abc123/output.png",
"cost": 0.02097152
}
}
```
```json 400 theme={"theme":{"light":"github-light","dark":"github-dark"}}
{
"id": "sync-a1b2c3d4-e5f6-7890-abcd-ef1234567890-u1",
"status": "FAILED",
"error": "Invalid image URL"
}
```
Image URLs expire after 7 days. Download and store edited images immediately if you need to keep them.
## Cost calculation
Qwen Image Edit charges \$0.02 per image edited.
# Qwen Image Edit 2511
Source: https://docs.runpod.io/public-endpoints/models/qwen-image-edit-2511
Advanced image editing with strong consistency and multi-person identity preservation. See model inputs and outputs on Runpod Public Endpoints.
Qwen Image Edit 2511 delivers stronger edit consistency, robust multi-person identity and pose consistency, built-in LoRA styles, and enhanced industrial and product design capabilities.
Test Qwen Image Edit 2511 in the Runpod Hub playground.
| | |
| ------------ | ------------------------------------------------------- |
| **Endpoint** | `https://api.runpod.ai/v2/qwen-image-edit-2511/runsync` |
| **Pricing** | \$0.02 per image |
| **Type** | Image editing |
## Request
All parameters are passed within the `input` object in the request body.
Text instructions describing the desired edits.
Array of 1-3 image URLs to edit.
Output dimensions. Options: `1024*1024`, `1024*1280`, `1280*1024`, `1280*1280`, `1280*1536`, `1536*1080`.
Seed for reproducible results. Set to -1 for random.
Output format. Options: `jpeg`, `png`, `webp`.
```bash cURL theme={"theme":{"light":"github-light","dark":"github-dark"}}
curl -X POST "https://api.runpod.ai/v2/qwen-image-edit-2511/runsync" \
-H "Authorization: Bearer $RUNPOD_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"input": {
"prompt": "Change the background to a futuristic cityscape",
"images": ["https://example.com/portrait.jpg"],
"size": "1024*1024",
"seed": -1,
"output_format": "png"
}
}'
```
```python Python theme={"theme":{"light":"github-light","dark":"github-dark"}}
import requests
response = requests.post(
"https://api.runpod.ai/v2/qwen-image-edit-2511/runsync",
headers={
"Authorization": f"Bearer {RUNPOD_API_KEY}",
"Content-Type": "application/json",
},
json={
"input": {
"prompt": "Change the background to a futuristic cityscape",
"images": ["https://example.com/portrait.jpg"],
"size": "1024*1024",
"seed": -1,
"output_format": "png",
}
},
)
result = response.json()
print(result["output"]["image_url"])
```
```javascript JavaScript theme={"theme":{"light":"github-light","dark":"github-dark"}}
const response = await fetch(
"https://api.runpod.ai/v2/qwen-image-edit-2511/runsync",
{
method: "POST",
headers: {
Authorization: `Bearer ${RUNPOD_API_KEY}`,
"Content-Type": "application/json",
},
body: JSON.stringify({
input: {
prompt: "Change the background to a futuristic cityscape",
images: ["https://example.com/portrait.jpg"],
size: "1024*1024",
seed: -1,
output_format: "png",
},
}),
}
);
const result = await response.json();
console.log(result.output.image_url);
```
## Response
Unique identifier for the request.
Request status. Returns `COMPLETED` on success, `FAILED` on error.
The generation result containing the image URL and cost.
URL of the edited image. This URL expires after 7 days.
Cost of the generation in USD.
```json 200 theme={"theme":{"light":"github-light","dark":"github-dark"}}
{
"id": "sync-a1b2c3d4-e5f6-7890-abcd-ef1234567890-u1",
"status": "COMPLETED",
"delayTime": 15,
"executionTime": 3856,
"output": {
"image_url": "https://image.runpod.ai/abc123/output.png",
"cost": 0.02
}
}
```
Image URLs expire after 7 days. Download and store edited images immediately if you need to keep them.
## Cost calculation
Qwen Image Edit 2511 charges \$0.02 per image edited.
# Qwen Image Edit 2511 LoRA
Source: https://docs.runpod.io/public-endpoints/models/qwen-image-edit-2511-lora
Advanced image editing with complex text rendering and LoRA support. Review inputs and output formats for this model on Runpod Public Endpoints.
Qwen Image Edit 2511 LoRA achieves significant advances in complex text rendering and precise image editing with LoRA support. It enables style customization through LoRA models while maintaining editing precision.
Test Qwen Image Edit 2511 LoRA in the Runpod Hub playground.
| | |
| ------------ | ------------------------------------------------------------ |
| **Endpoint** | `https://api.runpod.ai/v2/qwen-image-edit-2511-lora/runsync` |
| **Pricing** | \$0.025 per image |
| **Type** | Image editing |
## Request
All parameters are passed within the `input` object in the request body.
Text instructions describing the desired edits.
Array of 1-3 image URLs to edit.
Array of LoRA configurations to apply.
URL or path to the LoRA model file.
Scale factor for the LoRA influence.
Output dimensions. Options: `1024*1024`, `1024*1280`, `1280*1024`, `1280*1280`, `1280*1536`, `1536*1080`.
Seed for reproducible results. Set to -1 for random.
Output format. Options: `jpeg`, `png`, `webp`.
```bash cURL theme={"theme":{"light":"github-light","dark":"github-dark"}}
curl -X POST "https://api.runpod.ai/v2/qwen-image-edit-2511-lora/runsync" \
-H "Authorization: Bearer $RUNPOD_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"input": {
"prompt": "Transform into anime style",
"images": ["https://example.com/photo.jpg"],
"loras": [
{
"path": "https://huggingface.co/example/anime-lora/model.safetensors",
"scale": 0.8
}
],
"size": "1024*1024",
"seed": -1,
"output_format": "png"
}
}'
```
```python Python theme={"theme":{"light":"github-light","dark":"github-dark"}}
import requests
response = requests.post(
"https://api.runpod.ai/v2/qwen-image-edit-2511-lora/runsync",
headers={
"Authorization": f"Bearer {RUNPOD_API_KEY}",
"Content-Type": "application/json",
},
json={
"input": {
"prompt": "Transform into anime style",
"images": ["https://example.com/photo.jpg"],
"loras": [
{
"path": "https://huggingface.co/example/anime-lora/model.safetensors",
"scale": 0.8,
}
],
"size": "1024*1024",
"seed": -1,
"output_format": "png",
}
},
)
result = response.json()
print(result["output"]["image_url"])
```
```javascript JavaScript theme={"theme":{"light":"github-light","dark":"github-dark"}}
const response = await fetch(
"https://api.runpod.ai/v2/qwen-image-edit-2511-lora/runsync",
{
method: "POST",
headers: {
Authorization: `Bearer ${RUNPOD_API_KEY}`,
"Content-Type": "application/json",
},
body: JSON.stringify({
input: {
prompt: "Transform into anime style",
images: ["https://example.com/photo.jpg"],
loras: [
{
path: "https://huggingface.co/example/anime-lora/model.safetensors",
scale: 0.8,
},
],
size: "1024*1024",
seed: -1,
output_format: "png",
},
}),
}
);
const result = await response.json();
console.log(result.output.image_url);
```
## Response
Unique identifier for the request.
Request status. Returns `COMPLETED` on success, `FAILED` on error.
The generation result containing the image URL and cost.
URL of the edited image. This URL expires after 7 days.
Cost of the generation in USD.
```json 200 theme={"theme":{"light":"github-light","dark":"github-dark"}}
{
"id": "sync-a1b2c3d4-e5f6-7890-abcd-ef1234567890-u1",
"status": "COMPLETED",
"delayTime": 18,
"executionTime": 4521,
"output": {
"image_url": "https://image.runpod.ai/abc123/output.png",
"cost": 0.025
}
}
```
Image URLs expire after 7 days. Download and store edited images immediately if you need to keep them.
## Cost calculation
Qwen Image Edit 2511 LoRA charges \$0.025 per image edited.
# Qwen Image LoRA
Source: https://docs.runpod.io/public-endpoints/models/qwen-image-lora
Image generation with LoRA support and advanced text rendering. Review inputs and output formats for this model on Runpod Public Endpoints.
Qwen Image LoRA extends the base Qwen Image model with LoRA support, allowing you to customize generation with fine-tuned LoRA models. It retains the advanced text rendering capabilities of Qwen Image while enabling style customization.
Test Qwen Image LoRA in the Runpod Hub playground.
| | |
| ------------ | ------------------------------------------------------ |
| **Endpoint** | `https://api.runpod.ai/v2/qwen-image-t2i-lora/runsync` |
| **Pricing** | \$0.025 per image |
| **Type** | Image generation |
## Request
All parameters are passed within the `input` object in the request body.
Text description of the desired image.
Array of LoRA configurations to apply.
URL or path to the LoRA model file.
Scale factor for the LoRA influence. Typically between 0 and 1.
Image dimensions in format "width\*height".
Seed for reproducible results. Set to -1 for random.
Enable content safety checking.
```bash cURL theme={"theme":{"light":"github-light","dark":"github-dark"}}
curl -X POST "https://api.runpod.ai/v2/qwen-image-t2i-lora/runsync" \
-H "Authorization: Bearer $RUNPOD_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"input": {
"prompt": "Real life Anime in a cozy kitchen",
"loras": [
{
"path": "https://huggingface.co/flymy-ai/qwen-image-anime-irl-lora/resolve/main/flymy_anime_irl.safetensors",
"scale": 1
}
],
"size": "1024*1024",
"seed": -1,
"enable_safety_checker": true
}
}'
```
```python Python theme={"theme":{"light":"github-light","dark":"github-dark"}}
import requests
response = requests.post(
"https://api.runpod.ai/v2/qwen-image-t2i-lora/runsync",
headers={
"Authorization": f"Bearer {RUNPOD_API_KEY}",
"Content-Type": "application/json",
},
json={
"input": {
"prompt": "Real life Anime in a cozy kitchen",
"loras": [
{
"path": "https://huggingface.co/flymy-ai/qwen-image-anime-irl-lora/resolve/main/flymy_anime_irl.safetensors",
"scale": 1,
}
],
"size": "1024*1024",
"seed": -1,
"enable_safety_checker": True,
}
},
)
result = response.json()
print(result["output"]["image_url"])
```
```javascript JavaScript theme={"theme":{"light":"github-light","dark":"github-dark"}}
const response = await fetch(
"https://api.runpod.ai/v2/qwen-image-t2i-lora/runsync",
{
method: "POST",
headers: {
Authorization: `Bearer ${RUNPOD_API_KEY}`,
"Content-Type": "application/json",
},
body: JSON.stringify({
input: {
prompt: "Real life Anime in a cozy kitchen",
loras: [
{
path: "https://huggingface.co/flymy-ai/qwen-image-anime-irl-lora/resolve/main/flymy_anime_irl.safetensors",
scale: 1,
},
],
size: "1024*1024",
seed: -1,
enable_safety_checker: true,
},
}),
}
);
const result = await response.json();
console.log(result.output.image_url);
```
## Response
Unique identifier for the request.
Request status. Returns `COMPLETED` on success, `FAILED` on error.
Time in milliseconds the request spent in queue before processing began.
Time in milliseconds the model took to generate the image.
Identifier of the worker that processed the request.
The generation result containing the image URL and cost.
URL of the generated image. This URL expires after 7 days.
Cost of the generation in USD, calculated based on the output megapixels.
```json 200 theme={"theme":{"light":"github-light","dark":"github-dark"}}
{
"id": "sync-a1b2c3d4-e5f6-7890-abcd-ef1234567890-u1",
"status": "COMPLETED",
"delayTime": 20,
"executionTime": 4125,
"workerId": "oqk7ao1uomckye",
"output": {
"image_url": "https://image.runpod.ai/abc123/output.png",
"cost": 0.02097152
}
}
```
```json 400 theme={"theme":{"light":"github-light","dark":"github-dark"}}
{
"id": "sync-a1b2c3d4-e5f6-7890-abcd-ef1234567890-u1",
"status": "FAILED",
"error": "Failed to load LoRA model from URL"
}
```
Image URLs expire after 7 days. Download and store generated images immediately if you need to keep them.
## Cost calculation
Qwen Image LoRA charges \$0.025 per image generated.
# Qwen3 32B AWQ
Source: https://docs.runpod.io/public-endpoints/models/qwen3-32b
Latest generation LLM with advanced reasoning, instruction-following, and multilingual support. See model inputs and outputs on Runpod Public Endpoints.
Qwen3 32B AWQ is the latest large language model in the Qwen series, offering advancements in reasoning, instruction-following, agent capabilities, and multilingual support. It uses AWQ quantization for efficient inference while maintaining high quality.
Test Qwen3 32B AWQ in the Runpod Hub playground.
| | |
| ------------ | ------------------------------------------------ |
| **Endpoint** | `https://api.runpod.ai/v2/qwen3-32b-awq/runsync` |
| **Pricing** | \$10.00 per 1M tokens |
| **Type** | Text generation |
This endpoint is fully compatible with the OpenAI API. See the [OpenAI compatibility examples](#openai-api-compatibility) below.
## Request
All parameters are passed within the `input` object in the request body.
Prompt for text generation.
Maximum number of tokens to output.
Randomness of the output. Lower values make output more predictable and deterministic. Range: 0.0-1.0.
Nucleus sampling threshold. Samples from the smallest set of words whose cumulative probability exceeds this threshold.
Restricts sampling to the top K most probable words.
Stops generation if the given string is encountered.
```bash cURL theme={"theme":{"light":"github-light","dark":"github-dark"}}
curl -X POST "https://api.runpod.ai/v2/qwen3-32b-awq/runsync" \
-H "Authorization: Bearer $RUNPOD_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"input": {
"prompt": "Write a Python function that checks if a number is prime:",
"max_tokens": 512,
"temperature": 0.7
}
}'
```
```python Python theme={"theme":{"light":"github-light","dark":"github-dark"}}
import requests
response = requests.post(
"https://api.runpod.ai/v2/qwen3-32b-awq/runsync",
headers={
"Authorization": f"Bearer {RUNPOD_API_KEY}",
"Content-Type": "application/json",
},
json={
"input": {
"prompt": "Write a Python function that checks if a number is prime:",
"max_tokens": 512,
"temperature": 0.7,
}
},
)
result = response.json()
print(result["output"])
```
```javascript JavaScript theme={"theme":{"light":"github-light","dark":"github-dark"}}
const response = await fetch(
"https://api.runpod.ai/v2/qwen3-32b-awq/runsync",
{
method: "POST",
headers: {
Authorization: `Bearer ${RUNPOD_API_KEY}`,
"Content-Type": "application/json",
},
body: JSON.stringify({
input: {
prompt: "Write a Python function that checks if a number is prime:",
max_tokens: 512,
temperature: 0.7,
},
}),
}
);
const result = await response.json();
console.log(result.output);
```
## Response
Unique identifier for the request.
Request status. Returns `COMPLETED` on success, `FAILED` on error.
Time in milliseconds the request spent in queue before processing began.
Time in milliseconds the model took to generate the response.
Identifier of the worker that processed the request.
The generation result containing the text and usage information.
Array containing the generated text.
Cost of the generation in USD.
Token usage information with `input` and `output` counts.
```json 200 theme={"theme":{"light":"github-light","dark":"github-dark"}}
{
"delayTime": 25,
"executionTime": 3153,
"id": "sync-0f3288b5-58e8-46fd-ba73-53945f5e8982-u2",
"output": [
{
"choices": [
{
"tokens": [
"def is_prime(n):\n if n <= 1:\n return False\n for i in range(2, int(n**0.5) + 1):\n if n % i == 0:\n return False\n return True"
]
}
],
"cost": 0.0001,
"usage": {
"input": 10,
"output": 100
}
}
],
"status": "COMPLETED",
"workerId": "pkej0t9bbyjrgy"
}
```
```json 400 theme={"theme":{"light":"github-light","dark":"github-dark"}}
{
"id": "sync-a1b2c3d4-e5f6-7890-abcd-ef1234567890-u1",
"status": "FAILED",
"error": "Invalid prompt"
}
```
## OpenAI API compatibility
Qwen3 32B AWQ is fully compatible with the OpenAI API format. You can use the OpenAI Python client to interact with this endpoint.
```python Python (OpenAI SDK) theme={"theme":{"light":"github-light","dark":"github-dark"}}
from openai import OpenAI
client = OpenAI(
api_key=RUNPOD_API_KEY,
base_url="https://api.runpod.ai/v2/qwen3-32b-awq/openai/v1",
)
response = client.chat.completions.create(
model="Qwen/Qwen3-32B-AWQ",
messages=[
{
"role": "system",
"content": "You are a helpful coding assistant.",
},
{
"role": "user",
"content": "Write a Python function that checks if a number is prime.",
},
],
max_tokens=525,
)
print(response.choices[0].message.content)
```
For streaming responses, add `stream=True`:
```python Python (Streaming) theme={"theme":{"light":"github-light","dark":"github-dark"}}
response = client.chat.completions.create(
model="Qwen/Qwen3-32B-AWQ",
messages=[
{"role": "user", "content": "Explain quantum computing in simple terms."}
],
max_tokens=525,
stream=True,
)
for chunk in response:
if chunk.choices[0].delta.content:
print(chunk.choices[0].delta.content, end="")
```
For more details, see [Send vLLM requests](/serverless/vllm/vllm-requests) and the [OpenAI API compatibility guide](/serverless/vllm/openai-compatibility).
## Cost calculation
Qwen3 32B AWQ charges \$10.00 per 1M tokens. Example costs:
| Tokens | Cost |
| ---------------- | ------- |
| 1,000 tokens | \$0.01 |
| 10,000 tokens | \$0.10 |
| 100,000 tokens | \$1.00 |
| 1,000,000 tokens | \$10.00 |
# Seedance 1.5 Pro I2V
Source: https://docs.runpod.io/public-endpoints/models/seedance-1-5-pro
Cinematic image-to-video with expressive motion and stable aesthetics. Explore this model's inputs and outputs on Runpod Public Endpoints.
Seedance 1.5 Pro I2V generates cinematic, live-action-leaning clips from a text prompt. It preserves the image's subject and composition while adding expressive motion and stable aesthetics.
Test Seedance 1.5 Pro I2V in the Runpod Hub playground.
| | |
| ------------ | -------------------------------------------------------- |
| **Endpoint** | `https://api.runpod.ai/v2/seedance-v1-5-pro-i2v/runsync` |
| **Pricing** | \$0.024–\$0.052 per second |
| **Type** | Video generation |
## Request
All parameters are passed within the `input` object in the request body.
Text description of the desired video motion and content.
URL of the source image to animate.
URL of an optional ending frame image.
Video duration in seconds. Range: 4-12 seconds.
Output resolution. Options: `480p`, `720p`.
Aspect ratio. Options: `21:9`, `16:9`, `9:16`, `1:1`, `4:3`, `3:4`.
Keep camera position fixed during video.
Generate accompanying audio.
Seed for reproducible results. Set to -1 for random.
```bash cURL theme={"theme":{"light":"github-light","dark":"github-dark"}}
curl -X POST "https://api.runpod.ai/v2/seedance-v1-5-pro-i2v/runsync" \
-H "Authorization: Bearer $RUNPOD_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"input": {
"prompt": "The character slowly turns and smiles at the camera",
"image": "https://example.com/portrait.jpg",
"duration": 5,
"resolution": "720p",
"aspect_ratio": "16:9",
"seed": -1
}
}'
```
```python Python theme={"theme":{"light":"github-light","dark":"github-dark"}}
import requests
response = requests.post(
"https://api.runpod.ai/v2/seedance-v1-5-pro-i2v/runsync",
headers={
"Authorization": f"Bearer {RUNPOD_API_KEY}",
"Content-Type": "application/json",
},
json={
"input": {
"prompt": "The character slowly turns and smiles at the camera",
"image": "https://example.com/portrait.jpg",
"duration": 5,
"resolution": "720p",
"aspect_ratio": "16:9",
"seed": -1,
}
},
)
result = response.json()
print(result["output"]["video_url"])
```
```javascript JavaScript theme={"theme":{"light":"github-light","dark":"github-dark"}}
const response = await fetch(
"https://api.runpod.ai/v2/seedance-v1-5-pro-i2v/runsync",
{
method: "POST",
headers: {
Authorization: `Bearer ${RUNPOD_API_KEY}`,
"Content-Type": "application/json",
},
body: JSON.stringify({
input: {
prompt: "The character slowly turns and smiles at the camera",
image: "https://example.com/portrait.jpg",
duration: 5,
resolution: "720p",
aspect_ratio: "16:9",
seed: -1,
},
}),
}
);
const result = await response.json();
console.log(result.output.video_url);
```
## Response
Unique identifier for the request.
Request status. Returns `COMPLETED` on success, `FAILED` on error.
The generation result containing the video URL and cost.
URL of the generated video. This URL expires after 7 days.
Cost of the generation in USD.
```json 200 theme={"theme":{"light":"github-light","dark":"github-dark"}}
{
"id": "sync-a1b2c3d4-e5f6-7890-abcd-ef1234567890-u1",
"status": "COMPLETED",
"delayTime": 25,
"executionTime": 65432,
"output": {
"video_url": "https://video.runpod.ai/abc123/output.mp4",
"cost": 0.26
}
}
```
Video URLs expire after 7 days. Download and store generated videos immediately if you need to keep them.
## Cost calculation
Seedance 1.5 Pro I2V charges per second of video:
| Resolution | Cost per second |
| ---------- | --------------- |
| 480p | \$0.024 |
| 720p | \$0.052 |
Example: A 5-second 720p video costs \$0.26.
# Seedream 4.0 Edit
Source: https://docs.runpod.io/public-endpoints/models/seedream-4-edit
New-generation image editing with unified generation and editing architecture. See model inputs and outputs on Runpod Public Endpoints.
Seedream 4.0 Edit provides advanced image editing capabilities using the same unified architecture as Seedream 4.0 T2I. It can edit or combine multiple source images based on text instructions.
Test Seedream 4.0 Edit in the Runpod Hub playground.
| | |
| ------------ | --------------------------------------------------- |
| **Endpoint** | `https://api.runpod.ai/v2/seedream-v4-edit/runsync` |
| **Pricing** | \$0.027 per image |
| **Type** | Image editing |
## Request
All parameters are passed within the `input` object in the request body.
Editing instructions describing the desired transformation.
Array of image URLs to edit or combine.
Output image dimensions in format "width\*height".
Enable content safety checking.
```bash cURL theme={"theme":{"light":"github-light","dark":"github-dark"}}
curl -X POST "https://api.runpod.ai/v2/seedream-v4-edit/runsync" \
-H "Authorization: Bearer $RUNPOD_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"input": {
"prompt": "Dress the model in the clothes and hat",
"images": [
"https://example.com/model-photo.jpg",
"https://example.com/clothes.jpg"
],
"size": "1024*1024",
"enable_safety_checker": true
}
}'
```
```python Python theme={"theme":{"light":"github-light","dark":"github-dark"}}
import requests
response = requests.post(
"https://api.runpod.ai/v2/seedream-v4-edit/runsync",
headers={
"Authorization": f"Bearer {RUNPOD_API_KEY}",
"Content-Type": "application/json",
},
json={
"input": {
"prompt": "Dress the model in the clothes and hat",
"images": [
"https://example.com/model-photo.jpg",
"https://example.com/clothes.jpg",
],
"size": "1024*1024",
"enable_safety_checker": True,
}
},
)
result = response.json()
print(result["output"]["image_url"])
```
```javascript JavaScript theme={"theme":{"light":"github-light","dark":"github-dark"}}
const response = await fetch(
"https://api.runpod.ai/v2/seedream-v4-edit/runsync",
{
method: "POST",
headers: {
Authorization: `Bearer ${RUNPOD_API_KEY}`,
"Content-Type": "application/json",
},
body: JSON.stringify({
input: {
prompt: "Dress the model in the clothes and hat",
images: [
"https://example.com/model-photo.jpg",
"https://example.com/clothes.jpg",
],
size: "1024*1024",
enable_safety_checker: true,
},
}),
}
);
const result = await response.json();
console.log(result.output.image_url);
```
## Response
Unique identifier for the request.
Request status. Returns `COMPLETED` on success, `FAILED` on error.
Time in milliseconds the request spent in queue before processing began.
Time in milliseconds the model took to edit the image.
Identifier of the worker that processed the request.
The generation result containing the image URL and cost.
URL of the edited image. This URL expires after 7 days.
Cost of the generation in USD, calculated based on the output megapixels.
```json 200 theme={"theme":{"light":"github-light","dark":"github-dark"}}
{
"id": "sync-a1b2c3d4-e5f6-7890-abcd-ef1234567890-u1",
"status": "COMPLETED",
"delayTime": 19,
"executionTime": 4856,
"workerId": "oqk7ao1uomckye",
"output": {
"image_url": "https://image.runpod.ai/abc123/output.png",
"cost": 0.028311
}
}
```
```json 400 theme={"theme":{"light":"github-light","dark":"github-dark"}}
{
"id": "sync-a1b2c3d4-e5f6-7890-abcd-ef1234567890-u1",
"status": "FAILED",
"error": "Invalid image URL in images array"
}
```
Image URLs expire after 7 days. Download and store edited images immediately if you need to keep them.
## Cost calculation
Seedream 4.0 Edit charges \$0.027 per image edited.
# Seedream 4.0 T2I
Source: https://docs.runpod.io/public-endpoints/models/seedream-4-t2i
New-generation image creation with unified generation and editing architecture. See model inputs and outputs on Runpod Public Endpoints.
Seedream 4.0 T2I is ByteDance's new-generation image creation model that integrates both generation and editing capabilities within a unified architecture. It produces high-quality images with excellent prompt adherence.
Test Seedream 4.0 T2I in the Runpod Hub playground.
| | |
| ------------ | -------------------------------------------------- |
| **Endpoint** | `https://api.runpod.ai/v2/seedream-v4-t2i/runsync` |
| **Pricing** | \$0.027 per image |
| **Type** | Image generation |
## Request
All parameters are passed within the `input` object in the request body.
Text description of the desired image.
Elements to exclude from the generated image.
Image dimensions in format "width\*height".
Seed for reproducible results. Set to -1 for random.
Enable content safety checking.
```bash cURL theme={"theme":{"light":"github-light","dark":"github-dark"}}
curl -X POST "https://api.runpod.ai/v2/seedream-v4-t2i/runsync" \
-H "Authorization: Bearer $RUNPOD_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"input": {
"prompt": "American retro 1950s illustration style, diner scene",
"negative_prompt": "",
"size": "2048*2048",
"seed": -1,
"enable_safety_checker": true
}
}'
```
```python Python theme={"theme":{"light":"github-light","dark":"github-dark"}}
import requests
response = requests.post(
"https://api.runpod.ai/v2/seedream-v4-t2i/runsync",
headers={
"Authorization": f"Bearer {RUNPOD_API_KEY}",
"Content-Type": "application/json",
},
json={
"input": {
"prompt": "American retro 1950s illustration style, diner scene",
"negative_prompt": "",
"size": "2048*2048",
"seed": -1,
"enable_safety_checker": True,
}
},
)
result = response.json()
print(result["output"]["image_url"])
```
```javascript JavaScript theme={"theme":{"light":"github-light","dark":"github-dark"}}
const response = await fetch(
"https://api.runpod.ai/v2/seedream-v4-t2i/runsync",
{
method: "POST",
headers: {
Authorization: `Bearer ${RUNPOD_API_KEY}`,
"Content-Type": "application/json",
},
body: JSON.stringify({
input: {
prompt: "American retro 1950s illustration style, diner scene",
negative_prompt: "",
size: "2048*2048",
seed: -1,
enable_safety_checker: true,
},
}),
}
);
const result = await response.json();
console.log(result.output.image_url);
```
## Response
Unique identifier for the request.
Request status. Returns `COMPLETED` on success, `FAILED` on error.
Time in milliseconds the request spent in queue before processing began.
Time in milliseconds the model took to generate the image.
Identifier of the worker that processed the request.
The generation result containing the image URL and cost.
URL of the generated image. This URL expires after 7 days.
Cost of the generation in USD, calculated based on the output megapixels.
```json 200 theme={"theme":{"light":"github-light","dark":"github-dark"}}
{
"id": "sync-a1b2c3d4-e5f6-7890-abcd-ef1234567890-u1",
"status": "COMPLETED",
"delayTime": 16,
"executionTime": 5234,
"workerId": "oqk7ao1uomckye",
"output": {
"image_url": "https://image.runpod.ai/abc123/output.png",
"cost": 0.113246
}
}
```
```json 400 theme={"theme":{"light":"github-light","dark":"github-dark"}}
{
"id": "sync-a1b2c3d4-e5f6-7890-abcd-ef1234567890-u1",
"status": "FAILED",
"error": "Invalid parameter: size format must be width*height"
}
```
Image URLs expire after 7 days. Download and store generated images immediately if you need to keep them.
## Cost calculation
Seedream 4.0 T2I charges \$0.027 per image generated.
# SORA 2 I2V
Source: https://docs.runpod.io/public-endpoints/models/sora-2
OpenAI's Sora 2 video and audio generation model. Review model inputs, output formats, and request guidance for Runpod Public Endpoints.
SORA 2 I2V is OpenAI's video and audio generation model that creates dynamic videos from static images. It excels at generating videos with complex actions, ambient sounds, and character dialogue based on detailed text prompts.
Test SORA 2 I2V in the Runpod Hub playground.
| | |
| ------------ | --------------------------------------------- |
| **Endpoint** | `https://api.runpod.ai/v2/sora-2-i2v/runsync` |
| **Pricing** | \$0.40–\$1.20 per video |
| **Type** | Video generation |
## Request
All parameters are passed within the `input` object in the request body.
Text description of the desired video, including action, ambient sound, and character dialogue.
URL of the source image to animate.
Video duration in seconds. Valid options: 4, 8, or 12.
```bash cURL theme={"theme":{"light":"github-light","dark":"github-dark"}}
curl -X POST "https://api.runpod.ai/v2/sora-2-i2v/runsync" \
-H "Authorization: Bearer $RUNPOD_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"input": {
"prompt": "Action: The mech slowly pushes itself up with a damaged mechanical arm, sparks flying. Ambient Sound: Distant explosions, electrical sizzle. Character Dialogue: (Processed mechanical voice) No retreat.",
"image": "https://example.com/mech.jpeg",
"duration": 4
}
}'
```
```python Python theme={"theme":{"light":"github-light","dark":"github-dark"}}
import requests
response = requests.post(
"https://api.runpod.ai/v2/sora-2-i2v/runsync",
headers={
"Authorization": f"Bearer {RUNPOD_API_KEY}",
"Content-Type": "application/json",
},
json={
"input": {
"prompt": "Action: The mech slowly pushes itself up with a damaged mechanical arm, sparks flying. Ambient Sound: Distant explosions, electrical sizzle. Character Dialogue: (Processed mechanical voice) No retreat.",
"image": "https://example.com/mech.jpeg",
"duration": 4,
}
},
)
result = response.json()
print(result["output"]["video_url"])
```
```javascript JavaScript theme={"theme":{"light":"github-light","dark":"github-dark"}}
const response = await fetch(
"https://api.runpod.ai/v2/sora-2-i2v/runsync",
{
method: "POST",
headers: {
Authorization: `Bearer ${RUNPOD_API_KEY}`,
"Content-Type": "application/json",
},
body: JSON.stringify({
input: {
prompt: "Action: The mech slowly pushes itself up with a damaged mechanical arm, sparks flying. Ambient Sound: Distant explosions, electrical sizzle. Character Dialogue: (Processed mechanical voice) No retreat.",
image: "https://example.com/mech.jpeg",
duration: 4,
},
}),
}
);
const result = await response.json();
console.log(result.output.video_url);
```
## Response
Unique identifier for the request.
Request status. Returns `COMPLETED` on success, `FAILED` on error.
Time in milliseconds the request spent in queue before processing began.
Time in milliseconds the model took to generate the video.
Identifier of the worker that processed the request.
The generation result containing the video URL and cost.
URL of the generated video. This URL expires after 7 days.
Cost of the generation in USD.
```json 200 theme={"theme":{"light":"github-light","dark":"github-dark"}}
{
"id": "sync-a1b2c3d4-e5f6-7890-abcd-ef1234567890-u1",
"status": "COMPLETED",
"delayTime": 35,
"executionTime": 120345,
"workerId": "oqk7ao1uomckye",
"output": {
"video_url": "https://video.runpod.ai/abc123/output.mp4",
"cost": 0.40
}
}
```
```json 400 theme={"theme":{"light":"github-light","dark":"github-dark"}}
{
"id": "sync-a1b2c3d4-e5f6-7890-abcd-ef1234567890-u1",
"status": "FAILED",
"error": "Invalid duration: must be 4, 8, or 12 seconds"
}
```
Video URLs expire after 7 days. Download and store generated videos immediately if you need to keep them.
## Prompt structure
For best results, structure your prompt with these components:
* **Action**: Describe what happens in the scene
* **Ambient Sound**: Describe background audio
* **Character Dialogue**: Include spoken lines with voice descriptions
## Cost calculation
SORA 2 I2V pricing varies by duration:
| Duration | Cost |
| ---------- | ------ |
| 4 seconds | \$0.40 |
| 8 seconds | \$0.80 |
| 12 seconds | \$1.20 |
# SORA 2 Pro I2V
Source: https://docs.runpod.io/public-endpoints/models/sora-2-pro
OpenAI's Sora 2 Pro professional-grade video and audio generation model. Explore this model's inputs and outputs on Runpod Public Endpoints.
SORA 2 Pro I2V is OpenAI's professional-grade video and audio generation model. It produces higher quality output than the standard SORA 2, with enhanced visual fidelity and more nuanced audio generation.
Test SORA 2 Pro I2V in the Runpod Hub playground.
| | |
| ------------ | ------------------------------------------------- |
| **Endpoint** | `https://api.runpod.ai/v2/sora-2-pro-i2v/runsync` |
| **Pricing** | \$1.20–\$6.00 per video |
| **Type** | Video generation |
## Request
All parameters are passed within the `input` object in the request body.
Text description of the desired video, including action, ambient sound, and character dialogue.
URL of the source image to animate.
Output video resolution.
Video duration in seconds. Valid options: 4, 8, or 12.
```bash cURL theme={"theme":{"light":"github-light","dark":"github-dark"}}
curl -X POST "https://api.runpod.ai/v2/sora-2-pro-i2v/runsync" \
-H "Authorization: Bearer $RUNPOD_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"input": {
"prompt": "Action: She opened her hands. Ambient Sound: The soft crackling of the dying fire, a happy ding from the timer. Character Dialogue: (Voice is high-pitched, bubbly) Welcome to my bakery!",
"image": "https://example.com/baker.jpeg",
"size": "720p",
"duration": 4
}
}'
```
```python Python theme={"theme":{"light":"github-light","dark":"github-dark"}}
import requests
response = requests.post(
"https://api.runpod.ai/v2/sora-2-pro-i2v/runsync",
headers={
"Authorization": f"Bearer {RUNPOD_API_KEY}",
"Content-Type": "application/json",
},
json={
"input": {
"prompt": "Action: She opened her hands. Ambient Sound: The soft crackling of the dying fire, a happy ding from the timer. Character Dialogue: (Voice is high-pitched, bubbly) Welcome to my bakery!",
"image": "https://example.com/baker.jpeg",
"size": "720p",
"duration": 4,
}
},
)
result = response.json()
print(result["output"]["video_url"])
```
```javascript JavaScript theme={"theme":{"light":"github-light","dark":"github-dark"}}
const response = await fetch(
"https://api.runpod.ai/v2/sora-2-pro-i2v/runsync",
{
method: "POST",
headers: {
Authorization: `Bearer ${RUNPOD_API_KEY}`,
"Content-Type": "application/json",
},
body: JSON.stringify({
input: {
prompt: "Action: She opened her hands. Ambient Sound: The soft crackling of the dying fire, a happy ding from the timer. Character Dialogue: (Voice is high-pitched, bubbly) Welcome to my bakery!",
image: "https://example.com/baker.jpeg",
size: "720p",
duration: 4,
},
}),
}
);
const result = await response.json();
console.log(result.output.video_url);
```
## Response
Unique identifier for the request.
Request status. Returns `COMPLETED` on success, `FAILED` on error.
Time in milliseconds the request spent in queue before processing began.
Time in milliseconds the model took to generate the video.
Identifier of the worker that processed the request.
The generation result containing the video URL and cost.
URL of the generated video. This URL expires after 7 days.
Cost of the generation in USD.
```json 200 theme={"theme":{"light":"github-light","dark":"github-dark"}}
{
"id": "sync-a1b2c3d4-e5f6-7890-abcd-ef1234567890-u1",
"status": "COMPLETED",
"delayTime": 42,
"executionTime": 185432,
"workerId": "oqk7ao1uomckye",
"output": {
"video_url": "https://video.runpod.ai/abc123/output.mp4",
"cost": 1.20
}
}
```
```json 400 theme={"theme":{"light":"github-light","dark":"github-dark"}}
{
"id": "sync-a1b2c3d4-e5f6-7890-abcd-ef1234567890-u1",
"status": "FAILED",
"error": "Invalid duration: must be 4, 8, or 12 seconds"
}
```
Video URLs expire after 7 days. Download and store generated videos immediately if you need to keep them.
## Prompt structure
For best results, structure your prompt with these components:
* **Action**: Describe what happens in the scene
* **Ambient Sound**: Describe background audio
* **Character Dialogue**: Include spoken lines with voice descriptions
## Cost calculation
SORA 2 Pro I2V pricing varies by resolution and duration:
| Resolution | 4 seconds | 8 seconds | 12 seconds |
| ---------- | --------- | --------- | ---------- |
| 720p | \$1.20 | \$2.40 | \$3.60 |
| 1080p | \$2.00 | \$4.00 | \$6.00 |
# Vidu Q3 I2V
Source: https://docs.runpod.io/public-endpoints/models/vidu-q3-i2v
Animate reference images into videos with text-driven motion and optional audio generation. See model inputs and outputs on Runpod Public Endpoints.
Vidu Q3 Image-to-Video animates a reference image into a video with motion driven by a text prompt. It supports multiple resolutions up to 1080p, adjustable duration up to 16 seconds, and optional synchronized audio generation with background music.
Test Vidu Q3 I2V in the Runpod Hub playground.
| | |
| ------------ | ---------------------------------------------- |
| **Endpoint** | `https://api.runpod.ai/v2/vidu-q3-i2v/runsync` |
| **Pricing** | \$0.15 per second |
| **Type** | Video generation |
## Request
All parameters are passed within the `input` object in the request body.
URL of the reference image to animate.
Text description of the desired motion and action.
Output video resolution. Options: `540p`, `720p`, `1080p`.
Video length in seconds (1-16).
Motion intensity control. Options: `auto`, `small`, `medium`, `large`.
Enable synchronized audio generation.
Enable background music.
Random seed for reproducibility. Set to -1 for random.
```bash cURL theme={"theme":{"light":"github-light","dark":"github-dark"}}
curl -X POST "https://api.runpod.ai/v2/vidu-q3-i2v/runsync" \
-H "Authorization: Bearer $RUNPOD_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"input": {
"image": "https://example.com/portrait.jpg",
"prompt": "Person turns head slowly and smiles at camera",
"resolution": "720p",
"duration": 5,
"movement_amplitude": "medium",
"generate_audio": true,
"bgm": true,
"seed": -1
}
}'
```
```python Python theme={"theme":{"light":"github-light","dark":"github-dark"}}
import requests
response = requests.post(
"https://api.runpod.ai/v2/vidu-q3-i2v/runsync",
headers={
"Authorization": f"Bearer {RUNPOD_API_KEY}",
"Content-Type": "application/json",
},
json={
"input": {
"image": "https://example.com/portrait.jpg",
"prompt": "Person turns head slowly and smiles at camera",
"resolution": "720p",
"duration": 5,
"movement_amplitude": "medium",
"generate_audio": True,
"bgm": True,
"seed": -1,
}
},
)
result = response.json()
print(result["output"]["video_url"])
```
```javascript JavaScript theme={"theme":{"light":"github-light","dark":"github-dark"}}
const response = await fetch(
"https://api.runpod.ai/v2/vidu-q3-i2v/runsync",
{
method: "POST",
headers: {
Authorization: `Bearer ${RUNPOD_API_KEY}`,
"Content-Type": "application/json",
},
body: JSON.stringify({
input: {
image: "https://example.com/portrait.jpg",
prompt: "Person turns head slowly and smiles at camera",
resolution: "720p",
duration: 5,
movement_amplitude: "medium",
generate_audio: true,
bgm: true,
seed: -1,
},
}),
}
);
const result = await response.json();
console.log(result.output.video_url);
```
## Response
Unique identifier for the request.
Request status. Returns `COMPLETED` on success, `FAILED` on error.
Time in milliseconds the request spent in queue before processing began.
Time in milliseconds the model took to generate the video.
The generation result containing the video URL and cost.
URL of the generated video. This URL expires after 7 days.
Cost of the generation in USD.
```json 200 theme={"theme":{"light":"github-light","dark":"github-dark"}}
{
"id": "sync-a1b2c3d4-e5f6-7890-abcd-ef1234567890-u1",
"status": "COMPLETED",
"delayTime": 32,
"executionTime": 45678,
"output": {
"video_url": "https://video.runpod.ai/abc123/output.mp4",
"cost": 0.75
}
}
```
```json 400 theme={"theme":{"light":"github-light","dark":"github-dark"}}
{
"id": "sync-a1b2c3d4-e5f6-7890-abcd-ef1234567890-u1",
"status": "FAILED",
"error": "Invalid image URL"
}
```
Video URLs expire after 7 days. Download and store generated videos immediately if you need to keep them.
## Cost calculation
Vidu Q3 I2V charges \$0.15 per second of video generated.
| Duration | Cost |
| ---------- | ------ |
| 5 seconds | \$0.75 |
| 10 seconds | \$1.50 |
| 16 seconds | \$2.40 |
# Vidu Q3 T2V
Source: https://docs.runpod.io/public-endpoints/models/vidu-q3-t2v
Generate high-quality videos from text descriptions with multiple resolutions and audio support. See model inputs and outputs on Runpod Public Endpoints.
Vidu Q3 Text-to-Video generates high-quality videos from text descriptions with support for multiple resolutions up to 1080p, various aspect ratios, and optional audio generation. It includes style options for general or anime aesthetics.
Test Vidu Q3 T2V in the Runpod Hub playground.
| | |
| ------------ | ---------------------------------------------- |
| **Endpoint** | `https://api.runpod.ai/v2/vidu-q3-t2v/runsync` |
| **Pricing** | \$0.15 per second |
| **Type** | Video generation |
## Request
All parameters are passed within the `input` object in the request body.
Text description of the video scene and action.
Visual aesthetic style. Options: `general`, `anime`.
Output video resolution. Options: `540p`, `720p`, `1080p`.
Video length in seconds (1-16).
Output aspect ratio. Options: `16:9`, `9:16`, `4:3`, `3:4`, `1:1`.
Motion intensity control. Options: `auto`, `small`, `medium`, `large`.
Enable synchronized audio generation.
Enable background music.
Random seed for reproducibility. Set to -1 for random.
```bash cURL theme={"theme":{"light":"github-light","dark":"github-dark"}}
curl -X POST "https://api.runpod.ai/v2/vidu-q3-t2v/runsync" \
-H "Authorization: Bearer $RUNPOD_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"input": {
"prompt": "A futuristic city at night with flying cars and neon lights",
"style": "general",
"resolution": "720p",
"duration": 5,
"aspect_ratio": "16:9",
"movement_amplitude": "auto",
"generate_audio": true,
"bgm": true,
"seed": -1
}
}'
```
```python Python theme={"theme":{"light":"github-light","dark":"github-dark"}}
import requests
response = requests.post(
"https://api.runpod.ai/v2/vidu-q3-t2v/runsync",
headers={
"Authorization": f"Bearer {RUNPOD_API_KEY}",
"Content-Type": "application/json",
},
json={
"input": {
"prompt": "A futuristic city at night with flying cars and neon lights",
"style": "general",
"resolution": "720p",
"duration": 5,
"aspect_ratio": "16:9",
"movement_amplitude": "auto",
"generate_audio": True,
"bgm": True,
"seed": -1,
}
},
)
result = response.json()
print(result["output"]["video_url"])
```
```javascript JavaScript theme={"theme":{"light":"github-light","dark":"github-dark"}}
const response = await fetch(
"https://api.runpod.ai/v2/vidu-q3-t2v/runsync",
{
method: "POST",
headers: {
Authorization: `Bearer ${RUNPOD_API_KEY}`,
"Content-Type": "application/json",
},
body: JSON.stringify({
input: {
prompt: "A futuristic city at night with flying cars and neon lights",
style: "general",
resolution: "720p",
duration: 5,
aspect_ratio: "16:9",
movement_amplitude: "auto",
generate_audio: true,
bgm: true,
seed: -1,
},
}),
}
);
const result = await response.json();
console.log(result.output.video_url);
```
## Response
Unique identifier for the request.
Request status. Returns `COMPLETED` on success, `FAILED` on error.
Time in milliseconds the request spent in queue before processing began.
Time in milliseconds the model took to generate the video.
The generation result containing the video URL and cost.
URL of the generated video. This URL expires after 7 days.
Cost of the generation in USD.
```json 200 theme={"theme":{"light":"github-light","dark":"github-dark"}}
{
"id": "sync-a1b2c3d4-e5f6-7890-abcd-ef1234567890-u1",
"status": "COMPLETED",
"delayTime": 28,
"executionTime": 52341,
"output": {
"video_url": "https://video.runpod.ai/abc123/output.mp4",
"cost": 0.75
}
}
```
```json 400 theme={"theme":{"light":"github-light","dark":"github-dark"}}
{
"id": "sync-a1b2c3d4-e5f6-7890-abcd-ef1234567890-u1",
"status": "FAILED",
"error": "Invalid prompt"
}
```
Video URLs expire after 7 days. Download and store generated videos immediately if you need to keep them.
## Cost calculation
Vidu Q3 T2V charges \$0.15 per second of video generated.
| Duration | Cost |
| ---------- | ------ |
| 5 seconds | \$0.75 |
| 10 seconds | \$1.50 |
| 16 seconds | \$2.40 |
# WAN 2.1 I2V 720p
Source: https://docs.runpod.io/public-endpoints/models/wan-2-1-i2v
Open-source image-to-video generation that converts static images into 720p videos. See model inputs and outputs on Runpod Public Endpoints.
WAN 2.1 I2V 720p is an open-source image-to-video generation model that converts static images into 720p videos. It uses a diffusion transformer architecture to create smooth, natural motion from still images.
Test WAN 2.1 I2V 720p in the Runpod Hub playground.
| | |
| ------------ | -------------------------------------------------- |
| **Endpoint** | `https://api.runpod.ai/v2/wan-2-1-i2v-720/runsync` |
| **Pricing** | \$0.30 per 5 seconds of video |
| **Type** | Video generation |
## Request
All parameters are passed within the `input` object in the request body.
Text description of the desired video motion and content.
URL of the input image to animate.
Elements to exclude from the generated video.
Video resolution in format "width\*height".
Number of denoising steps. Range: 1-50.
How closely to follow the prompt. Range: 0.0-10.0.
Video duration in seconds.
Controls the motion flow in the generated video.
Seed for reproducible results. Set to -1 for random.
Whether to automatically optimize the prompt.
Whether to run safety checks on the output.
```bash cURL theme={"theme":{"light":"github-light","dark":"github-dark"}}
curl -X POST "https://api.runpod.ai/v2/wan-2-1-i2v-720/runsync" \
-H "Authorization: Bearer $RUNPOD_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"input": {
"prompt": "The family of three just took a selfie. They lean in together, smiling and relaxed. The daughter holds the phone and shows the screen",
"image": "https://example.com/family-photo.png",
"num_inference_steps": 30,
"guidance": 5,
"negative_prompt": "",
"size": "1280*720",
"duration": 5,
"flow_shift": 5,
"seed": -1,
"enable_prompt_optimization": false,
"enable_safety_checker": true
}
}'
```
```python Python theme={"theme":{"light":"github-light","dark":"github-dark"}}
import requests
response = requests.post(
"https://api.runpod.ai/v2/wan-2-1-i2v-720/runsync",
headers={
"Authorization": f"Bearer {RUNPOD_API_KEY}",
"Content-Type": "application/json",
},
json={
"input": {
"prompt": "The family of three just took a selfie. They lean in together, smiling and relaxed. The daughter holds the phone and shows the screen",
"image": "https://example.com/family-photo.png",
"num_inference_steps": 30,
"guidance": 5,
"negative_prompt": "",
"size": "1280*720",
"duration": 5,
"flow_shift": 5,
"seed": -1,
"enable_prompt_optimization": False,
"enable_safety_checker": True,
}
},
)
result = response.json()
print(result["output"]["video_url"])
```
```javascript JavaScript theme={"theme":{"light":"github-light","dark":"github-dark"}}
const response = await fetch(
"https://api.runpod.ai/v2/wan-2-1-i2v-720/runsync",
{
method: "POST",
headers: {
Authorization: `Bearer ${RUNPOD_API_KEY}`,
"Content-Type": "application/json",
},
body: JSON.stringify({
input: {
prompt: "The family of three just took a selfie. They lean in together, smiling and relaxed. The daughter holds the phone and shows the screen",
image: "https://example.com/family-photo.png",
num_inference_steps: 30,
guidance: 5,
negative_prompt: "",
size: "1280*720",
duration: 5,
flow_shift: 5,
seed: -1,
enable_prompt_optimization: false,
enable_safety_checker: true,
},
}),
}
);
const result = await response.json();
console.log(result.output.video_url);
```
## Response
Unique identifier for the request.
Request status. Returns `COMPLETED` on success, `FAILED` on error.
Time in milliseconds the request spent in queue before processing began.
Time in milliseconds the model took to generate the video.
Identifier of the worker that processed the request.
The generation result containing the video URL and cost.
URL of the generated video. This URL expires after 7 days.
Cost of the generation in USD.
```json 200 theme={"theme":{"light":"github-light","dark":"github-dark"}}
{
"id": "sync-a1b2c3d4-e5f6-7890-abcd-ef1234567890-u1",
"status": "COMPLETED",
"delayTime": 22,
"executionTime": 65432,
"workerId": "oqk7ao1uomckye",
"output": {
"video_url": "https://video.runpod.ai/abc123/output.mp4",
"cost": 0.30
}
}
```
```json 400 theme={"theme":{"light":"github-light","dark":"github-dark"}}
{
"id": "sync-a1b2c3d4-e5f6-7890-abcd-ef1234567890-u1",
"status": "FAILED",
"error": "Invalid image URL"
}
```
Video URLs expire after 7 days. Download and store generated videos immediately if you need to keep them.
## Cost calculation
WAN 2.1 I2V 720p charges \$0.30 per 5 seconds of video. Example costs:
| Duration | Cost |
| ---------- | ------ |
| 5 seconds | \$0.30 |
| 10 seconds | \$0.60 |
| 15 seconds | \$0.90 |
# WAN 2.2 I2V 720p
Source: https://docs.runpod.io/public-endpoints/models/wan-2-2-i2v
Open-source image-to-video generation using diffusion transformer architecture. See model inputs and outputs on Runpod Public Endpoints.
WAN 2.2 I2V 720p is an open-source AI video generation model that uses a diffusion transformer architecture for image-to-video generation. It creates smooth, high-quality 720p video content from static images.
Test WAN 2.2 I2V 720p in the Runpod Hub playground.
| | |
| ------------ | -------------------------------------------------- |
| **Endpoint** | `https://api.runpod.ai/v2/wan-2-2-i2v-720/runsync` |
| **Pricing** | \$0.30–\$0.90 per video |
| **Type** | Video generation |
## Request
All parameters are passed within the `input` object in the request body.
Text description of the desired video motion and content.
URL of the input image to animate.
Elements to exclude from the generated video.
Video resolution in format "width\*height".
Number of denoising steps. Range: 1-50.
How closely to follow the prompt. Range: 0.0-10.0.
Video duration in seconds.
Controls the motion flow in the generated video.
Seed for reproducible results. Set to -1 for random.
Whether to automatically optimize the prompt.
Whether to run safety checks on the output.
```bash cURL theme={"theme":{"light":"github-light","dark":"github-dark"}}
curl -X POST "https://api.runpod.ai/v2/wan-2-2-i2v-720/runsync" \
-H "Authorization: Bearer $RUNPOD_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"input": {
"prompt": "cinematic shot: slow-tracking camera glides parallel to a giant white origami boat drifting down a jade-green river",
"image": "https://example.com/origami-boat.png",
"num_inference_steps": 30,
"guidance": 5,
"negative_prompt": "",
"size": "1280*720",
"duration": 5,
"flow_shift": 5,
"seed": -1,
"enable_prompt_optimization": false,
"enable_safety_checker": true
}
}'
```
```python Python theme={"theme":{"light":"github-light","dark":"github-dark"}}
import requests
response = requests.post(
"https://api.runpod.ai/v2/wan-2-2-i2v-720/runsync",
headers={
"Authorization": f"Bearer {RUNPOD_API_KEY}",
"Content-Type": "application/json",
},
json={
"input": {
"prompt": "cinematic shot: slow-tracking camera glides parallel to a giant white origami boat drifting down a jade-green river",
"image": "https://example.com/origami-boat.png",
"num_inference_steps": 30,
"guidance": 5,
"negative_prompt": "",
"size": "1280*720",
"duration": 5,
"flow_shift": 5,
"seed": -1,
"enable_prompt_optimization": False,
"enable_safety_checker": True,
}
},
)
result = response.json()
print(result["output"]["video_url"])
```
```javascript JavaScript theme={"theme":{"light":"github-light","dark":"github-dark"}}
const response = await fetch(
"https://api.runpod.ai/v2/wan-2-2-i2v-720/runsync",
{
method: "POST",
headers: {
Authorization: `Bearer ${RUNPOD_API_KEY}`,
"Content-Type": "application/json",
},
body: JSON.stringify({
input: {
prompt: "cinematic shot: slow-tracking camera glides parallel to a giant white origami boat drifting down a jade-green river",
image: "https://example.com/origami-boat.png",
num_inference_steps: 30,
guidance: 5,
negative_prompt: "",
size: "1280*720",
duration: 5,
flow_shift: 5,
seed: -1,
enable_prompt_optimization: false,
enable_safety_checker: true,
},
}),
}
);
const result = await response.json();
console.log(result.output.video_url);
```
## Response
Unique identifier for the request.
Request status. Returns `COMPLETED` on success, `FAILED` on error.
Time in milliseconds the request spent in queue before processing began.
Time in milliseconds the model took to generate the video.
Identifier of the worker that processed the request.
The generation result containing the video URL and cost.
URL of the generated video. This URL expires after 7 days.
Cost of the generation in USD.
```json 200 theme={"theme":{"light":"github-light","dark":"github-dark"}}
{
"id": "sync-a1b2c3d4-e5f6-7890-abcd-ef1234567890-u1",
"status": "COMPLETED",
"delayTime": 24,
"executionTime": 67543,
"workerId": "oqk7ao1uomckye",
"output": {
"video_url": "https://video.runpod.ai/abc123/output.mp4",
"cost": 0.30
}
}
```
```json 400 theme={"theme":{"light":"github-light","dark":"github-dark"}}
{
"id": "sync-a1b2c3d4-e5f6-7890-abcd-ef1234567890-u1",
"status": "FAILED",
"error": "Invalid image URL"
}
```
Video URLs expire after 7 days. Download and store generated videos immediately if you need to keep them.
## Cost calculation
WAN 2.2 I2V 720p pricing:
| Duration | Cost |
| ---------- | ------ |
| 5 seconds | \$0.30 |
| 8 seconds | \$0.48 |
| 10 seconds | \$0.60 |
| 15 seconds | \$0.90 |
# WAN 2.2 I2V 720p LoRA
Source: https://docs.runpod.io/public-endpoints/models/wan-2-2-i2v-lora
Open-source video generation with LoRA support for customized camera movements. See model inputs and outputs on Runpod Public Endpoints.
WAN 2.2 I2V 720p LoRA is an open-source video generation model with LoRA support for customized camera movements and effects. It uses separate high-noise and low-noise LoRA configurations to achieve precise control over motion and style.
Test WAN 2.2 I2V 720p LoRA in the Runpod Hub playground.
| | |
| ------------ | ------------------------------------------------------- |
| **Endpoint** | `https://api.runpod.ai/v2/wan-2-2-t2v-720-lora/runsync` |
| **Pricing** | \$0.35–\$0.56 per video |
| **Type** | Video generation |
## Request
All parameters are passed within the `input` object in the request body.
Text description of the desired video motion.
URL of the source image to animate.
LoRA configurations for high-noise denoising stages.
URL or path to the LoRA model file.
Scale factor for the LoRA influence.
LoRA configurations for low-noise denoising stages.
URL or path to the LoRA model file.
Scale factor for the LoRA influence.
Video duration in seconds.
Seed for reproducible results. Set to -1 for random.
Enable content safety checking.
```bash cURL theme={"theme":{"light":"github-light","dark":"github-dark"}}
curl -X POST "https://api.runpod.ai/v2/wan-2-2-t2v-720-lora/runsync" \
-H "Authorization: Bearer $RUNPOD_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"input": {
"prompt": "orbit 180 around an astronaut on the moon",
"image": "https://example.com/astronaut.png",
"high_noise_loras": [
{
"path": "https://huggingface.co/ostris/wan22_i2v_14b_orbit_shot_lora/resolve/main/wan22_14b_i2v_orbit_high_noise.safetensors",
"scale": 1
}
],
"low_noise_loras": [
{
"path": "https://huggingface.co/ostris/wan22_i2v_14b_orbit_shot_lora/resolve/main/wan22_14b_i2v_orbit_low_noise.safetensors",
"scale": 1
}
],
"duration": 5,
"seed": -1,
"enable_safety_checker": true
}
}'
```
```python Python theme={"theme":{"light":"github-light","dark":"github-dark"}}
import requests
response = requests.post(
"https://api.runpod.ai/v2/wan-2-2-t2v-720-lora/runsync",
headers={
"Authorization": f"Bearer {RUNPOD_API_KEY}",
"Content-Type": "application/json",
},
json={
"input": {
"prompt": "orbit 180 around an astronaut on the moon",
"image": "https://example.com/astronaut.png",
"high_noise_loras": [
{
"path": "https://huggingface.co/ostris/wan22_i2v_14b_orbit_shot_lora/resolve/main/wan22_14b_i2v_orbit_high_noise.safetensors",
"scale": 1,
}
],
"low_noise_loras": [
{
"path": "https://huggingface.co/ostris/wan22_i2v_14b_orbit_shot_lora/resolve/main/wan22_14b_i2v_orbit_low_noise.safetensors",
"scale": 1,
}
],
"duration": 5,
"seed": -1,
"enable_safety_checker": True,
}
},
)
result = response.json()
print(result["output"]["video_url"])
```
```javascript JavaScript theme={"theme":{"light":"github-light","dark":"github-dark"}}
const response = await fetch(
"https://api.runpod.ai/v2/wan-2-2-t2v-720-lora/runsync",
{
method: "POST",
headers: {
Authorization: `Bearer ${RUNPOD_API_KEY}`,
"Content-Type": "application/json",
},
body: JSON.stringify({
input: {
prompt: "orbit 180 around an astronaut on the moon",
image: "https://example.com/astronaut.png",
high_noise_loras: [
{
path: "https://huggingface.co/ostris/wan22_i2v_14b_orbit_shot_lora/resolve/main/wan22_14b_i2v_orbit_high_noise.safetensors",
scale: 1,
},
],
low_noise_loras: [
{
path: "https://huggingface.co/ostris/wan22_i2v_14b_orbit_shot_lora/resolve/main/wan22_14b_i2v_orbit_low_noise.safetensors",
scale: 1,
},
],
duration: 5,
seed: -1,
enable_safety_checker: true,
},
}),
}
);
const result = await response.json();
console.log(result.output.video_url);
```
## Response
Unique identifier for the request.
Request status. Returns `COMPLETED` on success, `FAILED` on error.
Time in milliseconds the request spent in queue before processing began.
Time in milliseconds the model took to generate the video.
Identifier of the worker that processed the request.
The generation result containing the video URL and cost.
URL of the generated video. This URL expires after 7 days.
Cost of the generation in USD.
```json 200 theme={"theme":{"light":"github-light","dark":"github-dark"}}
{
"id": "sync-a1b2c3d4-e5f6-7890-abcd-ef1234567890-u1",
"status": "COMPLETED",
"delayTime": 32,
"executionTime": 95432,
"workerId": "oqk7ao1uomckye",
"output": {
"video_url": "https://video.runpod.ai/abc123/output.mp4",
"cost": 0.35
}
}
```
```json 400 theme={"theme":{"light":"github-light","dark":"github-dark"}}
{
"id": "sync-a1b2c3d4-e5f6-7890-abcd-ef1234567890-u1",
"status": "FAILED",
"error": "Failed to load LoRA model from URL"
}
```
Video URLs expire after 7 days. Download and store generated videos immediately if you need to keep them.
## Cost calculation
WAN 2.2 I2V 720p LoRA pricing:
| Duration | Cost |
| --------- | ------ |
| 5 seconds | \$0.35 |
| 8 seconds | \$0.56 |
# WAN 2.5
Source: https://docs.runpod.io/public-endpoints/models/wan-2-5
Image-to-video generation model with prompt expansion support. Review inputs and output formats for this model on Runpod Public Endpoints.
WAN 2.5 is Alibaba's image-to-video generation model that creates videos from static images. It features optional prompt expansion to automatically enhance your prompts for better results.
Test WAN 2.5 in the Runpod Hub playground.
| | |
| ------------ | ------------------------------------------ |
| **Endpoint** | `https://api.runpod.ai/v2/wan-2-5/runsync` |
| **Pricing** | \$0.25–\$1.50 per video |
| **Type** | Video generation |
## Request
All parameters are passed within the `input` object in the request body.
Text description of the desired video.
URL of the source image to animate.
Elements to exclude from the video.
Video dimensions in format "width\*height".
Video duration in seconds.
Seed for reproducible results. Set to -1 for random.
Automatically expand and enhance the prompt.
Enable content safety checking.
```bash cURL theme={"theme":{"light":"github-light","dark":"github-dark"}}
curl -X POST "https://api.runpod.ai/v2/wan-2-5/runsync" \
-H "Authorization: Bearer $RUNPOD_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"input": {
"prompt": "A stand-up comedian delivering a dad joke",
"image": "https://example.com/comedian.png",
"negative_prompt": "",
"size": "1280*720",
"duration": 5,
"seed": -1,
"enable_prompt_expansion": false,
"enable_safety_checker": true
}
}'
```
```python Python theme={"theme":{"light":"github-light","dark":"github-dark"}}
import requests
response = requests.post(
"https://api.runpod.ai/v2/wan-2-5/runsync",
headers={
"Authorization": f"Bearer {RUNPOD_API_KEY}",
"Content-Type": "application/json",
},
json={
"input": {
"prompt": "A stand-up comedian delivering a dad joke",
"image": "https://example.com/comedian.png",
"negative_prompt": "",
"size": "1280*720",
"duration": 5,
"seed": -1,
"enable_prompt_expansion": False,
"enable_safety_checker": True,
}
},
)
result = response.json()
print(result["output"]["video_url"])
```
```javascript JavaScript theme={"theme":{"light":"github-light","dark":"github-dark"}}
const response = await fetch(
"https://api.runpod.ai/v2/wan-2-5/runsync",
{
method: "POST",
headers: {
Authorization: `Bearer ${RUNPOD_API_KEY}`,
"Content-Type": "application/json",
},
body: JSON.stringify({
input: {
prompt: "A stand-up comedian delivering a dad joke",
image: "https://example.com/comedian.png",
negative_prompt: "",
size: "1280*720",
duration: 5,
seed: -1,
enable_prompt_expansion: false,
enable_safety_checker: true,
},
}),
}
);
const result = await response.json();
console.log(result.output.video_url);
```
## Response
Unique identifier for the request.
Request status. Returns `COMPLETED` on success, `FAILED` on error.
Time in milliseconds the request spent in queue before processing began.
Time in milliseconds the model took to generate the video.
Identifier of the worker that processed the request.
The generation result containing the video URL and cost.
URL of the generated video. This URL expires after 7 days.
Cost of the generation in USD.
```json 200 theme={"theme":{"light":"github-light","dark":"github-dark"}}
{
"id": "sync-a1b2c3d4-e5f6-7890-abcd-ef1234567890-u1",
"status": "COMPLETED",
"delayTime": 27,
"executionTime": 78901,
"workerId": "oqk7ao1uomckye",
"output": {
"video_url": "https://video.runpod.ai/abc123/output.mp4",
"cost": 0.50
}
}
```
```json 400 theme={"theme":{"light":"github-light","dark":"github-dark"}}
{
"id": "sync-a1b2c3d4-e5f6-7890-abcd-ef1234567890-u1",
"status": "FAILED",
"error": "Invalid image URL"
}
```
Video URLs expire after 7 days. Download and store generated videos immediately if you need to keep them.
## Cost calculation
WAN 2.5 pricing varies by resolution and duration:
| Resolution | 5 seconds | 10 seconds |
| ---------- | --------- | ---------- |
| 480p | \$0.25 | \$0.50 |
| 720p | \$0.50 | \$1.00 |
| 1080p | \$0.75 | \$1.50 |
# WAN 2.6 I2V
Source: https://docs.runpod.io/public-endpoints/models/wan-2-6-i2v
Image-to-video generation with audio support and resolutions up to 1080p. Explore this model's inputs and outputs on Runpod Public Endpoints.
WAN 2.6 Image-to-Video transforms static images into dynamic videos with support for audio integration, multiple resolutions up to 1080p, and durations up to 15 seconds. It features optional prompt expansion and multi-shot composition modes.
Test WAN 2.6 I2V in the Runpod Hub playground.
| | |
| ------------ | ---------------------------------------------- |
| **Endpoint** | `https://api.runpod.ai/v2/wan-2-6-i2v/runsync` |
| **Pricing** | \$0.10/s (720p), \$0.15/s (1080p) |
| **Type** | Video generation |
## Request
All parameters are passed within the `input` object in the request body.
Text description of the desired video motion and content.
URL of the source image to animate.
URL of an audio file to include in the video.
Elements to exclude from the generated video.
Video resolution. Options: `1280*720`, `1920*1080`.
Video duration in seconds. Options: `5`, `10`, `15`.
Shot composition mode. Options: `single`, `multi`.
Seed for reproducible results. Set to -1 for random.
Automatically expand and enhance the prompt.
Enable content safety checking.
```bash cURL theme={"theme":{"light":"github-light","dark":"github-dark"}}
curl -X POST "https://api.runpod.ai/v2/wan-2-6-i2v/runsync" \
-H "Authorization: Bearer $RUNPOD_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"input": {
"prompt": "A person walking through a sunny park, leaves gently swaying",
"image": "https://example.com/person-park.jpg",
"size": "1280*720",
"duration": 5,
"shot_type": "single",
"seed": -1,
"enable_prompt_expansion": false
}
}'
```
```python Python theme={"theme":{"light":"github-light","dark":"github-dark"}}
import requests
response = requests.post(
"https://api.runpod.ai/v2/wan-2-6-i2v/runsync",
headers={
"Authorization": f"Bearer {RUNPOD_API_KEY}",
"Content-Type": "application/json",
},
json={
"input": {
"prompt": "A person walking through a sunny park, leaves gently swaying",
"image": "https://example.com/person-park.jpg",
"size": "1280*720",
"duration": 5,
"shot_type": "single",
"seed": -1,
"enable_prompt_expansion": False,
}
},
)
result = response.json()
print(result["output"]["video_url"])
```
```javascript JavaScript theme={"theme":{"light":"github-light","dark":"github-dark"}}
const response = await fetch(
"https://api.runpod.ai/v2/wan-2-6-i2v/runsync",
{
method: "POST",
headers: {
Authorization: `Bearer ${RUNPOD_API_KEY}`,
"Content-Type": "application/json",
},
body: JSON.stringify({
input: {
prompt: "A person walking through a sunny park, leaves gently swaying",
image: "https://example.com/person-park.jpg",
size: "1280*720",
duration: 5,
shot_type: "single",
seed: -1,
enable_prompt_expansion: false,
},
}),
}
);
const result = await response.json();
console.log(result.output.video_url);
```
## Response
Unique identifier for the request.
Request status. Returns `COMPLETED` on success, `FAILED` on error.
Time in milliseconds the request spent in queue before processing began.
Time in milliseconds the model took to generate the video.
The generation result containing the video URL and cost.
URL of the generated video. This URL expires after 7 days.
Cost of the generation in USD.
```json 200 theme={"theme":{"light":"github-light","dark":"github-dark"}}
{
"id": "sync-a1b2c3d4-e5f6-7890-abcd-ef1234567890-u1",
"status": "COMPLETED",
"delayTime": 28,
"executionTime": 95432,
"output": {
"video_url": "https://video.runpod.ai/abc123/output.mp4",
"cost": 0.50
}
}
```
```json 400 theme={"theme":{"light":"github-light","dark":"github-dark"}}
{
"id": "sync-a1b2c3d4-e5f6-7890-abcd-ef1234567890-u1",
"status": "FAILED",
"error": "Invalid image URL"
}
```
Video URLs expire after 7 days. Download and store generated videos immediately if you need to keep them.
## Cost calculation
WAN 2.6 I2V pricing varies by resolution:
| Resolution | Rate |
| ----------------- | ----------------- |
| 720p (1280x720) | \$0.10 per second |
| 1080p (1920x1080) | \$0.15 per second |
Example costs:
| Resolution | 5 seconds | 10 seconds | 15 seconds |
| ---------- | --------- | ---------- | ---------- |
| 720p | \$0.50 | \$1.00 | \$1.50 |
| 1080p | \$0.75 | \$1.50 | \$2.25 |
# WAN 2.6 T2I
Source: https://docs.runpod.io/public-endpoints/models/wan-2-6-t2i
High-quality text-to-image with strong prompt adherence and clean composition. See model inputs and outputs on Runpod Public Endpoints.
WAN 2.6 Text-to-Image generates high-quality images from natural-language prompts with strong prompt adherence and clean composition. It produces detailed, coherent images across a variety of styles.
Test WAN 2.6 T2I in the Runpod Hub playground.
| | |
| ------------ | ---------------------------------------------- |
| **Endpoint** | `https://api.runpod.ai/v2/wan-2-6-t2i/runsync` |
| **Pricing** | \$0.03 per image |
| **Type** | Image generation |
## Request
All parameters are passed within the `input` object in the request body.
Text description of the desired image.
Image dimensions. Options: `1024*1024`, `1024*768`, `1440*1024`.
Seed for reproducible results. Set to -1 for random.
Enable content safety checking.
```bash cURL theme={"theme":{"light":"github-light","dark":"github-dark"}}
curl -X POST "https://api.runpod.ai/v2/wan-2-6-t2i/runsync" \
-H "Authorization: Bearer $RUNPOD_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"input": {
"prompt": "A modern tea shop interior, warm afternoon light, minimalist wood design",
"size": "1024*1024",
"seed": -1,
"enable_safety_checker": true
}
}'
```
```python Python theme={"theme":{"light":"github-light","dark":"github-dark"}}
import requests
response = requests.post(
"https://api.runpod.ai/v2/wan-2-6-t2i/runsync",
headers={
"Authorization": f"Bearer {RUNPOD_API_KEY}",
"Content-Type": "application/json",
},
json={
"input": {
"prompt": "A modern tea shop interior, warm afternoon light, minimalist wood design",
"size": "1024*1024",
"seed": -1,
"enable_safety_checker": True,
}
},
)
result = response.json()
print(result["output"]["image_url"])
```
```javascript JavaScript theme={"theme":{"light":"github-light","dark":"github-dark"}}
const response = await fetch(
"https://api.runpod.ai/v2/wan-2-6-t2i/runsync",
{
method: "POST",
headers: {
Authorization: `Bearer ${RUNPOD_API_KEY}`,
"Content-Type": "application/json",
},
body: JSON.stringify({
input: {
prompt: "A modern tea shop interior, warm afternoon light, minimalist wood design",
size: "1024*1024",
seed: -1,
enable_safety_checker: true,
},
}),
}
);
const result = await response.json();
console.log(result.output.image_url);
```
## Response
Unique identifier for the request.
Request status. Returns `COMPLETED` on success, `FAILED` on error.
The generation result containing the image URL and cost.
URL of the generated image. This URL expires after 7 days.
Cost of the generation in USD.
```json 200 theme={"theme":{"light":"github-light","dark":"github-dark"}}
{
"id": "sync-a1b2c3d4-e5f6-7890-abcd-ef1234567890-u1",
"status": "COMPLETED",
"delayTime": 15,
"executionTime": 3254,
"output": {
"image_url": "https://image.runpod.ai/abc123/output.png",
"cost": 0.03
}
}
```
Image URLs expire after 7 days. Download and store generated images immediately if you need to keep them.
## Cost calculation
WAN 2.6 T2I charges \$0.03 per image generated.
# WAN 2.6 T2V
Source: https://docs.runpod.io/public-endpoints/models/wan-2-6-t2v
Text-to-video with cinematic quality, stable motion, and strong instruction-following. See model inputs and outputs on Runpod Public Endpoints.
WAN 2.6 Text-to-Video turns plain prompts into coherent, cinematic clips with crisp detail, stable motion, and strong instruction-following. It supports multiple resolutions and durations up to 15 seconds.
Test WAN 2.6 T2V in the Runpod Hub playground.
| | |
| ------------ | ---------------------------------------------- |
| **Endpoint** | `https://api.runpod.ai/v2/wan-2-6-t2v/runsync` |
| **Pricing** | \$0.50–\$2.25 per video |
| **Type** | Video generation |
## Request
All parameters are passed within the `input` object in the request body.
Text description of the desired video content.
Elements to exclude from the generated video.
Video duration in seconds. Options: `5`, `10`, `15`.
Video resolution. Options: `1280*720`, `1920*1080`, `720*1280`, `1080*1920`.
Shot composition. Options: `single`, `multi`.
Seed for reproducible results. Set to -1 for random.
Automatically expand and enhance the prompt.
Optional audio URL to include in the video.
```bash cURL theme={"theme":{"light":"github-light","dark":"github-dark"}}
curl -X POST "https://api.runpod.ai/v2/wan-2-6-t2v/runsync" \
-H "Authorization: Bearer $RUNPOD_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"input": {
"prompt": "A golden retriever running through a field of sunflowers at sunset",
"duration": 5,
"size": "1280*720",
"seed": -1,
"enable_prompt_expansion": false
}
}'
```
```python Python theme={"theme":{"light":"github-light","dark":"github-dark"}}
import requests
response = requests.post(
"https://api.runpod.ai/v2/wan-2-6-t2v/runsync",
headers={
"Authorization": f"Bearer {RUNPOD_API_KEY}",
"Content-Type": "application/json",
},
json={
"input": {
"prompt": "A golden retriever running through a field of sunflowers at sunset",
"duration": 5,
"size": "1280*720",
"seed": -1,
"enable_prompt_expansion": False,
}
},
)
result = response.json()
print(result["output"]["video_url"])
```
```javascript JavaScript theme={"theme":{"light":"github-light","dark":"github-dark"}}
const response = await fetch(
"https://api.runpod.ai/v2/wan-2-6-t2v/runsync",
{
method: "POST",
headers: {
Authorization: `Bearer ${RUNPOD_API_KEY}`,
"Content-Type": "application/json",
},
body: JSON.stringify({
input: {
prompt: "A golden retriever running through a field of sunflowers at sunset",
duration: 5,
size: "1280*720",
seed: -1,
enable_prompt_expansion: false,
},
}),
}
);
const result = await response.json();
console.log(result.output.video_url);
```
## Response
Unique identifier for the request.
Request status. Returns `COMPLETED` on success, `FAILED` on error.
The generation result containing the video URL and cost.
URL of the generated video. This URL expires after 7 days.
Cost of the generation in USD.
```json 200 theme={"theme":{"light":"github-light","dark":"github-dark"}}
{
"id": "sync-a1b2c3d4-e5f6-7890-abcd-ef1234567890-u1",
"status": "COMPLETED",
"delayTime": 28,
"executionTime": 85432,
"output": {
"video_url": "https://video.runpod.ai/abc123/output.mp4",
"cost": 0.50
}
}
```
Video URLs expire after 7 days. Download and store generated videos immediately if you need to keep them.
## Cost calculation
WAN 2.6 T2V pricing varies by resolution and duration:
| Resolution | 5 seconds | 10 seconds | 15 seconds |
| ---------- | --------- | ---------- | ---------- |
| 720p | \$0.50 | \$1.00 | \$1.50 |
| 1080p | \$0.75 | \$1.50 | \$2.25 |
# Z-Image Turbo
Source: https://docs.runpod.io/public-endpoints/models/z-image-turbo
A 6-billion-parameter image generation model with text-to-image and image-to-image support. See model inputs and outputs on Runpod Public Endpoints.
Z-Image Turbo is a powerful and highly efficient 6B parameter image generation model that supports both text-to-image and image-to-image generation. It delivers high-quality results with fast inference times.
Test Z-Image Turbo in the Runpod Hub playground.
| | |
| ------------ | ------------------------------------------------ |
| **Endpoint** | `https://api.runpod.ai/v2/z-image-turbo/runsync` |
| **Pricing** | \$0.005 per image |
| **Type** | Image generation |
## Request
All parameters are passed within the `input` object in the request body.
Text description of the desired image. Supports English and Chinese.
URL of an input image for image-to-image generation.
Image dimensions. Options: `512*512`, `768*768`, `1024*1024`, `1280*1280`, `1024*768`, `768*1024`, `1280*720`, `720*1280`.
Strength of modifications for image-to-image. Range: 0.0-1.0.
Seed for reproducible results. Set to -1 for random.
Output format. Options: `png`, `jpeg`, `webp`.
Enable content safety checking.
```bash cURL theme={"theme":{"light":"github-light","dark":"github-dark"}}
curl -X POST "https://api.runpod.ai/v2/z-image-turbo/runsync" \
-H "Authorization: Bearer $RUNPOD_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"input": {
"prompt": "A serene Japanese garden with cherry blossoms",
"size": "1024*1024",
"seed": -1,
"output_format": "png",
"enable_safety_checker": true
}
}'
```
```python Python theme={"theme":{"light":"github-light","dark":"github-dark"}}
import requests
response = requests.post(
"https://api.runpod.ai/v2/z-image-turbo/runsync",
headers={
"Authorization": f"Bearer {RUNPOD_API_KEY}",
"Content-Type": "application/json",
},
json={
"input": {
"prompt": "A serene Japanese garden with cherry blossoms",
"size": "1024*1024",
"seed": -1,
"output_format": "png",
"enable_safety_checker": True,
}
},
)
result = response.json()
print(result["output"]["image_url"])
```
```javascript JavaScript theme={"theme":{"light":"github-light","dark":"github-dark"}}
const response = await fetch(
"https://api.runpod.ai/v2/z-image-turbo/runsync",
{
method: "POST",
headers: {
Authorization: `Bearer ${RUNPOD_API_KEY}`,
"Content-Type": "application/json",
},
body: JSON.stringify({
input: {
prompt: "A serene Japanese garden with cherry blossoms",
size: "1024*1024",
seed: -1,
output_format: "png",
enable_safety_checker: true,
},
}),
}
);
const result = await response.json();
console.log(result.output.image_url);
```
## Response
Unique identifier for the request.
Request status. Returns `COMPLETED` on success, `FAILED` on error.
Time in milliseconds the request spent in queue before processing began.
Time in milliseconds the model took to generate the image.
The generation result containing the image URL and cost.
URL of the generated image. This URL expires after 7 days.
Cost of the generation in USD.
```json 200 theme={"theme":{"light":"github-light","dark":"github-dark"}}
{
"id": "sync-a1b2c3d4-e5f6-7890-abcd-ef1234567890-u1",
"status": "COMPLETED",
"delayTime": 12,
"executionTime": 1856,
"output": {
"image_url": "https://image.runpod.ai/abc123/output.png",
"cost": 0.005
}
}
```
Image URLs expire after 7 days. Download and store generated images immediately if you need to keep them.
## Cost calculation
Z-Image Turbo charges \$0.005 per image generated.
# Overview
Source: https://docs.runpod.io/public-endpoints/overview
Test and deploy production-ready AI models using Public Endpoints. Review setup and request guidance for Runpod Public Endpoints.
Runpod offers Public Endpoints for instant API access to pre-deployed AI models for image, video, audio, and text generation. No deployment or infrastructure required—just [create an API key](/get-started/api-keys) and make a request:
```python Python theme={"theme":{"light":"github-light","dark":"github-dark"}}
import requests
response = requests.post(
"https://api.runpod.ai/v2/black-forest-labs-flux-1-schnell/runsync",
headers={
"Authorization": "Bearer YOUR_API_KEY", # Replace YOUR_API_KEY with your actual API key
"Content-Type": "application/json"
},
json={
"input": {
"prompt": "A beautiful sunset over mountains", # Customize your prompt
"width": 1024,
"height": 1024
}
}
)
result = response.json()
print(result["output"]["image_url"])
```
```bash cURL theme={"theme":{"light":"github-light","dark":"github-dark"}}
# Replace YOUR_API_KEY with your actual API key
curl -X POST "https://api.runpod.ai/v2/black-forest-labs-flux-1-schnell/runsync" \
-H "Authorization: Bearer YOUR_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"input": {
"prompt": "A beautiful sunset over mountains",
"width": 1024,
"height": 1024
}
}'
```
## Get started
Generate your first image in under 5 minutes.
Browse available models and their parameters.
Use the playground and REST API.
Integrate with JavaScript and TypeScript projects.
Chain multiple endpoints to generate videos from text.
## How it works
When you call a Public Endpoint, Runpod routes your request to a pre-deployed model running on optimized GPU infrastructure. The model processes your input and returns the result.
Public Endpoints support two request modes:
* **Synchronous (`/runsync`)**: Wait for the result and receive it in the response. Best for quick generations.
* **Asynchronous (`/run`)**: Receive a job ID immediately and poll for results. Best for longer generations or batch processing.
For JavaScript and TypeScript projects, the [`@runpod/ai-sdk-provider`](/public-endpoints/ai-sdk) package integrates Public Endpoints with the Vercel AI SDK, providing a streamlined interface for text generation, streaming, and image generation.
## Available model types
Public Endpoints offer models across four categories:
| Type | Example models | Use cases |
| --------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ---------------------------------------- |
| **Image** | [Flux Dev](/public-endpoints/models/flux-dev), [Flux Schnell](/public-endpoints/models/flux-schnell), [Qwen Image](/public-endpoints/models/qwen-image), [Seedream](/public-endpoints/models/seedream-4-t2i) | Text-to-image generation, image editing |
| **Video** | [WAN 2.5](/public-endpoints/models/wan-2-5), [Kling](/public-endpoints/models/kling-v2-1), [Seedance](/public-endpoints/models/seedance-1-5-pro), [SORA 2](/public-endpoints/models/sora-2) | Image-to-video, text-to-video generation |
| **Audio** | [Minimax Speech](/public-endpoints/models/minimax-speech), [Chatterbox Turbo](/public-endpoints/models/chatterbox-turbo) | Text-to-speech, voice cloning |
| **Text** | [Qwen3 32B](/public-endpoints/models/qwen3-32b), [IBM Granite](/public-endpoints/models/granite-4), [Moonshot Kimi](/public-endpoints/models/moonshot-kimi) | Chat, code generation, text completion |
For a complete list of models with endpoint URLs and parameters, see the [model reference](/public-endpoints/reference).
## Pricing
Public Endpoints use transparent, usage-based pricing:
| Model type | Example | Price |
| ---------------- | --------------------------------------------------------- | -------------------------- |
| Image generation | [Flux Dev](/public-endpoints/models/flux-dev) | \$0.02 per megapixel |
| Image generation | [Flux Schnell](/public-endpoints/models/flux-schnell) | \$0.0024 per megapixel |
| Video generation | [WAN 2.5](/public-endpoints/models/wan-2-5) | \$0.50 per 5 seconds |
| Text-to-speech | [Minimax Speech](/public-endpoints/models/minimax-speech) | \$0.05 per 1000 characters |
| Text generation | [Qwen3 32B](/public-endpoints/models/qwen3-32b) | \$10.00 per 1M tokens |
Pricing is calculated based on actual output. You will not be charged for failed generations.
**Example cost calculations for image generation:**
* 512x512 image (0.26 MP) with [Flux Dev](/public-endpoints/models/flux-dev): \~\$0.005
* 1024x1024 image (1.05 MP) with [Flux Dev](/public-endpoints/models/flux-dev): \~\$0.021
* 1024x1024 image (1.05 MP) with [Flux Schnell](/public-endpoints/models/flux-schnell): \~\$0.0025
For complete pricing information, see the [model reference](/public-endpoints/reference).
# Quickstart
Source: https://docs.runpod.io/public-endpoints/quickstart
Generate your first image with Public Endpoints in under 5 minutes. Review setup and request guidance for Runpod Public Endpoints.
This quickstart walks you through generating an image using Runpod Public Endpoints. You'll use the [Flux Schnell](/public-endpoints/models/flux-schnell) model, which is optimized for fast generation.
## Requirements
* A [Runpod account](/accounts-billing/manage-accounts) with at least \$1 in credits
* A [Runpod API key](/get-started/api-keys)
## Step 1: Generate an image in the playground
The fastest way to test Public Endpoints is through the browser-based playground.
1. Go to the [Flux Schnell endpoint](https://console.runpod.io/hub/playground/image/black-forest-labs-flux-1-schnell) in the Runpod console.
2. Under **Input**, enter a prompt: `A golden retriever playing fetch on a sunny beach` (or any other prompt you like).
3. Click **Run**.
4. Wait a few seconds for the image to generate. The result appears under **Result**.
You've just generated your first image. The playground shows the estimated cost (\~\$0.0025 for a 1024x1024 image).
## Step 2: Generate an image with the API
Now let's generate an image programmatically using the REST API.
1. Open a terminal on your local machine.
2. Copy the following command, replacing `YOUR_API_KEY` with your Runpod API key:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
curl -X POST "https://api.runpod.ai/v2/black-forest-labs-flux-1-schnell/runsync" \
-H "Authorization: Bearer YOUR_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"input": {
"prompt": "A golden retriever playing fetch on a sunny beach",
"width": 1024,
"height": 1024
}
}'
```
3. Paste the command into your terminal and press **Enter**.
4. Wait for the response (this takes about 10-20 seconds).
1. Create a new file called `generate_image.py` and paste the following code:
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
import requests
response = requests.post(
"https://api.runpod.ai/v2/black-forest-labs-flux-1-schnell/runsync",
headers={
"Authorization": "Bearer YOUR_API_KEY",
"Content-Type": "application/json",
},
json={
"input": {
"prompt": "A golden retriever playing fetch on a sunny beach",
"width": 1024,
"height": 1024,
}
},
)
result = response.json()
print(result["output"]["image_url"])
```
2. Replace `YOUR_API_KEY` with your Runpod API key.
3. Open a terminal, navigate to the directory containing the file, and run:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
python generate_image.py
```
4. Wait for the script to print the image URL (this takes about 10-20 seconds).
### Response
Both methods return a JSON response with your generated image:
```json theme={"theme":{"light":"github-light","dark":"github-dark"}}
{
"status": "COMPLETED",
"output": {
"image_url": "https://image.runpod.ai/...",
"cost": 0.02097152
}
}
```
Open the `image_url` in your browser to view the generated image.
Image URLs expire after 7 days. Download images immediately if you need to keep them.
## Next steps
* [Make API requests](/public-endpoints/requests): Learn about async requests, SDKs, and best practices.
* [Model reference](/public-endpoints/reference): Explore all available models and their parameters.
* [Connect AI coding tools](/public-endpoints/ai-coding-tools): Use Public Endpoints with Cursor, Cline, and OpenCode.
* [Build a text-to-video pipeline](/tutorials/public-endpoints/text-to-video-pipeline): Chain multiple endpoints to generate videos from text prompts.
# Available models
Source: https://docs.runpod.io/public-endpoints/reference
Browse all available models for Runpod Public Endpoints. Review setup, model selection, request options, and usage guidance for Runpod Public Endpoints.
This page lists all available models for Runpod Public Endpoints. Select a model below to view its parameters, pricing, and usage examples. You can also browse and test models in the [Runpod Hub playground](https://console.runpod.io/hub?tabSelected=public_endpoints).
## Image models
Generate and edit images with text prompts or reference images.
| Model | Description | Price |
| ------------------------------------------------------------------------------- | ---------------------------------------------------------------- | ---------------------- |
| [Flux Dev](/public-endpoints/models/flux-dev) | High-quality image generation with exceptional prompt adherence. | \$0.02/megapixel |
| [Flux Schnell](/public-endpoints/models/flux-schnell) | Fast, lightweight generation for prototyping. | \$0.0024/megapixel |
| [Flux Kontext Dev](/public-endpoints/models/flux-kontext-dev) | Edit images based on text instructions. | \$0.025/image |
| [P-Image T2I](/public-endpoints/models/p-image-t2i) | Ultra-fast text-to-image with automatic prompt enhancement. | \$0.005/image |
| [P-Image Edit](/public-endpoints/models/p-image-edit) | Premium image editing with complex compositions. | \$0.01/image |
| [Qwen Image](/public-endpoints/models/qwen-image) | Image generation with advanced text rendering. | \$0.02/image |
| [Qwen Image LoRA](/public-endpoints/models/qwen-image-lora) | Image generation with LoRA customization. | \$0.025/image |
| [Qwen Image Edit](/public-endpoints/models/qwen-image-edit) | Image editing with text rendering capabilities. | \$0.02/image |
| [Qwen Image Edit 2511](/public-endpoints/models/qwen-image-edit-2511) | Enhanced image editing with improved consistency. | \$0.02/image |
| [Qwen Image Edit 2511 LoRA](/public-endpoints/models/qwen-image-edit-2511-lora) | Advanced editing with LoRA support. | \$0.025/image |
| [Seedream 4.0 T2I](/public-endpoints/models/seedream-4-t2i) | New-generation text-to-image creation. | \$0.027/image |
| [Seedream 4.0 Edit](/public-endpoints/models/seedream-4-edit) | New-generation image editing. | \$0.027/image |
| [WAN 2.6 T2I](/public-endpoints/models/wan-2-6-t2i) | Open-source text-to-image at 1024x1024. | \$0.03/image |
| [Z-Image Turbo](/public-endpoints/models/z-image-turbo) | Fast 6B parameter image generation. | \$0.005/image |
| [Nano Banana Edit](/public-endpoints/models/nano-banana-edit) | Google's model for combining multiple images. | \$0.038/image |
| [Nano Banana Pro Edit](/public-endpoints/models/nano-banana-pro-edit) | Advanced multi-image editing with resolution options. | \$0.14–\$0.24/image |
| [Nano Banana 2 Edit](/public-endpoints/models/nano-banana-2-edit) | Google's latest multi-image editing with resolution options. | \$0.0875–\$0.175/image |
## Video models
Create videos from images or text prompts. Pricing varies by resolution and duration.
| Model | Description | Price |
| ------------------------------------------------------------------------------- | --------------------------------------------------- | -------------------------------------- |
| [InfiniteTalk](/public-endpoints/models/infinitetalk) | Audio-driven talking/singing video generation. | \$0.25 (480p), \$0.50 (720p) |
| [Kling v2.1 I2V Pro](/public-endpoints/models/kling-v2-1) | Professional image-to-video with enhanced fidelity. | \$0.45/5s, \$0.90/10s |
| [Kling v2.6 Motion Control](/public-endpoints/models/kling-v2-6-motion-control) | Motion transfer from reference videos. | \$0.21/3s, \$0.63/10s |
| [Kling Video O1 R2V](/public-endpoints/models/kling-video-o1-r2v) | Creative video with multi-reference images. | \$0.112/second |
| [Seedance 1.5 Pro I2V](/public-endpoints/models/seedance-1-5-pro) | Cinematic image-to-video with expressive motion. | \$0.024–\$0.052/second |
| [SORA 2 I2V](/public-endpoints/models/sora-2) | OpenAI's video and audio generation. | \$0.40 (4s), \$0.80 (8s), \$1.20 (12s) |
| [SORA 2 Pro I2V](/public-endpoints/models/sora-2-pro) | Professional-grade SORA video generation. | From \$1.20 (720p/4s) |
| [WAN 2.6 T2V](/public-endpoints/models/wan-2-6-t2v) | Text-to-video with resolution options. | \$0.50/5s (480p), \$2.25/10s (720p) |
| [WAN 2.5](/public-endpoints/models/wan-2-5) | Image-to-video with prompt expansion. | From \$0.25/5s |
| [WAN 2.2 I2V LoRA](/public-endpoints/models/wan-2-2-i2v-lora) | Image-to-video with LoRA camera controls. | \$0.35/5s, \$0.56/8s |
| [WAN 2.2 I2V](/public-endpoints/models/wan-2-2-i2v) | Open-source image-to-video at 720p. | \$0.30/5s |
| [WAN 2.1 I2V](/public-endpoints/models/wan-2-1-i2v) | Image-to-video at 720p. | \$0.30/5s |
| [WAN 2.6 I2V](/public-endpoints/models/wan-2-6-i2v) | Image-to-video with audio support up to 1080p. | \$0.10/s (720p), \$0.15/s (1080p) |
| [Pruna Video](/public-endpoints/models/p-video) | Premium AI video from text, images, and audio. | \$0.02/s (720p), \$0.04/s (1080p) |
| [Vidu Q3 I2V](/public-endpoints/models/vidu-q3-i2v) | Animate reference images with text-driven motion. | \$0.15/second |
| [Vidu Q3 T2V](/public-endpoints/models/vidu-q3-t2v) | High-quality video from text with audio support. | \$0.15/second |
## Text models
Generate text with large language models.
| Model | Description | Price |
| ------------------------------------------------------- | ------------------------------------------------------------------- | ------------------------ |
| [IBM Granite 4.0](/public-endpoints/models/granite-4) | 32B parameter long-context instruct model. | \$10.00/1M tokens |
| [Moonshot Kimi](/public-endpoints/models/moonshot-kimi) | Kimi family for reasoning, chat, and coding with extended thinking. | \$4.00–\$15.00/1M tokens |
| [Qwen3 32B AWQ](/public-endpoints/models/qwen3-32b) | Advanced reasoning and multilingual support. OpenAI-compatible. | \$10.00/1M tokens |
## Audio models
Generate audio from text.
| Model | Description | Price |
| --------------------------------------------------------------- | ------------------------------------------------- | --------------- |
| [Chatterbox Turbo](/public-endpoints/models/chatterbox-turbo) | Fast TTS with 20 preset voices and voice cloning. | \$0.001/second |
| [Minimax Speech 02 HD](/public-endpoints/models/minimax-speech) | Text-to-speech with emotional control. | \$0.05/1K chars |
## Next steps
* [Quickstart](/public-endpoints/quickstart): Get started with your first API request.
* [Make API requests](/public-endpoints/requests): Learn about request/response formats.
* [Vercel AI SDK](/public-endpoints/ai-sdk): Use the TypeScript SDK for easier integration.
* [Build a text-to-video pipeline](/tutorials/public-endpoints/text-to-video-pipeline): Chain multiple endpoints in a Python application.
# Make API requests
Source: https://docs.runpod.io/public-endpoints/requests
Use the playground, REST API, and SDKs to interact with Public Endpoints. Review setup and request guidance for Runpod Public Endpoints.
This guide covers all the ways to interact with Public Endpoints, from testing in the browser to integrating with your applications.
## Requirements
* A [Runpod account](/accounts-billing/manage-accounts) with credits
* A [Runpod API key](/get-started/api-keys) for making API requests
## Use the playground
The Public Endpoint playground lets you test models directly in your browser before writing any code.
The playground offers:
* **Interactive parameter adjustment**: Modify prompts, dimensions, and model settings in real-time.
* **Instant preview**: Generate images directly in the browser.
* **Cost estimation**: See estimated costs before running generation.
* **API code generation**: Create working code examples for your applications.
### Access the playground
1. Navigate to the [Runpod Hub](https://console.runpod.io/hub) in the console.
2. Select the **Public Endpoints** section.
3. Browse the available models and select one that fits your needs.
### Test a model
1. Select a model from the [Runpod Hub](https://www.console.runpod.io/hub).
2. Under **Input**, enter a prompt in the text box.
3. Enter a negative prompt if needed. Negative prompts tell the model what to exclude from the output.
4. Under **Additional settings**, you can adjust the seed, aspect ratio, number of inference steps, guidance scale, and output format.
5. Click **Run** to start generating.
Under **Result**, you can use the dropdown menu to show either a preview of the output, or the raw JSON.
### Generate code from the playground
After testing a model in the playground, you can automatically generate an API request to use in your application.
1. Click **API** (above the **Prompt** field).
2. Using the dropdown menus on the right, select the programming language (Python, JavaScript, cURL, etc.) and POST command you want to use (`/run` or `/runsync`).
3. Click the **Copy** icon to copy the code to your clipboard.
## Make API requests
You can make API requests to Public Endpoints using any HTTP client. All requests require authentication using your Runpod API key, passed in the `Authorization` header.
### Synchronous requests
Synchronous requests (`/runsync`) wait for the model to finish processing and return the result directly. Use these for quick generations where you want an immediate response.
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
curl -X POST "https://api.runpod.ai/v2/black-forest-labs-flux-1-dev/runsync" \
-H "Authorization: Bearer YOUR_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"input": {
"prompt": "A serene mountain landscape at sunset",
"width": 1024,
"height": 1024,
"num_inference_steps": 20,
"guidance": 7.5
}
}'
```
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
import requests
response = requests.post(
"https://api.runpod.ai/v2/black-forest-labs-flux-1-dev/runsync",
headers={
"Authorization": "Bearer YOUR_API_KEY",
"Content-Type": "application/json",
},
json={
"input": {
"prompt": "A serene mountain landscape at sunset",
"width": 1024,
"height": 1024,
"num_inference_steps": 20,
"guidance": 7.5,
}
},
)
result = response.json()
print(result["output"]["image_url"])
```
```javascript theme={"theme":{"light":"github-light","dark":"github-dark"}}
async function generateImage() {
const response = await fetch(
"https://api.runpod.ai/v2/black-forest-labs-flux-1-dev/runsync",
{
method: "POST",
headers: {
"Authorization": "Bearer YOUR_API_KEY",
"Content-Type": "application/json",
},
body: JSON.stringify({
input: {
prompt: "A serene mountain landscape at sunset",
width: 1024,
height: 1024,
num_inference_steps: 20,
guidance: 7.5,
},
}),
}
);
const result = await response.json();
console.log(result.output.image_url);
return result;
}
generateImage();
```
### Asynchronous requests
Asynchronous requests (`/run`) return immediately with a job ID. Use these for longer generations or when you want to queue multiple requests.
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
curl -X POST "https://api.runpod.ai/v2/black-forest-labs-flux-1-dev/run" \
-H "Authorization: Bearer YOUR_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"input": {
"prompt": "A futuristic cityscape with flying cars",
"width": 1024,
"height": 1024,
"num_inference_steps": 50,
"guidance": 8.0
}
}'
```
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
import requests
response = requests.post(
"https://api.runpod.ai/v2/black-forest-labs-flux-1-dev/run",
headers={
"Authorization": "Bearer YOUR_API_KEY",
"Content-Type": "application/json",
},
json={
"input": {
"prompt": "A futuristic cityscape with flying cars",
"width": 1024,
"height": 1024,
"num_inference_steps": 50,
"guidance": 8.0,
}
},
)
result = response.json()
job_id = result["id"]
print(f"Job submitted: {job_id}")
```
```javascript theme={"theme":{"light":"github-light","dark":"github-dark"}}
async function submitJob() {
const response = await fetch(
"https://api.runpod.ai/v2/black-forest-labs-flux-1-dev/run",
{
method: "POST",
headers: {
"Authorization": "Bearer YOUR_API_KEY",
"Content-Type": "application/json",
},
body: JSON.stringify({
input: {
prompt: "A futuristic cityscape with flying cars",
width: 1024,
height: 1024,
num_inference_steps: 50,
guidance: 8.0,
},
}),
}
);
const result = await response.json();
console.log(`Job submitted: ${result.id}`);
return result;
}
submitJob();
```
### Check job status
After submitting an asynchronous request, use the `/status` endpoint to check progress and retrieve results. Replace `JOB_ID` with the job ID returned from the `/run` request.
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
curl -X GET "https://api.runpod.ai/v2/black-forest-labs-flux-1-dev/status/JOB_ID" \
-H "Authorization: Bearer YOUR_API_KEY"
```
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
import requests
import time
job_id = "your-job-id"
while True:
response = requests.get(
f"https://api.runpod.ai/v2/black-forest-labs-flux-1-dev/status/{job_id}",
headers={"Authorization": "Bearer YOUR_API_KEY"},
)
result = response.json()
status = result["status"]
if status == "COMPLETED":
print(result["output"]["image_url"])
break
elif status == "FAILED":
print(f"Job failed: {result}")
break
else:
print(f"Status: {status}, waiting...")
time.sleep(2)
```
```javascript theme={"theme":{"light":"github-light","dark":"github-dark"}}
async function checkStatus(jobId) {
while (true) {
const response = await fetch(
`https://api.runpod.ai/v2/black-forest-labs-flux-1-dev/status/${jobId}`,
{
headers: {
"Authorization": "Bearer YOUR_API_KEY",
},
}
);
const result = await response.json();
if (result.status === "COMPLETED") {
console.log(result.output.image_url);
return result;
} else if (result.status === "FAILED") {
throw new Error(`Job failed: ${JSON.stringify(result)}`);
}
console.log(`Status: ${result.status}, waiting...`);
await new Promise((resolve) => setTimeout(resolve, 2000));
}
}
checkStatus("your-job-id");
```
### Response format
All endpoints return a consistent JSON response format:
```json theme={"theme":{"light":"github-light","dark":"github-dark"}}
{
"delayTime": 17,
"executionTime": 3986,
"id": "sync-0965434e-ff63-4a1c-a9f9-5b705f66e176-u2",
"output": {
"cost": 0.02097152,
"image_url": "https://image.runpod.ai/..."
},
"status": "COMPLETED",
"workerId": "oqk7ao1uomckye"
}
```
Output URLs (`image_url`, `video_url`, and `audio_url`) expire after 7 days. Download and store your generated files immediately if you need to keep them longer.
## Vercel AI SDK
For JavaScript and TypeScript projects, you can use the `@runpod/ai-sdk-provider` package to integrate Public Endpoints with the [Vercel AI SDK](https://ai-sdk.dev/docs/introduction). This provides a streamlined, type-safe interface for text generation, streaming, and image generation.
See the [Vercel AI SDK guide](/public-endpoints/ai-sdk) for installation, configuration, and usage examples.
## Best practices
### Prompt engineering
* **Be specific**: Detailed prompts generally produce better results.
* **Include style modifiers**: Specify art styles, camera angles, or lighting conditions.
A good prompt example: "A professional portrait of a woman in business attire, studio lighting, high quality, detailed, corporate headshot style."
### Performance optimization
* **Choose the right model**: Use smaller, cheaper models (e.g. [Flux Schnell](/public-endpoints/models/flux-schnell)) for testing and development, and more powerful models (e.g. [Flux Dev](/public-endpoints/models/flux-dev)) for production.
* **Batch with async**: For multiple images, use `/run` to queue requests.
* **Cache results**: Store generated images to avoid regenerating identical prompts.
## Next steps
* [Model reference](/public-endpoints/reference): View all available models and their parameters.
* [Connect AI coding tools](/public-endpoints/ai-coding-tools): Configure Cursor, Cline, and OpenCode with Public Endpoints.
* [Build a text-to-video pipeline](/tutorials/public-endpoints/text-to-video-pipeline): Chain multiple endpoints to generate videos from text prompts.
* [Build custom endpoints](/serverless/overview): Deploy your own models with Runpod Serverless.
# CPU types
Source: https://docs.runpod.io/references/cpu-types
Review the CPU models available on Runpod, including core counts and threads per core, to compare compute options for your workload.
The following list contains all CPU types available on Runpod.
| Display Name | Cores | Threads Per Core |
| ----------------------------------------------- | ----- | ---------------- |
| 11th Gen Intel(R) Core(TM) i5-11400 @ 2.60GHz | 6 | 2 |
| 11th Gen Intel(R) Core(TM) i5-11400F @ 2.60GHz | 6 | 2 |
| 11th Gen Intel(R) Core(TM) i7-1165G7 @ 2.80GHz | 2 | 1 |
| 11th Gen Intel(R) Core(TM) i7-11700 @ 2.50GHz | 8 | 2 |
| 11th Gen Intel(R) Core(TM) i7-11700F @ 2.50GHz | 8 | 2 |
| 11th Gen Intel(R) Core(TM) i7-11700K @ 3.60GHz | 8 | 2 |
| 11th Gen Intel(R) Core(TM) i7-11700KF @ 3.60GHz | 8 | 2 |
| 11th Gen Intel(R) Core(TM) i9-11900K @ 3.50GHz | 8 | 2 |
| 11th Gen Intel(R) Core(TM) i9-11900KF @ 3.50GHz | 8 | 2 |
| 12th Gen Intel(R) Core(TM) i3-12100 | 4 | 2 |
| 12th Gen Intel(R) Core(TM) i7-12700F | 12 | 1 |
| 12th Gen Intel(R) Core(TM) i7-12700K | 12 | 1 |
| 13th Gen Intel(R) Core(TM) i3-13100F | 4 | 2 |
| 13th Gen Intel(R) Core(TM) i5-13600K | 14 | 1 |
| 13th Gen Intel(R) Core(TM) i7-13700K | 16 | 1 |
| 13th Gen Intel(R) Core(TM) i7-13700KF | 16 | 1 |
| 13th Gen Intel(R) Core(TM) i9-13900F | 24 | 1 |
| 13th Gen Intel(R) Core(TM) i9-13900K | 24 | 1 |
| 13th Gen Intel(R) Core(TM) i9-13900KF | 24 | 1 |
| AMD Eng Sample: 100-000000053-04\_32/20\_N | 48 | 1 |
| AMD Eng Sample: 100-000000314-04\_30/16\_N | 64 | 2 |
| AMD Eng Sample: 100-000000897-03 | 32 | 2 |
| AMD Eng Sample: 100-000001245-08 | 128 | 2 |
| AMD EPYC 4564P 16-Core Processor | 16 | 2 |
| AMD EPYC 7251 8-Core Processor | 8 | 2 |
| AMD EPYC 7252 8-Core Processor | 8 | 2 |
| AMD EPYC 7272 12-Core Processor | 12 | 2 |
| AMD EPYC 7281 16-Core Processor | 16 | 2 |
| AMD EPYC 7282 16-Core Processor | 16 | 2 |
| AMD EPYC 7302 16-Core Processor | 16 | 2 |
| AMD EPYC 7302P 16-Core Processor | 16 | 2 |
| AMD EPYC 7313 16-Core Processor | 16 | 2 |
| AMD EPYC 7313P 16-Core Processor | 16 | 2 |
| AMD EPYC 7343 16-Core Processor | 16 | 2 |
| AMD EPYC 7351P 16-Core Processor | 16 | 2 |
| AMD EPYC 7352 24-Core Processor | 24 | 2 |
| AMD EPYC 7371 16-Core Processor | 16 | 2 |
| AMD EPYC 7402 24-Core Processor | 24 | 2 |
| AMD EPYC 7402P 24-Core Processor | 24 | 2 |
| AMD EPYC 7413 24-Core Processor | 24 | 2 |
| AMD EPYC 7443 24-Core Processor | 48 | 1 |
| AMD EPYC 7443P 24-Core Processor | 24 | 2 |
| AMD EPYC 7452 32-Core Processor | 32 | 2 |
| AMD EPYC 7453 28-Core Processor | 28 | 1 |
| AMD EPYC 74F3 24-Core Processor | 24 | 2 |
| AMD EPYC 7502 32-Core Processor | 32 | 1 |
| AMD EPYC 7502P 32-Core Processor | 32 | 1 |
| AMD EPYC 7513 32-Core Processor | 32 | 2 |
| AMD EPYC 7532 32-Core Processor | 32 | 2 |
| AMD EPYC 7542 32-Core Processor | 32 | 2 |
| AMD EPYC 7543 32-Core Processor | 28 | 1 |
| AMD EPYC 7543P 32-Core Processor | 32 | 2 |
| AMD EPYC 7551 32-Core Processor | 32 | 2 |
| AMD EPYC 7551P 32-Core Processor | 32 | 2 |
| AMD EPYC 7552 48-Core Processor | 48 | 2 |
| AMD EPYC 7573X 32-Core Processor | 32 | 2 |
| AMD EPYC 75F3 32-Core Processor | 32 | 2 |
| AMD EPYC 7601 32-Core Processor | 32 | 2 |
| AMD EPYC 7642 48-Core Processor | 48 | 2 |
| AMD EPYC 7643 48-Core Processor | 48 | 2 |
| AMD EPYC 7663 56-Core Processor | 56 | 2 |
| AMD EPYC 7702 64-Core Processor | 64 | 2 |
| AMD EPYC 7702P 64-Core Processor | 64 | 2 |
| AMD EPYC 7713 64-Core Processor | 64 | 1 |
| AMD EPYC 7713P 64-Core Processor | 64 | 2 |
| AMD EPYC 7742 64-Core Processor | 64 | 2 |
| AMD EPYC 7763 64-Core Processor | 64 | 2 |
| AMD EPYC 7773X 64-Core Processor | 64 | 2 |
| AMD EPYC 7B12 64-Core Processor | 64 | 2 |
| AMD EPYC 7B13 64-Core Processor | 64 | 1 |
| AMD EPYC 7C13 64-Core Processor | 64 | 2 |
| AMD EPYC 7F32 8-Core Processor | 8 | 2 |
| AMD EPYC 7F52 16-Core Processor | 16 | 2 |
| AMD EPYC 7F72 24-Core Processor | 24 | 2 |
| AMD EPYC 7H12 64-Core Processor | 64 | 2 |
| AMD EPYC 7J13 64-Core Processor | 64 | 2 |
| AMD EPYC 7K62 48-Core Processor | 48 | 2 |
| AMD EPYC 7R13 Processor | 48 | 2 |
| AMD EPYC 7R32 48-Core Processor | 48 | 2 |
| AMD EPYC 7T83 64-Core Processor | 127 | 1 |
| AMD EPYC 7V13 64-Core Processor | 24 | 1 |
| AMD EPYC 9124 16-Core Processor | 16 | 2 |
| AMD EPYC 9254 24-Core Processor | 24 | 2 |
| AMD EPYC 9274F 24-Core Processor | 24 | 2 |
| AMD EPYC 9334 32-Core Processor | 32 | 2 |
| AMD EPYC 9335 32-Core Processor | 32 | 2 |
| AMD EPYC 9354 32-Core Emb Processor | 32 | 2 |
| AMD EPYC 9354 32-Core Processor | 32 | 2 |
| AMD EPYC 9354P | 64 | 1 |
| AMD EPYC 9354P 32-Core Processor | 32 | 2 |
| AMD EPYC 9355 32-Core Processor | 32 | 2 |
| AMD EPYC 9355P 32-Core Processor | 32 | 2 |
| AMD EPYC 9374F 32-Core Processor | 32 | 1 |
| AMD EPYC 9454 48-Core Processor | 48 | 2 |
| AMD EPYC 9454P 48-Core Emb Processor | 48 | 2 |
| AMD EPYC 9455 48-Core Processor | 48 | 2 |
| AMD EPYC 9455P 48-Core Processor | 48 | 2 |
| AMD EPYC 9474F 48-Core Processor | 48 | 2 |
| AMD EPYC 9534 64-Core Processor | 64 | 2 |
| AMD EPYC 9535 64-Core Processor | 64 | 2 |
| AMD EPYC 9554 64-Core Emb Processor | 64 | 1 |
| AMD EPYC 9554 64-Core Processor | 126 | 1 |
| AMD EPYC 9555 64-Core Processor | 56 | 2 |
| AMD EPYC 9575F 64-Core Processor | 64 | 2 |
| AMD EPYC 9654 96-Core Emb Processor | 96 | 1 |
| AMD EPYC 9654 96-Core Processor | 96 | 2 |
| AMD EPYC 9655 96-Core Processor | 56 | 2 |
| AMD EPYC 9655P 96-Core Processor | 96 | 2 |
| AMD EPYC 9754 128-Core Processor | 128 | 2 |
| AMD EPYC 9755 128-Core Processor | 128 | 2 |
| AMD EPYC 9B14 96-Core Processor | 96 | 1 |
| AMD EPYC 9J14 96-Core Processor | 96 | 2 |
| AMD EPYC Processor | 1 | 1 |
| AMD EPYC Processor (with IBPB) | 16 | 1 |
| AMD EPYC-Rome Processor | 16 | 1 |
| AMD Ryzen 3 2200G with Radeon Vega Graphics | 4 | 1 |
| AMD Ryzen 3 3200G with Radeon Vega Graphics | 4 | 1 |
| AMD Ryzen 3 4100 4-Core Processor | 4 | 2 |
| AMD Ryzen 5 1600 Six-Core Processor | 6 | 2 |
| AMD Ryzen 5 2600 Six-Core Processor | 6 | 2 |
| AMD Ryzen 5 2600X Six-Core Processor | 6 | 2 |
| AMD Ryzen 5 3600 6-Core Processor | 6 | 2 |
| AMD Ryzen 5 3600X 6-Core Processor | 6 | 2 |
| AMD Ryzen 5 5500 | 6 | 2 |
| AMD Ryzen 5 5600G with Radeon Graphics | 6 | 2 |
| Ryzen 5 5600X | 6 | 2 |
| AMD Ryzen 5 7600 6-Core Processor | 6 | 2 |
| AMD Ryzen 5 8600G w/ Radeon 760M Graphics | 6 | 2 |
| AMD Ryzen 5 PRO 2600 Six-Core Processor | 6 | 2 |
| AMD Ryzen 7 1700 Eight-Core Processor | 8 | 2 |
| AMD Ryzen 7 1700X Eight-Core Processor | 8 | 2 |
| AMD Ryzen 7 3700X 8-Core Processor | 16 | 1 |
| AMD Ryzen 7 5700G with Radeon Graphics | 8 | 2 |
| AMD Ryzen 7 5700X 8-Core Processor | 8 | 2 |
| AMD Ryzen 7 5800X 8-Core Processor | 8 | 2 |
| AMD Ryzen 7 7700 8-Core Processor | 8 | 2 |
| AMD Ryzen 7 PRO 3700 8-Core Processor | 8 | 2 |
| AMD Ryzen 9 3900X 12-Core Processor | 12 | 2 |
| Ryzen 9 5900X | 12 | 2 |
| AMD Ryzen 9 5950X 16-Core Processor | 16 | 2 |
| AMD Ryzen 9 7900 12-Core Processor | 12 | 2 |
| AMD Ryzen 9 7950X 16-Core Processor | 16 | 2 |
| AMD Ryzen 9 7950X3D 16-Core Processor | 16 | 2 |
| AMD Ryzen 9 9900X 12-Core Processor | 12 | 2 |
| AMD Ryzen 9 9950X 16-Core Processor | 16 | 2 |
| AMD Ryzen Threadripper 1900X 8-Core Processor | 8 | 2 |
| AMD Ryzen Threadripper 1920X 12-Core Processor | 12 | 2 |
| AMD Ryzen Threadripper 1950X 16-Core Processor | 16 | 2 |
| AMD Ryzen Threadripper 2920X 12-Core Processor | 12 | 2 |
| AMD Ryzen Threadripper 2950X 16-Core Processor | 16 | 2 |
| AMD Ryzen Threadripper 2970WX 24-Core Processor | 24 | 1 |
| AMD Ryzen Threadripper 2990WX 32-Core Processor | 32 | 2 |
| AMD Ryzen Threadripper 3960X 24-Core Processor | 24 | 2 |
| AMD Ryzen Threadripper 7960X 24-Cores | 24 | 2 |
| AMD Ryzen Threadripper 9960X 24-Cores | 24 | 2 |
| Ryzen Threadripper PRO 3955WX | 16 | 2 |
| AMD Ryzen Threadripper PRO 3975WX 32-Cores | 32 | 2 |
| AMD Ryzen Threadripper PRO 3995WX 64-Cores | 64 | 2 |
| AMD Ryzen Threadripper PRO 5945WX 12-Cores | 12 | 2 |
| AMD Ryzen Threadripper PRO 5955WX 16-Cores | 16 | 2 |
| AMD Ryzen Threadripper PRO 5965WX 24-Cores | 24 | 2 |
| AMD Ryzen Threadripper PRO 5975WX 32-Cores | 32 | 2 |
| AMD Ryzen Threadripper PRO 5995WX 64-Cores | 18 | 1 |
| AMD Ryzen Threadripper PRO 7955WX 16-Cores | 16 | 2 |
| AMD Ryzen Threadripper PRO 7965WX 24-Cores | 24 | 2 |
| AMD Ryzen Threadripper PRO 7975WX 32-Cores | 32 | 2 |
| AMD Ryzen Threadripper PRO 7985WX 64-Cores | 112 | 1 |
| Common KVM processor | 28 | 1 |
| Genuine Intel(R) CPU @ 2.20GHz | 14 | 2 |
| Genuine Intel(R) CPU \$0000%@ | 24 | 2 |
| Intel Xeon E3-12xx v2 (Ivy Bridge) | 1 | 1 |
| Intel Xeon Processor (Icelake) | 40 | 2 |
| Intel(R) Celeron(R) CPU G3900 @ 2.80GHz | 2 | 1 |
| Intel(R) Celeron(R) G5905 CPU @ 3.50GHz | 2 | 1 |
| Intel(R) Core(TM) i3-10100F CPU @ 3.60GHz | 4 | 2 |
| Intel(R) Core(TM) i3-10105F CPU @ 3.70GHz | 4 | 2 |
| Intel(R) Core(TM) i3-6100 CPU @ 3.70GHz | 2 | 2 |
| Intel(R) Core(TM) i3-9100F CPU @ 3.60GHz | 4 | 1 |
| Intel(R) Core(TM) i5-10400 CPU @ 2.90GHz | 6 | 2 |
| Intel(R) Core(TM) i5-10400F CPU @ 2.90GHz | 6 | 2 |
| Intel(R) Core(TM) i5-10600 CPU @ 3.30GHz | 6 | 2 |
| Intel(R) Core(TM) i5-14500 | 14 | 2 |
| Intel(R) Core(TM) i5-14600K | 14 | 2 |
| Intel(R) Core(TM) i5-14600KF | 14 | 2 |
| Intel(R) Core(TM) i5-4570 CPU @ 3.20GHz | 4 | 1 |
| Intel(R) Core(TM) i5-6400 CPU @ 2.70GHz | 4 | 1 |
| Intel(R) Core(TM) i5-6500 CPU @ 3.20GHz | 4 | 1 |
| Intel(R) Core(TM) i5-7400 CPU @ 3.00GHz | 4 | 1 |
| Intel(R) Core(TM) i5-9400F CPU @ 2.90GHz | 6 | 1 |
| Intel(R) Core(TM) i7-10700F CPU @ 2.90GHz | 8 | 2 |
| Intel(R) Core(TM) i7-10700K CPU @ 3.80GHz | 8 | 2 |
| Intel(R) Core(TM) i7-3770 CPU @ 3.40GHz | 4 | 2 |
| Intel(R) Core(TM) i7-4790 CPU @ 3.60GHz | 4 | 2 |
| Intel(R) Core(TM) i7-6700 CPU @ 3.40GHz | 4 | 2 |
| Intel(R) Core(TM) i7-6700K CPU @ 4.00GHz | 4 | 2 |
| Intel(R) Core(TM) i7-6800K CPU @ 3.40GHz | 6 | 2 |
| Intel(R) Core(TM) i7-7700K CPU @ 4.20GHz | 4 | 2 |
| Intel(R) Core(TM) i7-8700 CPU @ 3.20GHz | 6 | 2 |
| Intel(R) Core(TM) i7-9700 CPU @ 3.00GHz | 8 | 1 |
| Intel(R) Core(TM) i9-10940X CPU @ 3.30GHz | 14 | 2 |
| Intel(R) Core(TM) i9-14900K | 24 | 1 |
| Intel(R) Core(TM) Ultra 5 245K | 1 | 1 |
| Intel(R) Pentium(R) CPU G3260 @ 3.30GHz | 2 | 1 |
| Intel(R) Pentium(R) CPU G4560 @ 3.50GHz | 2 | 2 |
| Intel(R) Xeon(R) 6732P | 32 | 2 |
| Intel(R) Xeon(R) 6747P | 48 | 2 |
| Intel(R) Xeon(R) 6767P | 64 | 2 |
| Intel(R) Xeon(R) 6787P | 86 | 1 |
| Intel(R) Xeon(R) 6952P | 96 | 2 |
| Intel(R) Xeon(R) 6960P | 72 | 2 |
| Intel(R) Xeon(R) Bronze 3204 CPU @ 1.90GHz | 6 | 1 |
| Intel(R) Xeon(R) CPU X5660 @ 2.80GHz | 6 | 2 |
| Intel(R) Xeon(R) CPU E3-1220 v3 @ 3.10GHz | 4 | 1 |
| Intel(R) Xeon(R) CPU E3-1225 V2 @ 3.20GHz | 4 | 1 |
| Intel(R) Xeon(R) CPU E5-1650 v4 @ 3.60GHz | 6 | 2 |
| Intel(R) Xeon(R) CPU E5-2603 v3 @ 1.60GHz | 6 | 1 |
| Intel(R) Xeon(R) CPU E5-2609 0 @ 2.40GHz | 4 | 1 |
| Intel(R) Xeon(R) CPU E5-2609 v3 @ 1.90GHz | 1 | 1 |
| Intel(R) Xeon(R) CPU E5-2620 v4 @ 2.10GHz | 8 | 2 |
| Intel(R) Xeon(R) CPU E5-2630 0 @ 2.30GHz | 6 | 2 |
| Intel(R) Xeon(R) CPU E5-2630 v2 @ 2.60GHz | 6 | 2 |
| Intel(R) Xeon(R) CPU E5-2630 v3 @ 2.40GHz | 8 | 2 |
| Intel(R) Xeon(R) CPU E5-2630 v4 @ 2.20GHz | 10 | 2 |
| Intel(R) Xeon(R) CPU E5-2637 v2 @ 3.50GHz | 4 | 2 |
| Intel(R) Xeon(R) CPU E5-2643 0 @ 3.30GHz | 4 | 1 |
| Intel(R) Xeon(R) CPU E5-2648L v3 @ 1.80GHz | 12 | 2 |
| Intel(R) Xeon(R) CPU E5-2650 v2 @ 2.60GHz | 16 | 1 |
| Intel(R) Xeon(R) CPU E5-2650 v3 @ 2.30GHz | 10 | 2 |
| Intel(R) Xeon(R) CPU E5-2650 v4 @ 2.20GHz | 12 | 2 |
| Intel(R) Xeon(R) CPU E5-2660 v2 @ 2.20GHz | 10 | 2 |
| Intel(R) Xeon(R) CPU E5-2667 v2 @ 3.30GHz | 1 | 1 |
| Intel(R) Xeon(R) CPU E5-2667 v3 @ 3.20GHz | 8 | 2 |
| Intel(R) Xeon(R) CPU E5-2667 v4 @ 3.20GHz | 1 | 1 |
| Intel(R) Xeon(R) CPU E5-2670 0 @ 2.60GHz | 8 | 2 |
| Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz | 10 | 2 |
| Intel(R) Xeon(R) CPU E5-2673 v4 @ 2.30GHz | 20 | 2 |
| Intel(R) Xeon(R) CPU E5-2678 v3 @ 2.50GHz | 12 | 2 |
| Intel(R) Xeon(R) CPU E5-2680 v3 @ 2.50GHz | 12 | 2 |
| Intel(R) Xeon(R) CPU E5-2680 v4 @ 2.40GHz | 14 | 2 |
| Intel(R) Xeon(R) CPU E5-2683 v4 @ 2.10GHz | 16 | 2 |
| Intel(R) Xeon(R) CPU E5-2690 0 @ 2.90GHz | 8 | 2 |
| Intel(R) Xeon(R) CPU E5-2690 v4 @ 2.60GHz | 14 | 2 |
| Intel(R) Xeon(R) CPU E5-2695 v4 @ 2.10GHz | 18 | 2 |
| Intel(R) Xeon(R) CPU E5-2696 v3 @ 2.30GHz | 18 | 2 |
| Intel(R) Xeon(R) CPU E5-2696 v4 @ 2.20GHz | 22 | 2 |
| Intel(R) Xeon(R) CPU E5-2698 v3 @ 2.30GHz | 16 | 2 |
| Intel(R) Xeon(R) CPU E5-2698 v4 @ 2.20GHz | 20 | 2 |
| Intel(R) Xeon(R) CPU E5-2699 v3 @ 2.30GHz | 1 | 1 |
| Intel(R) Xeon(R) CPU E5-2699 v4 @ 2.20GHz | 22 | 2 |
| Intel(R) Xeon(R) CPU E5-4667 v3 @ 2.00GHz | 16 | 2 |
| Intel(R) Xeon(R) Gold 5118 CPU @ 2.30GHz | 12 | 2 |
| Intel(R) Xeon(R) Gold 5218R CPU @ 2.10GHz | 20 | 2 |
| Intel(R) Xeon(R) Gold 5220R CPU @ 2.20GHz | 32 | 1 |
| Intel(R) Xeon(R) Gold 5318N CPU @ 2.10GHz | 24 | 2 |
| Intel(R) Xeon(R) Gold 5320 CPU @ 2.20GHz | 26 | 2 |
| Intel(R) Xeon(R) Gold 5420+ | 28 | 2 |
| Intel(R) Xeon(R) Gold 6130 CPU @ 2.10GHz | 16 | 2 |
| Intel(R) Xeon(R) Gold 6133 CPU @ 2.50GHz | 40 | 1 |
| Intel(R) Xeon(R) Gold 6136 CPU @ 3.00GHz | 12 | 2 |
| Intel(R) Xeon(R) Gold 6138 CPU @ 2.00GHz | 20 | 2 |
| Intel(R) Xeon(R) Gold 6150 CPU @ 2.70GHz | 18 | 2 |
| Intel(R) Xeon(R) Gold 6226 CPU @ 2.70GHz | 12 | 2 |
| Intel(R) Xeon(R) Gold 6230R CPU @ 2.10GHz | 8 | 2 |
| Intel(R) Xeon(R) Gold 6238R CPU @ 2.20GHz | 28 | 2 |
| Intel(R) Xeon(R) Gold 6240R CPU @ 2.40GHz | 24 | 2 |
| Intel(R) Xeon(R) Gold 6248R CPU @ 3.00GHz | 16 | 1 |
| Intel(R) Xeon(R) Gold 6252 CPU @ 2.10GHz | 24 | 1 |
| Intel(R) Xeon(R) Gold 6266C CPU @ 3.00GHz | 22 | 2 |
| Intel(R) Xeon(R) Gold 6342 CPU @ 2.80GHz | 24 | 2 |
| Intel(R) Xeon(R) Gold 6348 CPU @ 2.60GHz | 28 | 2 |
| Intel(R) Xeon(R) Gold 6448Y | 32 | 2 |
| INTEL(R) XEON(R) GOLD 6530 | 32 | 2 |
| INTEL(R) XEON(R) GOLD 6548Y+ | 32 | 2 |
| Intel(R) Xeon(R) Platinum 8160 CPU @ 2.10GHz | 24 | 2 |
| Intel(R) Xeon(R) Platinum 8168 CPU @ 2.70GHz | 24 | 2 |
| Intel(R) Xeon(R) Platinum 8171M CPU @ 2.60GHz | 26 | 2 |
| Intel(R) Xeon(R) Platinum 8173M CPU @ 2.00GHz | 28 | 2 |
| Intel(R) Xeon(R) Platinum 8176M CPU @ 2.10GHz | 28 | 2 |
| Intel(R) Xeon(R) Platinum 8180 CPU @ 2.50GHz | 28 | 2 |
| Intel(R) Xeon(R) Platinum 8352V CPU @ 2.10GHz | 36 | 2 |
| Intel(R) Xeon(R) Platinum 8352Y CPU @ 2.20GHz | 32 | 2 |
| Intel(R) Xeon(R) Platinum 8452Y | 36 | 2 |
| Intel(R) Xeon(R) Platinum 8460Y+ | 40 | 2 |
| Intel(R) Xeon(R) Platinum 8462Y+ | 32 | 2 |
| Intel(R) Xeon(R) Platinum 8468 | 48 | 2 |
| Intel(R) Xeon(R) Platinum 8468V | 44 | 2 |
| Intel(R) Xeon(R) Platinum 8470 | 52 | 2 |
| Intel(R) Xeon(R) Platinum 8480+ | 56 | 2 |
| Intel(R) Xeon(R) Platinum 8480C | 56 | 2 |
| Intel(R) Xeon(R) Platinum 8480CL | 56 | 2 |
| INTEL(R) XEON(R) PLATINUM 8558 | 48 | 2 |
| INTEL(R) XEON(R) PLATINUM 8568Y+ | 48 | 2 |
| INTEL(R) XEON(R) PLATINUM 8570 | 56 | 2 |
| INTEL(R) XEON(R) PLATINUM 8592V | 64 | 2 |
| Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz | 10 | 2 |
| Intel(R) Xeon(R) Silver 4210 CPU @ 2.20GHz | 10 | 2 |
| Intel(R) Xeon(R) Silver 4214 CPU @ 2.20GHz | 24 | 1 |
| Intel(R) Xeon(R) Silver 4310T CPU @ 2.30GHz | 10 | 2 |
| Intel(R) Xeon(R) Silver 4314 CPU @ 2.40GHz | 16 | 2 |
| Intel(R) Xeon(R) W-2223 CPU @ 3.60GHz | 4 | 2 |
| Intel(R) Xeon(R) w5-2455X | 12 | 2 |
| Intel(R) Xeon(R) w7-3465X | 28 | 2 |
| QEMU Virtual CPU version 2.5+ | 16 | 1 |
# GPU types
Source: https://docs.runpod.io/references/gpu-types
Explore the GPUs available on Runpod. Review supported options, requirements, and reference information for the Runpod platform.
For information on pricing, see [GPU pricing](https://www.runpod.io/pricing).
## GPU types
This table lists all GPU types available on Runpod:
| GPU ID | Display Name | Memory (GB) |
| -------------------------------------------------------- | ----------------- | ----------- |
| AMD Instinct MI300X OAM | MI300X | 192 |
| NVIDIA A100 80GB PCIe | A100 PCIe | 80 |
| NVIDIA A100-SXM4-80GB | A100 SXM | 80 |
| NVIDIA A30 | A30 | 24 |
| NVIDIA A40 | A40 | 48 |
| NVIDIA B200 | B200 | 180 |
| NVIDIA B300 SXM6 AC | B300 | 288 |
| NVIDIA GeForce RTX 3070 | RTX 3070 | 8 |
| NVIDIA GeForce RTX 3080 | RTX 3080 | 10 |
| NVIDIA GeForce RTX 3080 Ti | RTX 3080 Ti | 12 |
| NVIDIA GeForce RTX 3090 | RTX 3090 | 24 |
| NVIDIA GeForce RTX 3090 Ti | RTX 3090 Ti | 24 |
| NVIDIA GeForce RTX 4070 Ti | RTX 4070 Ti | 12 |
| NVIDIA GeForce RTX 4080 | RTX 4080 | 16 |
| NVIDIA GeForce RTX 4080 SUPER | RTX 4080 SUPER | 16 |
| NVIDIA GeForce RTX 4090 | RTX 4090 | 24 |
| NVIDIA GeForce RTX 5080 | RTX 5080 | 16 |
| NVIDIA GeForce RTX 5090 | RTX 5090 | 32 |
| NVIDIA H100 80GB HBM3 | H100 SXM | 80 |
| NVIDIA H100 NVL | H100 NVL | 94 |
| NVIDIA H100 PCIe | H100 PCIe | 80 |
| NVIDIA H200 | H200 SXM | 141 |
| NVIDIA H200 NVL | NVIDIA H200 NVL | 143 |
| NVIDIA L4 | L4 | 24 |
| NVIDIA L40 | L40 | 48 |
| NVIDIA L40S | L40S | 48 |
| NVIDIA RTX 2000 Ada Generation | RTX 2000 Ada | 16 |
| NVIDIA RTX 4000 Ada Generation | RTX 4000 Ada | 20 |
| NVIDIA RTX 4000 SFF Ada Generation | RTX 4000 Ada SFF | 20 |
| NVIDIA RTX 5000 Ada Generation | RTX 5000 Ada | 32 |
| NVIDIA RTX 6000 Ada Generation | RTX 6000 Ada | 48 |
| NVIDIA RTX A2000 | RTX A2000 | 6 |
| NVIDIA RTX A4000 | RTX A4000 | 16 |
| NVIDIA RTX A4500 | RTX A4500 | 20 |
| NVIDIA RTX A5000 | RTX A5000 | 24 |
| NVIDIA RTX A6000 | RTX A6000 | 48 |
| NVIDIA RTX PRO 4500 Blackwell | RTX PRO 4500 | 32 |
| NVIDIA RTX PRO 6000 Blackwell Max-Q Workstation Edition | RTX PRO 6000 MaxQ | 96 |
| NVIDIA RTX PRO 6000 Blackwell Server Edition MIG 1g.24gb | PRO 6000 MIG 24GB | 24 |
| NVIDIA RTX PRO 6000 Blackwell Server Edition MIG 2g.48gb | PRO 6000 MIG 48GB | 48 |
| NVIDIA RTX PRO 6000 Blackwell Server Edition | RTX PRO 6000 | 96 |
| NVIDIA RTX PRO 6000 Blackwell Workstation Edition | RTX PRO 6000 WK | 96 |
| Tesla V100-PCIE-16GB | Tesla V100 | 16 |
| Tesla V100-SXM2-16GB | V100 SXM2 | 16 |
| Tesla V100-SXM2-32GB | V100 SXM2 32GB | 32 |
## GPU pools
The table below lists the GPU pools that you can use to define which GPUs are available to workers on an endpoint after deployment.
Use GPU pools when defining requirements for repositories published to the [Runpod Hub](/hub/publishing-guide#runpod-configuration), or when specifying GPU requirements for an endpoint with the [Runpod GraphQL API](https://graphql-spec.runpod.io/).
| Pool ID | GPUs Included | Memory (GB) |
| :----------- | :------------------------------- | :---------- |
| `AMPERE_16` | A4000, A4500, RTX 4000, RTX 2000 | 16 GB |
| `AMPERE_24` | L4, A5000, 3090 | 24 GB |
| `ADA_24` | 4090 | 24 GB |
| `AMPERE_48` | A6000, A40 | 48 GB |
| `ADA_48_PRO` | L40, L40S, 6000 Ada | 48 GB |
| `AMPERE_80` | A100 | 80 GB |
| `ADA_80_PRO` | H100 | 80 GB |
| `HOPPER_141` | H200 | 141 GB |
# GraphQL Spec
Source: https://docs.runpod.io/references/graphql-spec
Explore the Runpod GraphQL schema reference for available queries, mutations, fields, arguments, inputs, and response types.
Use the GraphQL schema reference to explore every available query, mutation, field, and input.
Browse the complete schema on the Runpod GraphQL reference site.
# Data security and legal compliance
Source: https://docs.runpod.io/references/security-and-compliance
Information about data security, GDPR compliance, and legal resources. Review supported options and requirements for Runpod.
This page explains how Runpod secures your data, complies with privacy regulations, and where to find legal documentation.
## Multi-tenant isolation
Your Pods and workers run in a multi-tenant environment with containerized isolation that prevents other users from accessing your data. Each Pod/worker operates in its own container with strict separation from other workloads.
For sensitive workloads requiring enhanced security, Secure Cloud operates in T3/T4 data centers with enterprise-grade security, high reliability, redundancy, and fast response times.
## Host access policies
Runpod's [terms of service](https://www.runpod.io/legal/terms-of-service) prohibit hosts from inspecting your Pod/worker data or analyzing your usage patterns. Any violation results in immediate removal from the platform.
For workloads requiring the highest level of security, Secure Cloud provides vetted infrastructure partners who meet enterprise security standards including SOC 2, ISO 27001, and PCI DSS certifications.
## GDPR compliance
Runpod is fully compliant with the General Data Protection Regulation (GDPR) for data processed in European data center regions. The platform implements comprehensive policies, procedures, and technical measures to meet GDPR requirements.
### Compliance measures
For servers hosted in GDPR-compliant regions like the European Union, Runpod maintains clear procedures for the collection, storage, processing, and deletion of personal data. These procedures ensure transparency and accountability in data processing activities.
Technical and organizational measures protect personal data against unauthorized access, disclosure, alteration, and destruction. Runpod obtains and records consent from individuals for processing their personal data, and provides mechanisms for individuals to withdraw consent.
The platform facilitates data subject rights under GDPR, including the right to access, rectify, erase, or restrict the processing of personal data. Data subject requests are handled promptly and efficiently.
For lawful transfer of personal data outside the EU, Runpod uses appropriate mechanisms such as adequacy decisions, standard contractual clauses, or binding corporate rules. Regular monitoring and internal audits ensure ongoing GDPR compliance, including data protection impact assessments as needed.
## Legal resources
For detailed information about terms, policies, and legal agreements, visit the [Runpod legal page](https://www.runpod.io/legal/compliance).
# Video resources
Source: https://docs.runpod.io/references/video-resources
Browse community video tutorials covering Runpod setup, Stable Diffusion, text generation, file transfer, SSH, and Linux tools.
## Runpod Usage
| Video Link | Topics Covered |
| --------------------------------------------------------------------------------------------------------- | -------------- |
| [How to redeem your Runpod Coupon](https://www.youtube.com/watch?v=IYqEKwpuyWk\&ab_channel=OpenCVCourses) | Runpod |
| [Runpod Introduction and Tour](https://www.youtube.com/watch?v=6O1oM_N6pcw\&ab_channel=OpenCVCourses) | General |
***
## Tutorials
| Video Link | Topics Covered |
| ------------------------------------------------------------------------------------------------------------------------------- | ----------------------------- |
| [Generate Stable Diffusion Images FAST with Runpod](https://www.youtube.com/watch?v=susnjHSWFq0\&t=32s\&ab_channel=BillMeeks) | StableDiffusion, Automatic111 |
| [Generate Text On Images with DeepFloyd IF](https://www.youtube.com/watch?v=Px7Vv9WYl88\&t=2s\&ab_channel=BillMeeks) | DeepFloyd |
| [Remix Your Pics With Stable Diffusion and ControlNet](https://www.youtube.com/watch?v=BqdIdk9LU4w\&t=1s\&ab_channel=BillMeeks) | StableDiffusion, ControlNet |
| [Using Automatic1111 WebUI on Runpod](https://www.youtube.com/watch?v=R6HUQOtsVic\&ab_channel=OpenCVCourses) | WebUI, Automatic1111 |
| [RUN TextGen AI WebUI LLM On Runpod & Colab!](https://www.youtube.com/watch?v=TP2yID7Ubr4\&ab_channel=Aitrepreneur) | GoogleColab, TextGeneration |
| [Make Your Renders 10x Faster With Runpod](https://www.youtube.com/watch?v=sJ-Diy93TAg\&ab_channel=RahulAhire) | Rendering, Blender |
***
## General/Generic Linux
| Video Link | Topics Covered |
| ------------------------------------------------------------------------------------------------------------------------------------ | ------------------ |
| [How to Use the rsync Command \| Linux Essentials Tutorial](https://www.youtube.com/watch?v=2PnAohLS-Q4\&ab_channel=AkamaiDeveloper) | File Transfer |
| [SSH Key Authentication \| How to Create SSH Key Pairs](https://www.youtube.com/watch?v=33dEcCKGBO4\&ab_channel=AkamaiDeveloper) | SSH Authentication |
# Product updates
Source: https://docs.runpod.io/release-notes
New features, fixes, and improvements for the Runpod platform. Review setup, configuration, workflows, and usage guidance in the Runpod documentation.
**August 19, 2026**
New Release [Batch Jobs (BETA)](/serverless/batch-jobs)
Submit large sets of inference requests to a Serverless endpoint as a single managed unit. Create a batch, finalize it to start processing, and poll for status and progress using per-request counts. See [Batch Jobs](/serverless/batch-jobs) to get started.
**August 18, 2026**
New Release [REST API v2](/api-reference-v2/overview)
REST API v2 is now generally available. v2 moves to a new base URL (`https://api.runpod.io/v2`), reorganizes resource paths, standardizes request and response shapes, and adds new capabilities including catalog endpoints, pod log streaming, and Serverless observability. See the [migration guide](/api-reference-v2/migrate-from-v1) to move your existing integrations.
Deprecation REST API v1
REST API v1 will be retired on **November 15, 2026**. Migrate your integrations to [REST API v2](/api-reference-v2/overview) before that date. See the [migration guide](/api-reference-v2/migrate-from-v1) to get started.
Deprecation GraphQL API
The GraphQL API will be retired in **early 2027**. For new integrations, use [REST API v2](/api-reference-v2/overview).
**August 11, 2026**
New Release [ECR Integration (BETA)](/tutorials/pods/use-private-ecr-images)
You can now pull private container images from AWS ECR into [Pods and Serverless endpoints](/pods/overview) without migrating registries or managing credentials. ECR delegation is available in beta.
**July 23, 2026**
New Release [Runpod API v2 (BETA)](/api-reference-v2/overview)
A new REST API is available in public beta. See the [API v2 reference](/api-reference-v2/overview) to get started. The [GraphQL API](/api-reference/overview) and REST API v1 continue to work for now, but will be deprecated in a future release, so new integrations should build on API v2.
**July 17, 2026**
New Release [Scale Instant Clusters - BETA](/instant-clusters/scale-clusters)
You can now add pods to a running Instant Cluster to increase GPU capacity without recreating it. New pods join the cluster's private network automatically, matching the existing GPU type, template, and network storage. Currently in beta for eligible accounts.
**July 14, 2026**
New Release [Worker affinity for Serverless load balancer endpoints](/serverless/load-balancing/worker-affinity)
You can now pin follow-up requests to the same worker using the `X-Runpod-Worker-Id` response header. Send the header back on subsequent requests to choose from three modes: soft (prefer the worker, fall back to normal selection if unavailable), strict (route only to that worker, wait if at capacity), or strict-resume (same as strict, but automatically resumes a scaled-down worker). Useful for stateful workloads where session state is held in worker memory and re-routing would require reloading it.
**July 1, 2026**
The endpoint creation flow now offers six deployment paths — Hello World, Hugging Face LLM, Docker, GitHub, Flash, and Hub (which replaces the previous "Ready-to-Deploy Repos" option). Each path walks you through the right setup for your use case.
New Release [Deploy Pods with private AWS ECR images - BETA](/tutorials/pods/use-private-ecr-images)
New tutorial covering how to pull container images from private AWS ECR repositories into Runpod Pods using cross-account IAM delegation. Includes configuring ECR repository policies, adding ECR credentials in the Runpod\
console, and deploying a Pod with a private image, without managing credentials directly.
Breaking Lifecycle operations are now CLI-only
Flash SDK methods for endpoint and app lifecycle operations—deploy, undeploy, update, and creating or deleting apps and environments—now raise a `FlashUsageError` that points to the equivalent `flash` command. Run these operations through the [Flash CLI](/flash/cli/overview) instead, which keeps the build and manifest pipeline and local state tracking consistent.
New Release High-Performance Network Volumes now available
You can now attach high-performance network volumes to [Pods, Serverless endpoints, and Instant Clusters](/storage/network-volumes) for significantly faster model load times. Look for the purple diamond icon to identify compatible datacenters.
New Release Deploy When Available
You can now request a GPU that's currently out of capacity and get notified by email when it becomes available. Runpod saves your pod configuration so you can deploy immediately when capacity returns.
New Release Sign in with Runpod for the MCP server
You can now connect AI coding agents to a hosted [Runpod MCP server](/get-started/mcp-servers) over HTTP and authenticate with the "Sign in with Runpod" OAuth flow, so you no longer need to store a Runpod API key on disk. Local connections that use an API key continue to work as before.
New Release Expanded MCP server tools and REST API v2
The [Runpod MCP server](/get-started/mcp-servers) now covers more of the platform: you can run and manage Serverless jobs, discover GPU types and data centers, and manage tags, billing, and endpoint workers through natural language. List results are now paginated, and the server defaults to the Runpod REST API v2. Note that `create-endpoint` now takes an image name and GPU pool instead of a template ID; set `RUNPOD_REST_VERSION=v1` to keep the previous behavior.
Improvement Hub navigation consolidated
Hub navigation items are now consolidated into a single unified entry, making it easier to find templates and repos.
Improvement Guided install command for the MCP server
Running `npx @runpod/mcp-server add` now detects your installed agents — Claude Code, Claude Desktop, Cursor, Windsurf, and VS Code — and writes the [MCP configuration](/get-started/mcp-servers) for each one you select. Run `remove` to undo those changes.
Bug Fix Billing records now show correct data for deleted resources
SKU, region, and creation timestamps now appear correctly in [billing views and exports](/pods/pricing) for deleted Pods and network volumes.
New Release Async Jobs for Serverless
You can now submit a job to a [Serverless endpoint](/serverless/overview) and retrieve the result asynchronously when capacity is available. Jobs queue and process automatically when a worker is free, with no always-on workers or polling loops required.
New Release Serverless Worker Fitness Checks
Serverless workers now run automated health checks before accepting jobs. Runpod automatically removes unhealthy workers from rotation, reducing failed requests and improving endpoint reliability.
New Release 24GB MiG instances now available
You can now partition H100 and RTX PRO 6000 GPUs into up to seven independent [24GB MiG instances](/references/gpu-types), giving you more granular, lower-cost access without reserving a full card.
New Release Cost Centers now generally available
[Cost Centers](/accounts-billing/manage-accounts) let teams allocate and track GPU spend by project, team, or business unit. Detailed cost breakdowns are now available in billing, and all users receive itemized invoices as of May 1.
Improvement New Pod deploy flow with workload-first GPU selection
The Pod deployment experience has been redesigned. Instead of picking a GPU first, you now choose a template or workload type and get recommended GPUs ranked as recommended, compatible, or incompatible. The new flow includes Save as Template, AI-assisted GPU selection, and a Notify Me When Available option for out-of-capacity cards.
New Release Flash is now generally available
[Flash](/flash/overview) is now generally available. You can run Python functions on cloud GPUs with a single `@Endpoint` decorator, with no containers or infrastructure setup required. Workers scale automatically, dependencies install on remote workers, and you can deploy production APIs with `flash deploy`.
New Release Instant Cluster Expansion and Priority FlashBoot now live
[Instant Clusters](/instant-clusters) can now expand to more nodes faster. Priority FlashBoot reduces cold-start times for cluster workers. Both features are live with no configuration changes needed. Expanding an existing cluster is currently only available to Runpod admins. To add nodes to an existing cluster, reach out to the Runpod team.
New Release FlashBoot for CPU Serverless now in public beta
CPU Serverless workers now support FlashBoot, dramatically reducing cold-start times for your CPU endpoints. GA is planned for later this quarter.
Improvement GPU price reductions across popular SKUs
GPU prices have been reduced across a range of SKUs, lowering the cost of your training and inference workloads. Updated pricing is reflected in the console and [pricing page](/pods/pricing).
Bug Fix Serverless GPU exclusions now correctly respected
GPU type exclusions set on Serverless endpoints were not being enforced, causing workloads to land on excluded GPU types and resulting in incorrect billing. The issue is now fixed, and new alerting has been added to detect recurrence.
We've updated our release notes format for easier navigation. Updates from April 2026 onwards are listed above. Browse earlier releases by year and month in the archive below.
## Flash beta: Run Python functions on cloud GPUs
[Flash](/flash/overview) is now in public beta. Flash is a Python SDK that lets you run functions on Runpod Serverless GPUs with a single decorator:
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
from runpod_flash import Endpoint, GpuType
@Endpoint(
name="hello-gpu",
gpu=GpuType.NVIDIA_GEFORCE_RTX_4090,
dependencies=["torch"]
)
async def hello(): # This function runs on Runpod
import torch
gpu_name = torch.cuda.get_device_name(0)
print(f"Hello from your GPU! ({gpu_name})")
return {"gpu": gpu_name}
asyncio.run(hello())
print("Done!") # This runs locally
```
**Key features:**
* **Remote execution**: Mark functions with `@Endpoint` to run on GPUs/CPUs automatically.
* **Auto-scaling**: Workers scale from 0 to N based on demand.
* **Dependency management**: Packages install automatically on remote workers.
* **Two patterns**: Queue-based endpoints for batch work, load-balanced endpoints for REST APIs
* **Flash apps**: Build production-ready APIs with `flash init`, `flash dev`, and `flash deploy`
**Get started:**
Learn more about Flash.
Run your first GPU workload in 5 minutes.
Learn queue-based and load-balanced patterns.
Development and deployment commands.
## Flash: Multi-datacenter deployments
Flash now supports deploying endpoints to [multiple datacenters](/flash/configuration/parameters#datacenter) simultaneously. Pass a list of datacenters to distribute your workload across regions for improved availability and reduced latency. You can also attach [network volumes per datacenter](/flash/configuration/storage#multi-datacenter-volumes) for region-specific data access.
## New Public Endpoints and expanded examples
**[New Public Endpoints](/public-endpoints/reference):** Expansion of available models across all categories.
* **Video:** [SORA 2](/public-endpoints/models/sora-2) and [SORA 2 Pro](/public-endpoints/models/sora-2-pro), [Kling v2.1](/public-endpoints/models/kling-v2-1) and [v2.6 Motion Control](/public-endpoints/models/kling-v2-6-motion-control), [WAN 2.6](/public-endpoints/models/wan-2-6-t2v).
* **Image:** [Seedream 4.0](/public-endpoints/models/seedream-4-t2i).
* **Text:** [Qwen3 32B](/public-endpoints/models/qwen3-32b), [IBM Granite 4.0](/public-endpoints/models/granite-4).
* **Audio:** [Chatterbox Turbo](/public-endpoints/models/chatterbox-turbo) for text-to-speech.
**New integrations and guides:**
* [Vercel AI SDK integration](/public-endpoints/ai-sdk): New `@runpod/ai-sdk-provider` package for TypeScript projects with streaming, text generation, and image generation support.
* [AI coding tools guide](/public-endpoints/ai-coding-tools): Configure OpenCode, Cursor, and Cline to use Runpod Public Endpoints as your model provider.
**[New tutorials](/tutorials/introduction/overview):**
* [Build a text-to-video pipeline](/tutorials/public-endpoints/text-to-video-pipeline): Chain multiple Public Endpoints to generate videos from text prompts.
* [Deploy cached models](/tutorials/serverless/model-caching-text): Reduce cold start times with model caching.
* [Integrate Serverless with web applications](/tutorials/serverless/generate-sdxl-turbo): Build a complete image generation app.
* [Build a chatbot with Gemma 3](/tutorials/serverless/run-gemma-7b): Deploy vLLM with OpenAI API compatibility.
* [Run Ollama on Pods](/tutorials/pods/run-ollama): Set up Ollama for LLM inference.
* [Build Docker images with Bazel](/tutorials/pods/build-docker-images): Containerize your applications.
## GitHub release rollback GA and load balancing Serverless repos in beta
* [GitHub release rollback](/serverless/workers/github-integration#roll-back-to-a-previous-build): Roll back your Serverless endpoint to any previous build from the console. Restore an earlier version when you encounter issues without waiting for a new GitHub release.
* [Load balancing Serverless repos (beta)](/hub/publishing-guide): Load balancing endpoints are now available in the Hub. Publish or convert any listing to load balancer type by setting `"endpointType": "LB"` in your hub.json file, then deploy as a Serverless endpoint or Pod from the Hub page. Maintain a single listing for your model and let users choose their deployment method—autoscaling Serverless or dedicated Pod resources.
## Pod migration in beta and Serverless development guides
* [Pod migration (beta)](/pods/troubleshooting/pod-migration): Migrate your Pod to a new machine when your stopped Pod's GPU is occupied. Provisions a new Pod with the same specifications and automatically transfers your data to an available machine.
* [New Serverless development guides](/serverless/overview): We've added a comprehensive new set of guides for developing, testing, and debugging Serverless endpoints.
## Slurm Clusters GA, cached models in beta, and new Public Endpoints available
* [Slurm Clusters are now generally available](/instant-clusters/slurm-clusters): Deploy production-ready HPC clusters in seconds. These clusters support multi-node performance for distributed training and large-scale simulations with pay-as-you-go billing and no idle costs.
* [Cached models are now in beta](/serverless/endpoints/model-caching): Eliminate model download times when starting workers. The system places cached models on host machines before workers start, prioritizing hosts with your model already available for instant startup.
* [New Public Endpoints available](/public-endpoints/overview): [WAN 2.5](/public-endpoints/models/wan-2-5) combines image and audio to create lifelike videos, while [Nano Banana](/public-endpoints/models/nano-banana-edit) merges multiple images for composite creations.
## Hub revenue sharing launches and Pods UI gets refreshed
* [Hub revenue share model](/hub/revenue-sharing): Publish to the Runpod Hub and earn credits when others deploy your repo. Earn up to 7% of compute revenue through monthly tiers with credits auto-deposited into your account.
* [Pods UI updated](/pods/overview): Refreshed modern interface for interacting with Runpod Pods.
## Public Endpoints arrive, Slurm Clusters in beta
* [Public Endpoints](/public-endpoints/overview): Access state-of-the-art AI models through simple API calls with an integrated playground. Available endpoints include [Qwen Image Edit](/public-endpoints/models/qwen-image-edit), [Flux Kontext](/public-endpoints/models/flux-kontext-dev), [Cogito 671B](/public-endpoints/models/cogito-671b), and [Minimax Speech](/public-endpoints/models/minimax-speech).
* [Slurm Clusters (beta)](/instant-clusters/slurm-clusters): Create on-demand multi-node clusters instantly with full Slurm scheduling support.
## S3-compatible storage and updated referral program
* [S3-compatible API for network volumes](/storage/s3-api): Upload and retrieve files from your network volumes without compute using AWS S3 CLI or Boto3. Integrate Runpod storage into any AI pipeline with zero-config ease and object-level control.
* [Referral program revamp](/accounts-billing/referrals): Updated rewards and tiers with clearer dashboards to track performance.
## Port labeling, price drops, Runpod Hub, and Tetra beta test
* [Port labeling](/pods/overview): Name exposed ports in the UI and API to help team members identify services like Jupyter or TensorBoard.
* [Price drops](/pods/pricing): Additional price reductions on popular GPU SKUs to lower training and inference costs.
* [Runpod Hub](/hub/overview): A curated catalog of one-click endpoints and templates for deploying community projects without starting from scratch.
* **Tetra beta test**: A Python library for running code on GPU with Runpod. Add a `@remote()` decorator to functions that need GPU power while the rest of your code runs locally.
## GitHub login, RTX 5090s, and global networking expansion
* **Login with GitHub**: OAuth sign-in and linking for faster onboarding and repo-driven workflows.
* **RTX 5090s on Runpod**: High-performance RTX 5090 availability for cost-efficient training and inference.
* [Global networking expansion](/pods/networking): Rollout to additional data centers approaching full global coverage.
## Enterprise features arrive, REST API goes GA, Instant Clusters in beta, and APAC expansion
* [CPU Pods get network storage access](/storage/network-volumes): GA support for network volumes on CPU Pods for persistent, shareable storage.
* **SOC 2 Type I certification**: Independent attestation of security controls for enterprise readiness.
* [REST API release](/api-reference/overview): REST API GA with broad resource coverage for full infrastructure-as-code workflows.
* [Instant Clusters](/instant-clusters): Spin up multi-node GPU clusters in minutes with private interconnect and per-second billing.
* **Bare metal**: Reserve dedicated GPU servers for maximum control, performance, and long-term savings.
* **AP-JP-1**: New Fukushima region for low-latency APAC access and in-country data residency.
## REST API enters beta with full-time community manager
* [REST API beta test](/api-reference/overview): RESTful endpoints for Pods, endpoints, and volumes for simpler automation than GraphQL.
* **Full-time community manager hire**: Dedicated programs, content, and faster community response.
* [Serverless GitHub integration release](/serverless/workers/github-integration): GA for GitHub-based Serverless deploys with production-ready stability.
## New silicon and LLM-focused Serverless upgrades
* **CPU Pods v2**: Docker runtime parity with GPU Pods for faster starts with network volume support.
* [H200s on Runpod](/references/gpu-types): NVIDIA H200 GPUs available for larger models and higher memory bandwidth.
* [Serverless upgrades](/serverless/overview): Higher GPU counts per worker, new quick-deploy runtimes, and simpler model selection.
## Global networking expands and GitHub deploys enter beta
* [Global networking expansion](/pods/networking): Added to CA-MTL-3, US-GA-1, US-GA-2, and US-KS-2 for expanded private mesh coverage.
* [Serverless GitHub integration beta test](/serverless/workers/github-integration): Deploy endpoints directly from GitHub repos with automatic builds.
* **Scoped API keys**: Least-privilege tokens with fine-grained scopes and expirations for safer automation.
* **Passkey auth**: Passwordless WebAuthn sign-in for phishing-resistant account access.
## Storage expansion and private cross-data-center connectivity
* [US-GA-2 added to network storage](/storage/network-volumes): Enable network volumes in US-GA-2.
* [Global networking](/pods/networking): Private cross-data-center networking with internal DNS for secure service-to-service traffic.
## Storage coverage grows with major price cuts and revamped referrals
* **US-TX-3 and EUR-IS-1 added to network storage**: Network volumes available in more regions for local persistence.
* **Runpod slashes GPU prices**: Broad GPU price reductions to lower training and inference total cost of ownership.
* [Referral program revamp](/accounts-billing/referrals): Updated commissions and bonuses with an affiliate tier and improved tracking.
## \$20M seed round, community event, and broader Serverless options
* **\$20M seed by Intel Capital and Dell Technologies Capital**: Funds infrastructure expansion and product acceleration.
* **First in-person hackathon**: Community projects, workshops, and real-world feedback.
* [Serverless CPU Pods](/references/cpu-types): Scale-to-zero CPU endpoints for services that don't need a GPU.
* [AMD GPUs](/references/gpu-types): AMD ROCm-compatible GPU SKUs as cost and performance alternatives to NVIDIA.
## CPU compute and first-class automation tooling
* **CPU Pods**: CPU-only instances with the same networking and storage primitives for cheaper non-GPU stages.
* [runpodctl](/runpodctl/overview): Official CLI for Pods, endpoints, and volumes to enable scripting and CI/CD workflows.
## Console navigation overhaul and documentation refresh
* **New navigational changes to Runpod UI**: Consolidated menus, consistent action placement, and fewer clicks for common tasks.
* **Docs revamp**: New information architecture, improved search, and more runnable examples and quickstarts.
* **Zhen AMA**: Roadmap Q\&A and community feedback session.
## New regions and investment in community support
* **US-OR-1**: Additional US region for lower latency and more capacity in the Pacific Northwest.
* **CA-MTL-1**: New Canadian region to improve latency and meet in-country data needs.
* **First community manager hire**: Dedicated community programs and faster feedback loops.
* **Building out the support team**: Expanded coverage and expertise for complex issues.
## Faster template starts and better multi-region hygiene
* **Serverless quick deploy**: One-click deploy of curated model templates with sensible defaults.
* **EU domain for Serverless**: EU-specific domain briefly offered for data residency, superseded by other region controls.
* **Data-center filter for Serverless**: Filter and manage endpoints by region for multi-region fleets.
## Self-service upgrades, clearer metrics, new pricing model, and cost visibility
* **Self-service worker upgrade**: Rebuild and roll workers from the dashboard without support tickets.
* **Edit template from endpoint page**: Inline edit and redeploy the underlying template directly from the endpoint view.
* **Improved Serverless metrics page**: Refinements to charts and filters for quicker root-cause analysis.
* [Flex and active workers](/serverless/pricing): Always-on "active" workers for baseline load with on-demand "flex" workers for bursts.
* **Billing explorer**: Inspect costs by resource, region, and time to identify optimization opportunities.
## Team governance, storage expansion, and better debugging
* [Teams](/accounts-billing/manage-accounts): Organization workspaces with role-based access control for Pods, endpoints, and billing.
* [Savings plans](/pods/pricing): Plans surfaced prominently in console with easier purchase and management for steady usage.
* **Network storage to US-KS-1**: Enable network volumes in US-KS-1 for local, persistent data workflows.
* [Serverless log view](/serverless/development/logs): Stream worker stdout and stderr in the UI and API for real-time debugging.
* **Serverless health endpoint**: Lightweight /health probe returning endpoint and worker status without creating a billable job.
* **SOC 2 Type II compliant**: Security and compliance certification for enterprise customers.
## Observability, top-tier GPUs, and commitment-based savings
* **Serverless metrics page**: Time-series charts for pXX latencies, queue delay, throughput, and worker states for faster debugging and tuning.
* [H100s on Runpod](/references/gpu-types): NVIDIA H100 instances for higher throughput and larger model footprints.
* [Savings plans](/pods/pricing): Commitment-based discounts for predictable workloads to lower effective hourly rates.
## Smoother auth and multi-region Serverless with persistent storage
* **The new and improved Runpod login experience**: Streamlined sign-in and team access for faster, more consistent auth flows.
* [Network volumes added to Serverless](/storage/network-volumes): Attach persistent storage to Serverless workers to retain models and artifacts across restarts and speed cold starts through caching.
* **Serverless region support**: Pin or allow specific regions for endpoints to reduce latency and meet data-residency needs.
## Deeper autoscaling controls, richer metrics, persistent storage, and job cancellation
* **Serverless scaling strategies**: Scale by queue delay and/or concurrency with min/max worker bounds to balance latency and cost.
* **Queue delay**: Expose time-in-queue as a first-class metric to drive autoscaling and SLO monitoring.
* **Request count**: Track success and failure totals over windows for quick health checks and alerting.
* **runsync**: Synchronous invocation path that returns results in the same HTTP call for short-running jobs.
* **Network storage beta**: Region-scoped, attachable volumes shareable across Pods and endpoints for model caches and datasets.
* **Job cancel API**: Programmatically terminate queued or running jobs to free capacity and enforce client timeouts.
## Serverless platform hardens with cleaner API
* **Serverless API v2**: Revised request and response schema with improved error semantics and new endpoints for better control over job lifecycle and observability.
## Better control over notifications and GPU allocation
* **Notification preferences**: Configure which platform events trigger alerts to reduce noise for teams and CI systems.
* **GPU priorities**: Influence scheduling by marking workloads as higher priority to reduce queue time for critical jobs.
## Encrypted volumes for persistent data
* **Runpod now offers encrypted volumes**: Enable at-rest encryption for persistent volumes with no application changes required using platform-managed keys.
# Overview
Source: https://docs.runpod.io/runpodctl/overview
Use Runpod CLI to manage Pods, Serverless endpoints, templates, and more from your local machine. Review commands and usage guidance for the Runpod CLI.
Runpod CLI is an [open source](https://github.com/runpod/runpodctl) command-line tool for managing your Runpod resources from your local machine. You can manage Pods, Serverless endpoints, templates, network volumes, and models, transfer files between your system and Runpod, diagnose issues, and view account information.
## Quick start
After installing and configuring `runpodctl`, you can start managing resources immediately:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
runpodctl doctor # First time setup (API key + SSH)
runpodctl gpu list # See available GPUs
runpodctl template search pytorch # Find a template
runpodctl pod create --template-id runpod-torch-v21 --gpu-id "NVIDIA RTX 4090"
runpodctl pod list # List your Pods
```
## Install Runpod CLI locally
Every you deploy comes preinstalled with `runpodctl` and a Pod-scoped API key. You can also install it on your local machine to manage resources remotely.
### Step 1: Choose an installation method
Choose the installation method that matches your operating system.
The install script automatically detects your operating system and architecture:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
bash <(wget -qO- cli.runpod.io)
```
Or using curl:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
bash <(curl -sL cli.runpod.io)
```
The script installs the latest version to `/usr/local/bin` when run as root, or to `~/.local/bin` for non-root users. If you don't have root access, the script displays instructions for adding `runpodctl` to your PATH.
The install script verifies the downloaded release against its SHA-256 checksum before installing, so it needs a checksum tool to be present: `sha256sum` on Linux or `shasum` on macOS. These are preinstalled in most environments, but on minimal images without one the install aborts.
**Homebrew:**
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
brew install runpod/runpodctl/runpodctl
```
**ARM (Apple Silicon):**
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
wget --quiet --show-progress https://github.com/runpod/runpodctl/releases/latest/download/runpodctl-darwin-arm64 -O runpodctl && chmod +x runpodctl && sudo mv runpodctl /usr/local/bin/runpodctl
```
**AMD (Intel):**
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
wget --quiet --show-progress https://github.com/runpod/runpodctl/releases/latest/download/runpodctl-darwin-amd64 -O runpodctl && chmod +x runpodctl && sudo mv runpodctl /usr/local/bin/runpodctl
```
**AMD64 (x86\_64):**
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
wget --quiet --show-progress https://github.com/runpod/runpodctl/releases/latest/download/runpodctl-linux-amd64 -O runpodctl && chmod +x runpodctl && sudo cp runpodctl /usr/bin/runpodctl
```
**ARM64 (aarch64):**
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
wget --quiet --show-progress https://github.com/runpod/runpodctl/releases/latest/download/runpodctl-linux-arm64 -O runpodctl && chmod +x runpodctl && sudo cp runpodctl /usr/bin/runpodctl
```
**Non-root installation:**
If you don't have root access, install to your user directory:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
mkdir -p ~/.local/bin && wget --quiet --show-progress https://github.com/runpod/runpodctl/releases/latest/download/runpodctl-linux-amd64 -O ~/.local/bin/runpodctl && chmod +x ~/.local/bin/runpodctl
```
Then add `~/.local/bin` to your PATH by adding this line to your `~/.bashrc` or `~/.zshrc`:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
export PATH="$HOME/.local/bin:$PATH"
```
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
wget https://github.com/runpod/runpodctl/releases/latest/download/runpodctl-windows-amd64.exe -O runpodctl.exe
```
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
!wget --quiet --show-progress https://github.com/runpod/runpodctl/releases/latest/download/runpodctl-linux-amd64 -O runpodctl
!chmod +x runpodctl
!cp runpodctl /usr/bin/runpodctl
```
Pre-built binaries are available on [conda-forge](https://anaconda.org/conda-forge/runpodctl) for Linux (x86\_64, aarch64, ppc64le), macOS (x86\_64, arm64), and Windows (x86\_64).
**conda:**
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
conda install conda-forge::runpodctl
```
**mamba:**
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
mamba install conda-forge::runpodctl
```
**pixi:**
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
pixi global install runpodctl
```
This installs `runpodctl` globally on your system, so you can run commands from any directory.
### Step 2: Configure your API key
Before you can use `runpodctl` locally, you must configure it with an [API key](/get-started/api-keys).
The easiest way to set up your API key and SSH configuration is with the `doctor` command:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
runpodctl doctor
```
This command guides you through first-time setup, including API key configuration and SSH key setup.
Alternatively, you can manually configure your API key:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
runpodctl config --apiKey YOUR_API_KEY
```
After running the command, you should see a confirmation message similar to this:
```text theme={"theme":{"light":"github-light","dark":"github-dark"}}
saved apiKey into config file: /Users/runpod/.runpod/config.toml
```
### Step 3: Verify installation
To verify that `runpodctl` installed successfully, run this command:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
runpodctl version
```
You should see which version is installed:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
runpodctl v2.0
```
## Command groups
Runpod CLI organizes commands into groups based on the resource type:
| Command | Alias | Description |
| -------------------------- | ----- | ----------------------------------------------- |
| `runpodctl pod` | | Manage Pods (create, list, start, stop, delete) |
| `runpodctl serverless` | `sls` | Manage Serverless endpoints |
| `runpodctl template` | `tpl` | List, search, and manage templates |
| `runpodctl hub` | | Browse and deploy from the Runpod Hub |
| `runpodctl network-volume` | `nv` | Manage network volumes |
| `runpodctl registry` | `reg` | Manage container registry authentications |
| `runpodctl gpu` | | List available GPUs |
| `runpodctl datacenter` | `dc` | List datacenters |
| `runpodctl billing` | | View billing history |
| `runpodctl user` | `me` | View account information |
| `runpodctl ssh` | | Manage SSH keys and get connection info |
## Help and reference
Learn how to use Runpod CLI commands by browsing the CLI reference using the sidebar to the left, or by running the `help` command:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
runpodctl help
```
Learn more about a particular command by running:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
runpodctl [command] --help
```
## Shell completion
Enable tab completion for your shell to make working with `runpodctl` easier:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
runpodctl completion
```
This command auto-detects your shell and adds the appropriate source command to your shell configuration file (`~/.bashrc` or `~/.zshrc`). The command is idempotent—running it again skips installation if completion is already configured.
Restart your shell or source the configuration file for changes to take effect.
# billing
Source: https://docs.runpod.io/runpodctl/reference/runpodctl-billing
Use runpodctl to retrieve billing history for Pods, Serverless endpoints, and network volumes across configurable time ranges.
View billing history for Pods, Serverless endpoints, and network volumes.
```bash Command theme={"theme":{"light":"github-light","dark":"github-dark"}}
runpodctl billing [flags]
```
## Subcommands
### View Pod billing
View billing history for Pods:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
runpodctl billing pods
```
#### Pod billing flags
Time bucket size (`hour`, `day`, `week`, `month`, `year`).
Start time in RFC3339 format (e.g., `2024-01-01T00:00:00Z`).
End time in RFC3339 format.
Group results by `podId` or `gpuId`.
Filter by specific Pod ID.
Filter by specific GPU type.
### View Serverless billing
View billing history for Serverless endpoints:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
runpodctl billing serverless
```
#### Serverless billing flags
Time bucket size (`hour`, `day`, `week`, `month`, `year`).
Start time in RFC3339 format.
End time in RFC3339 format.
Group results by `endpointId`, `podId`, or `gpuId`.
Filter by specific endpoint ID.
Filter by specific GPU type.
### View network volume billing
View billing history for network volumes:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
runpodctl billing network-volume
```
#### Network volume billing flags
Time bucket size (`hour`, `day`, `week`, `month`, `year`).
Start time in RFC3339 format.
End time in RFC3339 format.
## Related commands
* [`runpodctl user`](/runpodctl/reference/runpodctl-user)
# config
Source: https://docs.runpod.io/runpodctl/reference/runpodctl-config
Configure runpodctl with your Runpod API key and API endpoint URL so the CLI can authenticate and manage account resources.
Configure the Runpod CLI with your API credentials and API URL to enable programmatic access to your Runpod resources.
```bash Command theme={"theme":{"light":"github-light","dark":"github-dark"}}
runpodctl config [flags]
```
## Example
Configure the CLI with your API key:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
runpodctl config --apiKey "your-api-key-here"
```
For first-time setup, we recommend using [`runpodctl doctor`](/runpodctl/reference/runpodctl-doctor) instead, which guides you through API key configuration and SSH key setup interactively.
## Flags
Your Runpod API key, which authenticates the CLI to access your account. You can generate an API key from the [Runpod console](https://console.runpod.io/user/settings).
The Runpod API endpoint URL. The default value should work for most users.
## Related commands
* [`runpodctl doctor`](/runpodctl/reference/runpodctl-doctor)
* [`runpodctl user`](/runpodctl/reference/runpodctl-user)
# datacenter
Source: https://docs.runpod.io/runpodctl/reference/runpodctl-datacenter
Use runpodctl to list Runpod data centers, review their locations, and inspect current GPU availability from the command line.
List available datacenters and their locations.
```bash Command theme={"theme":{"light":"github-light","dark":"github-dark"}}
runpodctl datacenter [flags]
```
## Alias
You can use `dc` as a shorthand for `datacenter`:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
runpodctl dc list
```
## Subcommands
### List datacenters
List all available datacenters with GPU availability:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
runpodctl datacenter list
```
## Example output
```json theme={"theme":{"light":"github-light","dark":"github-dark"}}
[
{
"gpuAvailability": [
{
"displayName": "RTX 4090",
"gpuId": "NVIDIA GeForce RTX 4090",
"stockStatus": "High"
}
],
"id": "US-GA-1",
"location": "United States",
"name": "US-GA-1"
},
{
"gpuAvailability": [
{
"displayName": "A100 PCIe",
"gpuId": "NVIDIA A100 80GB PCIe",
"stockStatus": "High"
}
],
"id": "EU-RO-1",
"location": "Europe",
"name": "EU-RO-1"
}
]
```
## Using datacenter IDs
When creating network volumes or specifying preferred datacenters for Pods, use the datacenter ID:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
# Create a network volume in a specific datacenter
runpodctl network-volume create --name "my-volume" --size 100 --data-center-id "US-GA-1"
# Create a Pod with preferred datacenter
runpodctl pod create --template-id runpod-torch-v21 --gpu-id "NVIDIA RTX 4090" --data-center-ids "US-GA-1"
```
## Related commands
* [`runpodctl network-volume create`](/runpodctl/reference/runpodctl-network-volume)
* [`runpodctl pod create`](/runpodctl/reference/runpodctl-pod)
* [`runpodctl gpu list`](/runpodctl/reference/runpodctl-gpu)
# doctor
Source: https://docs.runpod.io/runpodctl/reference/runpodctl-doctor
Run runpodctl doctor to configure API and SSH credentials, verify the CLI setup, and diagnose common connection or configuration problems.
Diagnose and fix CLI issues, including first-time setup for API keys and SSH configuration.
```bash Command theme={"theme":{"light":"github-light","dark":"github-dark"}}
runpodctl doctor
```
## Description
The `doctor` command is the recommended way to set up `runpodctl` for the first time. It guides you through:
* Configuring your API key
* Setting up SSH keys for Pod access
* Verifying your configuration
If you encounter issues with the CLI, running `doctor` can help diagnose and fix common problems.
## Example
Run the doctor command to set up or troubleshoot your CLI:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
runpodctl doctor
```
The command will interactively guide you through the setup process and report any issues it finds.
## Related commands
* [`runpodctl config`](/runpodctl/reference/runpodctl-config)
* [`runpodctl ssh add-key`](/runpodctl/reference/runpodctl-ssh)
# gpu
Source: https://docs.runpod.io/runpodctl/reference/runpodctl-gpu
Use runpodctl to list available GPU types, include unavailable options, and review hardware specifications from the command line.
List available GPU types and their specifications.
```bash Command theme={"theme":{"light":"github-light","dark":"github-dark"}}
runpodctl gpu [flags]
```
## Subcommands
### List available GPUs
List GPUs that are currently available:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
runpodctl gpu list
```
Include unavailable GPUs in the list:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
runpodctl gpu list --include-unavailable
```
#### List flags
Include GPUs that are currently unavailable.
## Example output
```json theme={"theme":{"light":"github-light","dark":"github-dark"}}
[
{
"available": true,
"communityCloud": true,
"displayName": "RTX 4090",
"gpuId": "NVIDIA GeForce RTX 4090",
"memoryInGb": 24,
"secureCloud": true,
"stockStatus": "High"
},
{
"available": true,
"communityCloud": true,
"displayName": "A100 PCIe",
"gpuId": "NVIDIA A100 80GB PCIe",
"memoryInGb": 80,
"secureCloud": true,
"stockStatus": "High"
}
]
```
## Using GPU IDs
When creating Pods or Serverless endpoints, use the GPU ID from the list with the `--gpu-id` flag:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
runpodctl pod create --template-id runpod-torch-v21 --gpu-id "NVIDIA RTX 4090"
```
## Related commands
* [`runpodctl pod create`](/runpodctl/reference/runpodctl-pod)
* [`runpodctl serverless create`](/runpodctl/reference/runpodctl-serverless)
* [`runpodctl datacenter list`](/runpodctl/reference/runpodctl-datacenter)
# hub
Source: https://docs.runpod.io/runpodctl/reference/runpodctl-hub
Browse and search Runpod Hub repos with runpodctl, inspect repository details, and use Hub IDs to deploy Serverless endpoints.
Browse and search the Runpod Hub marketplace to discover deployable repos. You can list popular repos, search by name, and get details for specific repos. Use Hub repo IDs with [`runpodctl serverless create --hub-id`](/runpodctl/reference/runpodctl-serverless) to deploy endpoints directly from the Hub.
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
runpodctl hub [flags]
```
## Subcommands
### List repos
List repos from the Hub marketplace. By default, shows the top 10 repos ordered by stars.
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
# List top repos by stars
runpodctl hub list
# List only Serverless repos
runpodctl hub list --type SERVERLESS
# List only Pod repos
runpodctl hub list --type POD
# Filter by category
runpodctl hub list --category ai --limit 20
# Order by deployment count
runpodctl hub list --order-by deploys
# Filter by repo owner
runpodctl hub list --owner runpod
```
#### List flags
Filter by deployment type (`POD` or `SERVERLESS`). This filter is applied client-side, so `--limit` may return fewer results than specified.
Filter by category.
Filter by repo owner.
Sort results by field: `createdAt`, `deploys`, `releasedAt`, `stars`, `updatedAt`, or `views`.
Sort direction: `asc` or `desc`.
Maximum number of results to return.
Number of results to skip for pagination.
### Search repos
Search for repos in the Hub by name:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
# Search for vLLM repos
runpodctl hub search vllm
# Search Serverless repos only
runpodctl hub search whisper --type SERVERLESS
# Limit search results
runpodctl hub search stable-diffusion --limit 5
```
#### Search flags
Filter by deployment type (`POD` or `SERVERLESS`). This filter is applied client-side, so `--limit` may return fewer results than specified.
Filter by category.
Filter by repo owner.
Sort results by field: `createdAt`, `deploys`, `releasedAt`, `stars`, `updatedAt`, or `views`.
Sort direction: `asc` or `desc`.
Maximum number of results to return.
Number of results to skip for pagination.
### Get repo details
Get detailed information about a specific Hub repo by its ID or owner/name:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
# Get by listing ID
runpodctl hub get cm8h09d9n000008jvh2rqdsmb
# Get by owner/name
runpodctl hub get runpod-workers/worker-vllm
```
## Deploy from the Hub
After finding a repo you want to deploy, use its listing ID with `serverless create`:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
# Find a repo
runpodctl hub search vllm
# Deploy it
runpodctl serverless create --hub-id cm8h09d9n000008jvh2rqdsmb --name "my-vllm"
```
GPU IDs and container disk size are automatically pulled from the Hub release config. You can override the GPU type with `--gpu-id`.
## Related commands
* [`runpodctl serverless create`](/runpodctl/reference/runpodctl-serverless)
* [`runpodctl template search`](/runpodctl/reference/runpodctl-template)
# network-volume
Source: https://docs.runpod.io/runpodctl/reference/runpodctl-network-volume
Use runpodctl to create, inspect, list, update, and remove persistent network volumes shared by Pods and Serverless endpoints.
Manage network volumes for persistent shared storage across Pods and Serverless endpoints.
```bash Command theme={"theme":{"light":"github-light","dark":"github-dark"}}
runpodctl network-volume [flags]
```
## Alias
You can use `nv` as a shorthand for `network-volume`:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
runpodctl nv list
```
## Subcommands
### List network volumes
List all your network volumes:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
runpodctl network-volume list
```
### Get network volume details
Get detailed information about a specific network volume:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
runpodctl network-volume get
```
### Create a network volume
Create a new network volume:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
runpodctl network-volume create --name "my-volume" --size 100 --data-center-id "US-GA-1"
```
#### Create flags
Volume name.
Volume size in GB (1-4000).
Datacenter ID where the volume will be created. Use [`runpodctl datacenter list`](/runpodctl/reference/runpodctl-datacenter) to see available datacenters.
### Update a network volume
Update network volume configuration:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
runpodctl network-volume update --name "new-name"
```
#### Update flags
New volume name.
New volume size in GB. Must be larger than the current size.
### Delete a network volume
Delete a network volume:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
runpodctl network-volume delete
```
Deleting a network volume permanently removes all data stored on it. Make sure to back up any important data before deleting.
## Related commands
* [`runpodctl pod create`](/runpodctl/reference/runpodctl-pod)
* [`runpodctl datacenter list`](/runpodctl/reference/runpodctl-datacenter)
# pod
Source: https://docs.runpod.io/runpodctl/reference/runpodctl-pod
Use runpodctl to create, list, start, stop, inspect, and delete Pods from the command line with status and resource filters.
Manage Pods, including creating, listing, starting, stopping, and deleting Pods.
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
runpodctl pod [flags]
```
## Subcommands
### List Pods
List your Pods. By default, this command shows only running Pods (similar to `docker ps`):
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
runpodctl pod list
```
List all Pods including exited ones:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
runpodctl pod list --all
```
Filter by status:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
runpodctl pod list --status exited
```
Filter by creation time:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
# Pods created in the last 24 hours
runpodctl pod list --since 24h
# Pods created in the last 7 days
runpodctl pod list --since 7d
# Pods created after a specific date
runpodctl pod list --created-after 2025-01-15
```
#### List flags
Show all Pods including exited ones. By default, only running Pods are shown.
Filter by Pod status (e.g., `RUNNING`, `EXITED`). Cannot be used with `--all`.
Filter Pods created within the specified duration (e.g., `1h`, `24h`, `7d`). Cannot be used with `--created-after`.
Filter Pods created after the specified date in `YYYY-MM-DD` format. Cannot be used with `--since`.
Filter by compute type (`GPU` or `CPU`).
Filter by Pod name.
### Get Pod details
Get detailed information about a specific Pod, including SSH connection info:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
runpodctl pod get
```
### Create a Pod
Create a new Pod from a template:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
runpodctl pod create --template-id runpod-torch-v21 --gpu-id "NVIDIA GeForce RTX 4090"
```
Create a Pod with a custom Docker image:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
runpodctl pod create --image "runpod/pytorch:1.0.3-cu1281-torch291-ubuntu2404" --gpu-id "NVIDIA GeForce RTX 4090"
```
Create a CPU-only Pod:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
runpodctl pod create --compute-type cpu --image ubuntu:22.04
```
#### Create flags
Template ID to use for Pod configuration. Use [`runpodctl template search`](/runpodctl/reference/runpodctl-template) to find templates.
Docker image to use (e.g., `runpod/pytorch:2.8.0-py3.11-cuda12.8.1-cudnn-devel-ubuntu22.04`). Required if no template specified.
Custom name for the Pod.
GPU type (e.g., `NVIDIA GeForce RTX 4090`, `NVIDIA A100 80GB PCIe`). Use [`runpodctl gpu list`](/runpodctl/reference/runpodctl-gpu) to see available GPUs.
Number of GPUs to allocate.
Compute type (`GPU` or `CPU`).
Container disk size in GB.
Persistent volume size in GB.
Mount path for the persistent volume.
Comma-separated list of ports to expose (e.g., `8888/http,22/tcp`).
Environment variables as a JSON object (e.g., `'{"KEY":"value"}'`).
Cloud tier (`SECURE` or `COMMUNITY`).
Comma-separated list of preferred datacenter IDs. Use [`runpodctl datacenter list`](/runpodctl/reference/runpodctl-datacenter) to see available datacenters.
Enable global networking (Secure Cloud only).
Require public IP (Community Cloud only).
Enable SSH on the Pod.
Network volume ID to attach. Use [`runpodctl network-volume list`](/runpodctl/reference/runpodctl-network-volume) to see available network volumes.
Minimum CUDA version required (e.g., `11.8`, `12.4`). The Pod will only be scheduled on machines that meet this CUDA version requirement.
Docker arguments passed to the container at runtime (e.g., `"sleep infinity"`).
Container registry authentication ID for pulling private images. Use [`runpodctl registry list`](/runpodctl/reference/runpodctl-registry) to see available registry credentials.
Country code for regional deployment (e.g., `US`, `CA`, `EU`). Restricts Pod placement to machines in the specified region.
Compliance settings for the Pod (e.g., regulatory requirements for data handling).
### Start a Pod
Start a stopped Pod:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
runpodctl pod start
```
### Stop a Pod
Stop a running Pod:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
runpodctl pod stop
```
### Restart a Pod
Restart a Pod:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
runpodctl pod restart
```
### Reset a Pod
Reset a Pod to its initial state:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
runpodctl pod reset
```
### Update a Pod
Update Pod configuration:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
runpodctl pod update --name "new-name"
```
#### Update flags
New name for the Pod.
New Docker image name.
New container disk size in GB.
New volume size in GB.
New volume mount path.
New comma-separated list of ports.
New environment variables as a JSON object.
### Delete a Pod
Delete a Pod:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
runpodctl pod delete
```
## Pod URLs
Access exposed ports on your Pod using the following URL pattern:
```
https://-.proxy.runpod.net
```
For example, if your Pod ID is `abc123xyz` and you exposed port 8888:
```
https://abc123xyz-8888.proxy.runpod.net
```
## Related commands
* [`runpodctl gpu list`](/runpodctl/reference/runpodctl-gpu)
* [`runpodctl template`](/runpodctl/reference/runpodctl-template)
* [`runpodctl ssh`](/runpodctl/reference/runpodctl-ssh)
# receive
Source: https://docs.runpod.io/runpodctl/reference/runpodctl-receive
Receive files or folders through runpodctl using a secure peer-to-peer connection code shared by the sending machine.
Receive files or folders sent from another machine using a secure peer-to-peer connection established with a connection code.
```bash Command theme={"theme":{"light":"github-light","dark":"github-dark"}}
runpodctl receive
```
## Example
Receive files using a connection code:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
runpodctl receive rainbow-unicorn-42
```
## Arguments
The connection code phrase that matches the code used by the sender with the [`send`](/runpodctl/reference/runpodctl-send) command.
## Related commands
* [`runpodctl send`](/runpodctl/reference/runpodctl-send)
# registry
Source: https://docs.runpod.io/runpodctl/reference/runpodctl-registry
Use runpodctl to list, inspect, create, update, and remove credentials for private container registries used by Runpod workloads.
Manage container registry authentications for private Docker images.
```bash Command theme={"theme":{"light":"github-light","dark":"github-dark"}}
runpodctl registry [flags]
```
## Alias
You can use `reg` as a shorthand for `registry`:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
runpodctl reg list
```
## Subcommands
### List registry authentications
List all your container registry authentications:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
runpodctl registry list
```
### Get registry authentication details
Get details about a specific registry authentication:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
runpodctl registry get
```
### Create a registry authentication
Create credentials for a private container registry. Supply the password in one of three ways:
* Pass it directly with the `--password` flag.
* Pipe or redirect it into `--password-stdin`.
* Omit both flags to enter it at an interactive prompt.
Pass the password directly with `--password`:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
runpodctl registry create --name "docker-hub" --username "myuser" --password "mypassword"
```
Omit the password flag to have the command prompt for the password without echoing it:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
runpodctl registry create --name "docker-hub" --username "myuser"
```
Pipe a token held in an environment variable into `--password-stdin`:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
printenv REGISTRY_TOKEN | runpodctl registry create --name "docker-hub" --username "myuser" --password-stdin
```
Redirect a token from a file into `--password-stdin`:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
runpodctl registry create --name "docker-hub" --username "myuser" --password-stdin < token.txt
```
For Google Artifact Registry or gcr.io, set `--username` to the literal `_json_key` and feed the multi-line service-account JSON key file through `--password-stdin`. The command supports multi-line passwords and strips only a single trailing newline added by the pipe or redirection.
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
runpodctl registry create --name "gcr" --username "_json_key" --password-stdin < service-account.json
```
Prefer `--password-stdin` or the interactive prompt over `--password`. A value passed to `--password` is visible in the process table and your shell history.
#### Create flags
Name for this registry authentication.
Registry username.
Registry password or access token. Cannot be used with `--password-stdin`. If neither `--password` nor `--password-stdin` is given, the command prompts for the password when run in an interactive terminal, or exits with an error otherwise.
Read the registry password from standard input (pipe, heredoc, or file redirection). Supports multi-line values. Cannot be used with `--password`.
### Delete a registry authentication
Delete a registry authentication:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
runpodctl registry delete
```
## Related commands
* [`runpodctl template create`](/runpodctl/reference/runpodctl-template)
* [`runpodctl pod create`](/runpodctl/reference/runpodctl-pod)
# send
Source: https://docs.runpod.io/runpodctl/reference/runpodctl-send
Send files or folders from your machine to a Pod or another computer through runpodctl using a secure peer-to-peer connection.
Transfer files or folders from your local machine to a Pod or another computer using a secure peer-to-peer connection.
```bash Command theme={"theme":{"light":"github-light","dark":"github-dark"}}
runpodctl send [flags]
```
## Example
Send a folder to a Pod using a connection code:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
runpodctl send ./my-dataset --code rainbow-unicorn-42
```
## Arguments
The path to the file or folder you want to send. Can be a single file or an entire directory.
## Flags
A custom code phrase used to establish the secure connection between sender and receiver. The receiver must use the same code with the [`receive`](/runpodctl/reference/runpodctl-receive) command.
## Related commands
* [`runpodctl receive`](/runpodctl/reference/runpodctl-receive)
# serverless
Source: https://docs.runpod.io/runpodctl/reference/runpodctl-serverless
Use runpodctl to create, list, inspect, update, and delete Serverless endpoints, including Hub-based deployments and templates.
Manage Serverless endpoints, including creating, listing, updating, and deleting endpoints.
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
runpodctl serverless [flags]
```
## Alias
You can use `sls` as a shorthand for `serverless`:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
runpodctl sls list
```
## Subcommands
### List endpoints
List all your Serverless endpoints:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
runpodctl serverless list
```
#### List flags
Include template information in the output.
Include workers information in the output.
### Get endpoint details
Get detailed information about a specific endpoint:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
runpodctl serverless get
```
#### Get flags
Include template information in the output.
Include workers information in the output.
### Create an endpoint
Create a new Serverless endpoint from a template or from a Hub repo:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
# Create from a template
runpodctl serverless create --template-id "tpl_abc123" --gpu-id "NVIDIA GeForce RTX 4090"
# Create from a template with a model reference
runpodctl serverless create --template-id "tpl_abc123" --gpu-id "NVIDIA GeForce RTX 4090" \
--model-reference https://huggingface.co/Qwen/Qwen2.5-0.5B-Instruct:main
# Create a CPU endpoint
runpodctl serverless create --template-id "tpl_abc123" --compute-type CPU
# Create from a Hub repo
runpodctl hub search vllm # Find the hub ID
runpodctl serverless create --hub-id cm8h09d9n000008jvh2rqdsmb --name "my-vllm"
# Create from a Hub repo and attach a model reference
runpodctl serverless create --hub-id cm8h09d9n000008jvh2rqdsmb --gpu-id "NVIDIA GeForce RTX 4090" \
--model-reference https://huggingface.co/Qwen/Qwen2.5-0.5B-Instruct:main
# Create from a Hub repo with custom environment variables
runpodctl serverless create --hub-id cm8h09d9n000008jvh2rqdsmb --name "my-vllm" \
--env MODEL_NAME=meta-llama/Llama-3.1-8B-Instruct \
--env MAX_TOKENS=4096
```
When using `--hub-id`, GPU IDs and container disk size are automatically pulled from the Hub release config. You can override the GPU type with `--gpu-id`. Environment variables from the Hub release are included automatically, and you can override or add to them with `--env`.
**Serverless templates vs Pod templates**: Serverless endpoints require a Serverless-specific template. Pod templates (like `runpod-torch-v21`) cannot be used because they include configuration, which Serverless does not support. When creating a template with [`runpodctl template create`](/runpodctl/reference/runpodctl-template), use the `--serverless` flag to create a Serverless template.
Each Serverless template can only be bound to one endpoint at a time. To create multiple endpoints with the same configuration, create separate templates for each.
#### Create flags
Name for the endpoint. Must be at least 3 characters. If omitted, a name is auto-generated in the format `endpoint-XXXXXXXX`.
Template ID to use (required if `--hub-id` is not specified). Use [`runpodctl template search`](/runpodctl/reference/runpodctl-template) to find templates.
Hub listing ID to deploy from (alternative to `--template-id`). Use [`runpodctl hub search`](/runpodctl/reference/runpodctl-hub) to find repos.
GPU type for workers. Accepts either a GPU type ID (e.g., `NVIDIA A40`, `NVIDIA GeForce RTX 4090`) or a GPU pool ID (e.g., `ADA_24`, `AMPERE_48`). Use [`runpodctl gpu list`](/runpodctl/reference/runpodctl-gpu) to see available GPUs.
Number of GPUs per worker.
Compute type (`GPU` or `CPU`). For CPU endpoints, use `--instance-id` to specify the CPU instance type.
CPU instance ID when using `--compute-type CPU`. If omitted, defaults to `cpu3g-4-16`. Only valid with `--compute-type CPU`.
Minimum number of workers.
Maximum number of workers.
Comma-separated list of preferred datacenter IDs. Use [`runpodctl datacenter list`](/runpodctl/reference/runpodctl-datacenter) to see available datacenters.
Network volume ID to attach for single-region deployments. Use [`runpodctl network-volume list`](/runpodctl/reference/runpodctl-network-volume) to see available network volumes. Mutually exclusive with `--network-volume-ids`.
Comma-separated list of network volume IDs for multi-region deployments. Mutually exclusive with `--network-volume-id`.
Minimum CUDA version required for workers (e.g., `12.4`). Workers will only be scheduled on machines that meet this CUDA version requirement.
Autoscaling strategy: `delay` (scales based on queue wait time in seconds) or `requests` (scales based on pending request count).
Trigger point for the autoscaler. For `delay`, this is the target queue wait time in seconds. For `requests`, this is the pending request count that triggers scaling.
Idle timeout in seconds. Workers shut down after being idle for this duration. Valid range: 1-3600 seconds.
Enable or disable flash boot for faster worker startup. When enabled, workers start from cached container images.
Execution timeout in seconds. Jobs that exceed this duration are terminated. The CLI accepts seconds but converts to milliseconds internally.
Environment variable in `KEY=VALUE` format. Use multiple `--env` flags to set multiple variables. These values only apply when deploying from `--hub-id`, where they override the Hub release defaults. With `--template-id`, environment variables come from the template, so `--env` is ignored and the CLI prints a note to that effect.
Model reference URL to attach to the endpoint. Use multiple `--model-reference` flags to attach multiple models. Works with both `--template-id` and `--hub-id`, and requires GPU compute type.
### Update an endpoint
Update endpoint configuration:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
runpodctl serverless update --workers-max 5
```
#### Update flags
New name for the endpoint.
New template ID to swap to. Use this to change the template attached to an existing endpoint without recreating it.
New minimum number of workers.
New maximum number of workers.
New idle timeout in seconds.
Scaler type (`QUEUE_DELAY` or `REQUEST_COUNT`).
Scaler value.
Enable or disable flash boot for faster worker startup.
Execution timeout in seconds. Jobs that exceed this duration are terminated.
### Delete an endpoint
Delete an endpoint:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
runpodctl serverless delete
```
## Serverless URLs
Access your Serverless endpoint using these URL patterns:
| Operation | URL |
| ------------- | -------------------------------------------------------- |
| Async request | `https://api.runpod.ai/v2//run` |
| Sync request | `https://api.runpod.ai/v2//runsync` |
| Health check | `https://api.runpod.ai/v2//health` |
| Job status | `https://api.runpod.ai/v2//status/` |
## Related commands
* [`runpodctl hub`](/runpodctl/reference/runpodctl-hub)
* [`runpodctl template`](/runpodctl/reference/runpodctl-template)
* [`runpodctl gpu list`](/runpodctl/reference/runpodctl-gpu)
# ssh
Source: https://docs.runpod.io/runpodctl/reference/runpodctl-ssh
Use runpodctl to manage SSH keys and retrieve the connection command and key details required to access a Runpod Pod.
Manage SSH keys and get SSH connection information for Pods.
```bash Command theme={"theme":{"light":"github-light","dark":"github-dark"}}
runpodctl ssh [flags]
```
## Subcommands
### Get SSH connection info
Get SSH connection details for a Pod. This returns the SSH command and key information, but does not initiate an interactive session:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
runpodctl ssh info
```
#### Info flags
Include Pod ID and name in output. Shorthand: `-v`.
The `ssh info` command returns connection details that you can use to connect via SSH manually. It does not start an interactive SSH session.
To connect to your Pod, use the SSH command provided in the output:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
ssh user@host -p -i
```
### List SSH keys
List all SSH keys associated with your account:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
runpodctl ssh list-keys
```
### Add an SSH key
Add a new SSH key to your account:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
# Add a key from a file
runpodctl ssh add-key --key-file ~/.ssh/id_ed25519.pub
# Add a key directly
runpodctl ssh add-key --key "ssh-ed25519 AAAA..."
```
#### Add-key flags
The public key string to add.
Path to a file containing the public key.
### Remove an SSH key
Remove an SSH key from your account by name or fingerprint:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
# Remove a key by name
runpodctl ssh remove-key --name my-laptop
# Remove a key by fingerprint
runpodctl ssh remove-key --fingerprint SHA256:abcd1234...
```
You must provide either `--name` or `--fingerprint`. If multiple keys share the same name, use `--fingerprint` instead.
#### Remove-key flags
Name of the key to remove.
Fingerprint of the key to remove. Use this when multiple keys have the same name.
## Related commands
* [`runpodctl pod get`](/runpodctl/reference/runpodctl-pod)
* [`runpodctl doctor`](/runpodctl/reference/runpodctl-doctor)
# template
Source: https://docs.runpod.io/runpodctl/reference/runpodctl-template
Use runpodctl to list, search, inspect, create, update, and delete reusable templates for Pods and Serverless endpoints.
List, search, and manage templates for Pods and Serverless endpoints.
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
runpodctl template [flags]
```
## Alias
You can use `tpl` as a shorthand for `template`:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
runpodctl tpl list
```
## Subcommands
### List templates
List available templates:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
# List official and community templates (first 10)
runpodctl template list
# List only official templates
runpodctl template list --type official
# List community templates
runpodctl template list --type community
# List your own templates
runpodctl template list --type user
# List all templates including user templates
runpodctl template list --all
# Show more results
runpodctl template list --limit 50
```
#### List flags
Filter by template type (`official`, `community`, `user`).
Maximum number of results to return.
Number of results to skip for pagination.
Include all templates including user templates.
### Search templates
Search for templates by name:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
# Search for PyTorch templates
runpodctl template search pytorch
# Search with limit
runpodctl template search comfyui --limit 5
# Search only official templates
runpodctl template search vllm --type official
```
#### Search flags
Filter by template type (`official`, `community`, `user`).
Maximum number of results to return.
Number of results to skip for pagination.
### Get template details
Get detailed information about a specific template, including README, environment variables, and exposed ports (with their labels):
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
runpodctl template get
```
### Create a template
Create a new template:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
# Create a Pod template
runpodctl template create --name "my-template" --image "runpod/pytorch:2.8.0-py3.11-cuda12.8.1-cudnn-devel-ubuntu22.04"
# Create a Serverless template
runpodctl template create --name "my-serverless-template" --image "my-image:latest" --serverless
# Create a template that pulls from a private registry
runpodctl template create --name "private-gpu" --image "registry.example.com/team/image:tag" --registry-auth-id
# Create a template with labeled ports
runpodctl template create --name "dev" --image "my-image:latest" --ports "22/tcp,8888/http" --port-labels "22=ssh,8888=jupyter lab"
```
**Pod templates vs Serverless templates**: Templates are either for Pods or for Serverless endpoints, not both. Pod templates support storage and are used with `runpodctl pod create`. Serverless templates do not support volume disks and are used with `runpodctl serverless create`. Use the `--serverless` flag when creating templates for Serverless endpoints.
#### Create flags
Template name.
Docker image (e.g., `runpod/pytorch:2.8.0-py3.11-cuda12.8.1-cudnn-devel-ubuntu22.04`).
Container disk size in GB.
Persistent volume size in GB.
Volume mount path.
Comma-separated list of ports to expose (e.g., `8888/http,22/tcp`).
Display names for the ports the template exposes, shown next to each port in the Runpod console. Provide comma-separated `port=name` pairs (e.g., `22=ssh,8888=jupyter lab`) or a JSON object (e.g., `{"22":"ssh","8888":"jupyter lab"}`). This flag requires `--ports`, and every labeled port must also appear in `--ports`. The pair form splits on commas, so use the JSON form for any label name that contains a comma.
ID of the container registry authentication used to pull the image from a private registry. Find the ID with [`runpodctl registry list`](/runpodctl/reference/runpodctl-registry).
Environment variables as a JSON object (e.g., `'{"KEY":"value"}'`).
Comma-separated Docker start commands.
Comma-separated Docker entrypoint commands.
Create as a Serverless template. Serverless templates do not support volume disks (`--volume-in-gb` is not supported). Each Serverless template can only be bound to one endpoint at a time.
Template README content.
### Update a template
Update an existing template:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
# Rename a template
runpodctl template update --name "new-name"
# Attach registry credentials
runpodctl template update --registry-auth-id
# Update port labels (or pass "" to clear them)
runpodctl template update --port-labels "22=ssh,8888=jupyter lab"
```
#### Update flags
New template name.
New Docker image name.
New comma-separated list of ports.
New display names for the template's exposed ports, as `port=name` pairs or a JSON object. Pass an empty value (`--port-labels ""`) to remove all port labels.
New container registry authentication ID for pulling private images. Pass an empty value (`--registry-auth-id ""`) to detach registry credentials from the template.
New environment variables as a JSON object.
New README content.
New container disk size in GB.
### Delete a template
Delete a template:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
runpodctl template delete
```
## Related commands
* [`runpodctl pod create`](/runpodctl/reference/runpodctl-pod)
* [`runpodctl serverless create`](/runpodctl/reference/runpodctl-serverless)
# update
Source: https://docs.runpod.io/runpodctl/reference/runpodctl-update
Update runpodctl to the latest release, verify the downloaded binary with its SHA-256 checksum, and confirm the installed version.
Update `runpodctl` to the latest version to access new features and bug fixes.
```bash Command theme={"theme":{"light":"github-light","dark":"github-dark"}}
runpodctl update
```
## Description
The `update` command downloads the latest version of `runpodctl` and verifies it against its SHA-256 checksum before replacing the installed binary. Verification always runs. If it fails, the update aborts and leaves your existing binary in place. After updating, verify the installation with [`runpodctl version`](/runpodctl/reference/runpodctl-version).
## Related commands
* [`runpodctl version`](/runpodctl/reference/runpodctl-version)
# user
Source: https://docs.runpod.io/runpodctl/reference/runpodctl-user
Use runpodctl to view your Runpod account email, credit balance, hourly spend, spending limit, and notification settings.
View your account information and current balance.
```bash Command theme={"theme":{"light":"github-light","dark":"github-dark"}}
runpodctl user
```
## Alias
You can use `me` as a shorthand for `user`:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
runpodctl me
```
## Description
The `user` command displays information about your Runpod account, including:
* Account email
* Current credit balance
* Current spend per hour
* Spend limit
* Notification settings
## Example output
```json theme={"theme":{"light":"github-light","dark":"github-dark"}}
{
"clientBalance": 435.85,
"currentSpendPerHr": 0.001,
"email": "user@example.com",
"id": "user_abc123",
"notifyLowBalance": true,
"notifyPodsGeneral": true,
"notifyPodsStale": true,
"spendLimit": 180
}
```
## Related commands
* [`runpodctl billing`](/runpodctl/reference/runpodctl-billing)
* [`runpodctl config`](/runpodctl/reference/runpodctl-config)
# version
Source: https://docs.runpod.io/runpodctl/reference/runpodctl-version
Use runpodctl to display the currently installed CLI version and confirm that an installation or update completed successfully.
Display the current version of `runpodctl` installed on your system.
```bash Command theme={"theme":{"light":"github-light","dark":"github-dark"}}
runpodctl version
```
## Example output
```
runpodctl 2.1.6-400ac40
```
## Related commands
* [`runpodctl update`](/runpodctl/reference/runpodctl-update)
# Overview
Source: https://docs.runpod.io/sdks/graphql/configurations
Use the GraphQL API to manage Pods, templates, and Serverless endpoints programmatically. Review operations and request patterns for this Runpod SDK.
Use the GraphQL API to manage Pods, templates, and Serverless endpoints through queries and mutations.
All requests go to `https://api.runpod.io/graphql` with your API key included as a query parameter. For the complete schema including all available queries, mutations, fields, and inputs, see the [GraphQL Spec](https://graphql-spec.runpod.io/).
The GraphQL API is deprecated and will be retired in early 2027. For new integrations, use [REST API v2](/api-reference-v2/overview). To migrate automatically, ask your coding agent to run `/runpod:migrate graphql` once you've installed the [Runpod skills plugin](/get-started/agent-skills#migrate-an-existing-integration).
## Quick start
Make a request using cURL:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
curl --request POST \
--header 'content-type: application/json' \
--url 'https://api.runpod.io/graphql?api_key=YOUR_API_KEY' \
--data '{"query": "query { myself { id email } }"}'
```
Or use a GraphQL client:
```graphql theme={"theme":{"light":"github-light","dark":"github-dark"}}
query {
myself {
id
email
}
}
```
## Common operations
Create, start, stop, and query Pods.
Create and manage Pod and Serverless templates.
Create and configure Serverless endpoints.
## Common input fields
The following fields are commonly used when creating Pods and templates.
| Field | Type | Description |
| ------------------------- | ------- | -------------------------------------------------------------------------------------------------------------------------------------------- |
| `containerDiskInGb` | Integer | Size of the container disk in GB. Used for the operating system, installed applications, and temporary data. |
| `volumeInGb` | Integer | Size of the persistent volume in GB. Data persists between Pod restarts. |
| `imageName` | String | Docker image name including repository and tag (e.g., `nginx:latest`). |
| `name` | String | Name for the resource. Must be unique within your account. |
| `dockerArgs` | String | Overrides the container's default start command. If not specified, uses the image's CMD. |
| `env` | Array | Environment variables as key-value pairs (e.g., `[{key: "VAR", value: "value"}]`). |
| `ports` | String | Ports to expose, formatted as `port/protocol` (e.g., `8888/http,22/tcp`). |
| `gpuTypeId` | String | GPU type identifier (e.g., `NVIDIA RTX A6000`). Use the `gpuTypes` query to list available options. |
| `gpuCount` | Integer | Number of GPUs to allocate. |
| `containerRegistryAuthId` | String | ID of saved registry credentials for private container images. Find this in your [Runpod settings](https://console.runpod.io/user/settings). |
# Manage endpoints
Source: https://docs.runpod.io/sdks/graphql/manage-endpoints
Create, modify, and delete Serverless endpoints using the GraphQL API. Review operations and request patterns for this Runpod SDK.
Create, modify, and delete Serverless endpoints using the GraphQL API.
For the complete schema, see the [GraphQL Spec](https://graphql-spec.runpod.io/).
## Quick reference
| Operation | Mutation/Query |
| --------------- | ------------------------------ |
| Create endpoint | `saveEndpoint` |
| Modify endpoint | `saveEndpoint` (with `id`) |
| List endpoints | `myself { endpoints { ... } }` |
| Delete endpoint | `deleteEndpoint` |
## Required fields
Endpoints require the following fields:
| Field | Type | Description |
| ------------ | ------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `gpuIds` | String | GPU tier identifier. Options: `AMPERE_16` (16GB), `AMPERE_24` (24GB), `ADA_24` (24GB Ada), `AMPERE_48` (48GB), `ADA_48_PRO` (48GB Ada Pro), `AMPERE_80` (80GB), `ADA_80_PRO` (80GB Ada Pro). |
| `name` | String | Endpoint name. |
| `templateId` | String | ID of the Serverless template to use. |
| `type` | String | Endpoint type. `QB` for queue-based (default), `LB` for [load balancing](/serverless/load-balancing/overview). |
## Create an endpoint
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
curl --request POST \
--header 'content-type: application/json' \
--url 'https://api.runpod.io/graphql?api_key=${YOUR_API_KEY}' \
--data '{"query": "mutation { saveEndpoint(input: { gpuIds: \"AMPERE_16\", idleTimeout: 5, locations: \"US\", name: \"My Endpoint\", flashBootType: FLASHBOOT, scalerType: \"QUEUE_DELAY\", scalerValue: 4, templateId: \"YOUR_TEMPLATE_ID\", workersMax: 3, workersMin: 0 }) { id name gpuIds idleTimeout locations flashBootType scalerType scalerValue templateId workersMax workersMin } }"}'
```
```graphql theme={"theme":{"light":"github-light","dark":"github-dark"}}
mutation {
saveEndpoint(input: {
gpuIds: "AMPERE_16",
idleTimeout: 5,
# Leave locations empty or null for any region
# Options: CZ, FR, GB, NO, RO, US
locations: "US",
name: "My Endpoint",
# Set to FLASHBOOT for faster cold starts (enum value, no quotes)
flashBootType: FLASHBOOT,
scalerType: "QUEUE_DELAY",
scalerValue: 4,
templateId: "YOUR_TEMPLATE_ID",
workersMax: 3,
workersMin: 0
# Optional: attach a network volume
# networkVolumeId: "YOUR_VOLUME_ID"
}) {
id
name
gpuIds
idleTimeout
locations
flashBootType
scalerType
scalerValue
templateId
workersMax
workersMin
}
}
```
```json theme={"theme":{"light":"github-light","dark":"github-dark"}}
{
"data": {
"saveEndpoint": {
"id": "i02xupws21hp6i",
"name": "My Endpoint",
"gpuIds": "AMPERE_16",
"idleTimeout": 5,
"locations": "US",
"flashBootType": "FLASHBOOT",
"scalerType": "QUEUE_DELAY",
"scalerValue": 4,
"templateId": "YOUR_TEMPLATE_ID",
"workersMax": 3,
"workersMin": 0
}
}
}
```
### Configuration options
| Field | Description |
| ----------------- | --------------------------------------------------------------------------------------------------- |
| `idleTimeout` | Seconds before idle workers shut down. |
| `locations` | Restrict to specific regions. Leave empty for any region. |
| `flashBootType` | Enum value for boot optimization. Set to `FLASHBOOT` for faster cold starts (no quotes in GraphQL). |
| `scalerType` | Autoscaling strategy. Options: `QUEUE_DELAY`, `REQUEST_COUNT`. |
| `scalerValue` | Target value for the scaler (e.g., queue delay in seconds). |
| `workersMin` | Minimum active workers. Set to `0` for scale-to-zero. |
| `workersMax` | Maximum concurrent workers. |
| `networkVolumeId` | Optional network volume to mount. |
## Modify an endpoint
Include the endpoint `id` to update an existing endpoint.
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
curl --request POST \
--header 'content-type: application/json' \
--url 'https://api.runpod.io/graphql?api_key=${YOUR_API_KEY}' \
--data '{"query": "mutation { saveEndpoint(input: { id: \"i02xupws21hp6i\", gpuIds: \"AMPERE_16\", name: \"My Endpoint\", templateId: \"YOUR_TEMPLATE_ID\", workersMax: 5 }) { id gpuIds name templateId workersMax } }"}'
```
```graphql theme={"theme":{"light":"github-light","dark":"github-dark"}}
mutation {
saveEndpoint(input: {
id: "i02xupws21hp6i",
gpuIds: "AMPERE_16",
name: "My Endpoint",
templateId: "YOUR_TEMPLATE_ID",
workersMax: 5
}) {
id
gpuIds
name
templateId
workersMax
}
}
```
```json theme={"theme":{"light":"github-light","dark":"github-dark"}}
{
"data": {
"saveEndpoint": {
"id": "i02xupws21hp6i",
"gpuIds": "AMPERE_16",
"name": "My Endpoint",
"templateId": "YOUR_TEMPLATE_ID",
"workersMax": 5
}
}
}
```
## List endpoints
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
curl --request POST \
--header 'content-type: application/json' \
--url 'https://api.runpod.io/graphql?api_key=${YOUR_API_KEY}' \
--data '{"query": "query { myself { endpoints { id name gpuIds idleTimeout locations networkVolumeId scalerType scalerValue templateId workersMax workersMin pods { desiredStatus } } serverlessDiscount { discountFactor type expirationDate } } }"}'
```
```graphql theme={"theme":{"light":"github-light","dark":"github-dark"}}
query {
myself {
endpoints {
id
name
gpuIds
idleTimeout
locations
networkVolumeId
scalerType
scalerValue
templateId
workersMax
workersMin
pods {
desiredStatus
}
}
serverlessDiscount {
discountFactor
type
expirationDate
}
}
}
```
```json theme={"theme":{"light":"github-light","dark":"github-dark"}}
{
"data": {
"myself": {
"endpoints": [
{
"id": "i02xupws21hp6i",
"name": "My Endpoint",
"gpuIds": "AMPERE_16",
"idleTimeout": 5,
"locations": "US",
"networkVolumeId": null,
"scalerType": "QUEUE_DELAY",
"scalerValue": 4,
"templateId": "YOUR_TEMPLATE_ID",
"workersMax": 5,
"workersMin": 0,
"pods": []
}
],
"serverlessDiscount": null
}
}
}
```
## Delete an endpoint
Before deleting, set both `workersMin` and `workersMax` to `0`.
The endpoint's min and max workers must both be zero before you can delete it.
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
curl --request POST \
--header 'content-type: application/json' \
--url 'https://api.runpod.io/graphql?api_key=${YOUR_API_KEY}' \
--data '{"query": "mutation { deleteEndpoint(id: \"i02xupws21hp6i\") }"}'
```
```graphql theme={"theme":{"light":"github-light","dark":"github-dark"}}
mutation {
deleteEndpoint(id: "i02xupws21hp6i")
}
```
```json theme={"theme":{"light":"github-light","dark":"github-dark"}}
{
"data": {
"deleteEndpoint": null
}
}
```
# Manage templates
Source: https://docs.runpod.io/sdks/graphql/manage-pod-templates
Create and manage templates for Pods and Serverless endpoints using the GraphQL API. Review operations and request patterns for this Runpod SDK.
Use the GraphQL API to create and manage templates for Pods and Serverless endpoints.
For the complete schema, see the [GraphQL Spec](https://graphql-spec.runpod.io/).
## Quick reference
| Operation | Mutation |
| --------------- | -------------------------- |
| Create template | `saveTemplate` |
| Modify template | `saveTemplate` (with `id`) |
| Delete template | `deleteTemplate` |
| Create secret | `secretCreate` |
| Delete secret | `secretDelete` |
## Required fields
Templates require the following fields:
| Field | Type | Description |
| ------------------- | ------- | ------------------------------------------------------- |
| `containerDiskInGb` | Integer | Container disk size in GB. |
| `imageName` | String | Docker image name with tag (e.g., `ubuntu:latest`). |
| `name` | String | Unique template name. |
| `volumeInGb` | Integer | Volume size in GB. Set to `0` for Serverless templates. |
Template names must be unique. Creating a template with a name that already exists will fail.
## Create a template
### GPU Pod template
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
curl --request POST \
--header 'content-type: application/json' \
--url 'https://api.runpod.io/graphql?api_key=${YOUR_API_KEY}' \
--data '{"query": "mutation { saveTemplate(input: { containerDiskInGb: 5, dockerArgs: \"sleep infinity\", env: [ { key: \"key1\", value: \"value1\" }, { key: \"key2\", value: \"value2\" } ], imageName: \"ubuntu:latest\", name: \"My GPU Template\", ports: \"8888/http,22/tcp\", readme: \"## Hello, World!\", volumeInGb: 15, volumeMountPath: \"/workspace\" }) { containerDiskInGb dockerArgs env { key value } id imageName name ports readme volumeInGb volumeMountPath } }"}'
```
```graphql theme={"theme":{"light":"github-light","dark":"github-dark"}}
mutation {
saveTemplate(input: {
containerDiskInGb: 5,
dockerArgs: "sleep infinity",
env: [
{ key: "key1", value: "value1" },
{ key: "key2", value: "value2" }
],
imageName: "ubuntu:latest",
name: "My GPU Template",
ports: "8888/http,22/tcp",
readme: "## Hello, World!",
volumeInGb: 15,
volumeMountPath: "/workspace"
}) {
id
name
imageName
containerDiskInGb
volumeInGb
dockerArgs
ports
env { key value }
}
}
```
```json theme={"theme":{"light":"github-light","dark":"github-dark"}}
{
"data": {
"saveTemplate": {
"id": "wphkv67a0p",
"name": "My GPU Template",
"imageName": "ubuntu:latest",
"containerDiskInGb": 5,
"volumeInGb": 15,
"dockerArgs": "sleep infinity",
"ports": "8888/http,22/tcp",
"env": [
{ "key": "key1", "value": "value1" },
{ "key": "key2", "value": "value2" }
]
}
}
}
```
### Serverless template
For Serverless templates, set `volumeInGb` to `0` and include `isServerless: true`.
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
curl --request POST \
--header 'content-type: application/json' \
--url 'https://api.runpod.io/graphql?api_key=${YOUR_API_KEY}' \
--data '{"query": "mutation { saveTemplate(input: { containerDiskInGb: 5, dockerArgs: \"python handler.py\", env: [ { key: \"key1\", value: \"value1\" } ], imageName: \"runpod/serverless-hello-world:latest\", isServerless: true, name: \"My Serverless Template\", volumeInGb: 0 }) { id name imageName isServerless containerDiskInGb } }"}'
```
```graphql theme={"theme":{"light":"github-light","dark":"github-dark"}}
mutation {
saveTemplate(input: {
containerDiskInGb: 5,
dockerArgs: "python handler.py",
env: [
{ key: "key1", value: "value1" }
],
imageName: "runpod/serverless-hello-world:latest",
isServerless: true,
name: "My Serverless Template",
volumeInGb: 0
}) {
id
name
imageName
isServerless
containerDiskInGb
dockerArgs
env { key value }
}
}
```
```json theme={"theme":{"light":"github-light","dark":"github-dark"}}
{
"data": {
"saveTemplate": {
"id": "xkhgg72fuo",
"name": "My Serverless Template",
"imageName": "runpod/serverless-hello-world:latest",
"isServerless": true,
"containerDiskInGb": 5,
"dockerArgs": "python handler.py",
"env": [
{ "key": "key1", "value": "value1" }
]
}
}
}
```
### Private container images
For private container images, use `containerRegistryAuthId` with the ID of your saved registry credentials from your [Runpod settings](https://www.console.runpod.io/user/settings).
```graphql theme={"theme":{"light":"github-light","dark":"github-dark"}}
mutation {
saveTemplate(input: {
containerDiskInGb: 5,
imageName: "myregistry.io/private-image:latest",
containerRegistryAuthId: "YOUR_REGISTRY_CREDENTIALS_ID",
name: "Private Image Template",
volumeInGb: 10
}) {
id
name
}
}
```
## Modify a template
Include the template `id` to update an existing template. The same mutation works for both GPU and Serverless templates.
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
curl --request POST \
--header 'content-type: application/json' \
--url 'https://api.runpod.io/graphql?api_key=${YOUR_API_KEY}' \
--data '{"query": "mutation { saveTemplate(input: { id: \"wphkv67a0p\", containerDiskInGb: 10, imageName: \"ubuntu:latest\", name: \"My GPU Template\", volumeInGb: 20, readme: \"## Updated readme\" }) { id containerDiskInGb volumeInGb readme } }"}'
```
```graphql theme={"theme":{"light":"github-light","dark":"github-dark"}}
mutation {
saveTemplate(input: {
id: "wphkv67a0p",
containerDiskInGb: 10,
imageName: "ubuntu:latest",
name: "My GPU Template",
volumeInGb: 20,
readme: "## Updated readme"
}) {
id
containerDiskInGb
volumeInGb
readme
}
}
```
```json theme={"theme":{"light":"github-light","dark":"github-dark"}}
{
"data": {
"saveTemplate": {
"id": "wphkv67a0p",
"containerDiskInGb": 10,
"volumeInGb": 20,
"readme": "## Updated readme"
}
}
}
```
## Delete a template
Delete a template by name. The template must not be in use by any Pods or Serverless endpoints.
It can take up to 2 minutes to delete a template after its most recent use by a Pod or Serverless endpoint.
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
curl --request POST \
--header 'content-type: application/json' \
--url 'https://api.runpod.io/graphql?api_key=${YOUR_API_KEY}' \
--data '{"query": "mutation { deleteTemplate(templateName: \"My GPU Template\") }"}'
```
```graphql theme={"theme":{"light":"github-light","dark":"github-dark"}}
mutation {
deleteTemplate(templateName: "My GPU Template")
}
```
```json theme={"theme":{"light":"github-light","dark":"github-dark"}}
{
"data": {
"deleteTemplate": null
}
}
```
## Create a secret
Secrets store sensitive values that can be referenced in templates.
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
curl --request POST \
--header 'content-type: application/json' \
--url 'https://api.runpod.io/graphql?api_key=${YOUR_API_KEY}' \
--data '{"query": "mutation { secretCreate(input: { value: \"my-secret-value\", name: \"my-secret\" }) { id name description } }"}'
```
```graphql theme={"theme":{"light":"github-light","dark":"github-dark"}}
mutation {
secretCreate(input: {
value: "my-secret-value",
name: "my-secret"
}) {
id
name
description
}
}
```
```json theme={"theme":{"light":"github-light","dark":"github-dark"}}
{
"data": {
"secretCreate": {
"id": "abc123",
"name": "my-secret",
"description": null
}
}
}
```
## Delete a secret
Delete a secret by its ID. You can find the secret ID from the `secretCreate` response or by querying your secrets.
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
curl --request POST \
--header 'content-type: application/json' \
--url 'https://api.runpod.io/graphql?api_key=${YOUR_API_KEY}' \
--data '{"query": "mutation { secretDelete(id: \"abc123\") }"}'
```
```graphql theme={"theme":{"light":"github-light","dark":"github-dark"}}
mutation {
secretDelete(id: "abc123")
}
```
```json theme={"theme":{"light":"github-light","dark":"github-dark"}}
{
"data": {
"secretDelete": null
}
}
```
# Manage Pods
Source: https://docs.runpod.io/sdks/graphql/manage-pods
Create, start, stop, and query Pods using the GraphQL API. Review authentication, operations, request patterns, and examples for this Runpod SDK.
Create, start, stop, and query Pods using the GraphQL API with cURL and GraphQL examples.
For the complete schema, see the [GraphQL Spec](https://graphql-spec.runpod.io/).
## Quick reference
| Operation | Mutation/Query |
| -------------- | ---------------------------- |
| Create Pod | `podFindAndDeployOnDemand` |
| Start Pod | `podResume` |
| Stop Pod | `podStop` |
| List all Pods | `myself { pods { ... } }` |
| Get Pod by ID | `pod(input: {podId: "..."})` |
| List GPU types | `gpuTypes` |
## Create a Pod
Pods provide guaranteed compute at a fixed price.
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
curl --request POST \
--header 'content-type: application/json' \
--url 'https://api.runpod.io/graphql?api_key=${YOUR_API_KEY}' \
--data '{"query": "mutation { podFindAndDeployOnDemand( input: { cloudType: ALL, gpuCount: 1, volumeInGb: 40, containerDiskInGb: 40, minVcpuCount: 2, minMemoryInGb: 15, gpuTypeId: \"NVIDIA RTX A6000\", name: \"Runpod Tensorflow\", imageName: \"runpod/tensorflow\", dockerArgs: \"\", ports: \"8888/http\", volumeMountPath: \"/workspace\", env: [{ key: \"JUPYTER_PASSWORD\", value: \"your-password\" }] } ) { id imageName env machineId machine { podHostId } } }"}'
```
```graphql theme={"theme":{"light":"github-light","dark":"github-dark"}}
mutation {
podFindAndDeployOnDemand(
input: {
cloudType: ALL
gpuCount: 1
volumeInGb: 40
containerDiskInGb: 40
minVcpuCount: 2
minMemoryInGb: 15
gpuTypeId: "NVIDIA RTX A6000"
name: "Runpod Tensorflow"
imageName: "runpod/tensorflow"
dockerArgs: ""
ports: "8888/http"
volumeMountPath: "/workspace"
env: [{ key: "JUPYTER_PASSWORD", value: "your-password" }]
}
) {
id
imageName
env
machineId
machine {
podHostId
}
}
}
```
```json theme={"theme":{"light":"github-light","dark":"github-dark"}}
{
"data": {
"podFindAndDeployOnDemand": {
"id": "50qynxzilsxoey",
"imageName": "runpod/tensorflow",
"env": ["JUPYTER_PASSWORD=your-password"],
"machineId": "hpvdausak8xb",
"machine": {
"podHostId": "50qynxzilsxoey-64410065"
}
}
}
}
```
### Filter by CUDA version
Use `allowedCudaVersions` to restrict Pods to machines with specific CUDA versions.
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
curl --request POST \
--header 'content-type: application/json' \
--url 'https://api.runpod.io/graphql?api_key=${YOUR_API_KEY}' \
--data '{
"query": "mutation { podFindAndDeployOnDemand( input: { cloudType: ALL, gpuCount: 1, volumeInGb: 40, containerDiskInGb: 40, gpuTypeId: \"NVIDIA RTX A6000\", name: \"Runpod Pytorch\", imageName: \"runpod/pytorch\", allowedCudaVersions: [\"12.0\", \"12.1\", \"12.2\", \"12.3\"] } ) { id imageName machineId } }"
}'
```
```graphql theme={"theme":{"light":"github-light","dark":"github-dark"}}
mutation {
podFindAndDeployOnDemand(
input: {
cloudType: ALL
gpuCount: 1
volumeInGb: 40
containerDiskInGb: 40
gpuTypeId: "NVIDIA RTX A6000"
name: "Runpod Pytorch"
imageName: "runpod/pytorch"
allowedCudaVersions: ["12.0", "12.1", "12.2", "12.3"]
}
) {
id
imageName
machineId
}
}
```
## Start a Pod
Resume a stopped Pod using the `podResume` mutation.
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
curl --request POST \
--header 'content-type: application/json' \
--url 'https://api.runpod.io/graphql?api_key=${YOUR_API_KEY}' \
--data '{"query": "mutation { podResume( input: { podId: \"YOUR_POD_ID\", gpuCount: 1 } ) { id desiredStatus imageName } }"}'
```
```graphql theme={"theme":{"light":"github-light","dark":"github-dark"}}
mutation {
podResume(input: { podId: "YOUR_POD_ID", gpuCount: 1 }) {
id
desiredStatus
imageName
}
}
```
```json theme={"theme":{"light":"github-light","dark":"github-dark"}}
{
"data": {
"podResume": {
"id": "YOUR_POD_ID",
"desiredStatus": "RUNNING",
"imageName": "runpod/tensorflow"
}
}
}
```
You can also filter by CUDA version when starting a Pod:
```graphql theme={"theme":{"light":"github-light","dark":"github-dark"}}
mutation {
podResume(input: {
podId: "YOUR_POD_ID",
gpuCount: 1,
allowedCudaVersions: ["12.0", "12.1", "12.2", "12.3"]
}) {
id
desiredStatus
}
}
```
## Stop a Pod
Stopping a Pod releases the GPU while preserving your volume data.
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
curl --request POST \
--header 'content-type: application/json' \
--url 'https://api.runpod.io/graphql?api_key=${YOUR_API_KEY}' \
--data '{"query": "mutation { podStop(input: {podId: \"YOUR_POD_ID\"}) { id desiredStatus } }"}'
```
```graphql theme={"theme":{"light":"github-light","dark":"github-dark"}}
mutation {
podStop(input: { podId: "YOUR_POD_ID" }) {
id
desiredStatus
}
}
```
```json theme={"theme":{"light":"github-light","dark":"github-dark"}}
{
"data": {
"podStop": {
"id": "YOUR_POD_ID",
"desiredStatus": "EXITED"
}
}
}
```
## Query Pods
### List all Pods
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
curl --request POST \
--header 'content-type: application/json' \
--url 'https://api.runpod.io/graphql?api_key=${YOUR_API_KEY}' \
--data '{"query": "query { myself { pods { id name runtime { uptimeInSeconds gpus { id gpuUtilPercent memoryUtilPercent } container { cpuPercent memoryPercent } } } } }"}'
```
```graphql theme={"theme":{"light":"github-light","dark":"github-dark"}}
query {
myself {
pods {
id
name
runtime {
uptimeInSeconds
ports {
ip
isIpPublic
privatePort
publicPort
type
}
gpus {
id
gpuUtilPercent
memoryUtilPercent
}
container {
cpuPercent
memoryPercent
}
}
}
}
}
```
```json theme={"theme":{"light":"github-light","dark":"github-dark"}}
{
"data": {
"myself": {
"pods": [
{
"id": "ldl1dxirsim64n",
"name": "Runpod Pytorch",
"runtime": {
"uptimeInSeconds": 3931,
"ports": [
{
"ip": "100.65.0.101",
"isIpPublic": false,
"privatePort": 8888,
"publicPort": 60141,
"type": "http"
}
],
"gpus": [
{
"id": "GPU-e0488b7e-6932-795b-a125-4472c16ea72c",
"gpuUtilPercent": 0,
"memoryUtilPercent": 0
}
],
"container": {
"cpuPercent": 0,
"memoryPercent": 0
}
}
}
]
}
}
}
```
### Get Pod by ID
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
curl --request POST \
--header 'content-type: application/json' \
--url 'https://api.runpod.io/graphql?api_key=${YOUR_API_KEY}' \
--data '{"query": "query { pod(input: {podId: \"YOUR_POD_ID\"}) { id name runtime { uptimeInSeconds gpus { id gpuUtilPercent memoryUtilPercent } } } }"}'
```
```graphql theme={"theme":{"light":"github-light","dark":"github-dark"}}
query {
pod(input: { podId: "YOUR_POD_ID" }) {
id
name
runtime {
uptimeInSeconds
ports {
ip
isIpPublic
privatePort
publicPort
type
}
gpus {
id
gpuUtilPercent
memoryUtilPercent
}
container {
cpuPercent
memoryPercent
}
}
}
}
```
```json theme={"theme":{"light":"github-light","dark":"github-dark"}}
{
"data": {
"pod": {
"id": "YOUR_POD_ID",
"name": "Runpod Pytorch",
"runtime": {
"uptimeInSeconds": 11,
"ports": [
{
"ip": "100.65.0.101",
"isIpPublic": false,
"privatePort": 8888,
"publicPort": 60141,
"type": "http"
}
],
"gpus": [
{
"id": "GPU-e0488b7e-6932-795b-a125-4472c16ea72c",
"gpuUtilPercent": 0,
"memoryUtilPercent": 0
}
],
"container": {
"cpuPercent": 0,
"memoryPercent": 0
}
}
}
}
}
```
## Query GPU types
List available GPU types to find the `gpuTypeId` needed when creating Pods.
### List all GPU types
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
curl --request POST \
--header 'content-type: application/json' \
--url 'https://api.runpod.io/graphql?api_key=${YOUR_API_KEY}' \
--data '{"query": "query { gpuTypes { id displayName memoryInGb } }"}'
```
```graphql theme={"theme":{"light":"github-light","dark":"github-dark"}}
query {
gpuTypes {
id
displayName
memoryInGb
}
}
```
```json theme={"theme":{"light":"github-light","dark":"github-dark"}}
{
"data": {
"gpuTypes": [
{
"id": "NVIDIA GeForce RTX 3070",
"displayName": "RTX 3070",
"memoryInGb": 8
},
{
"id": "NVIDIA GeForce RTX 3080",
"displayName": "RTX 3080",
"memoryInGb": 10
},
{
"id": "NVIDIA RTX A6000",
"displayName": "RTX A6000",
"memoryInGb": 48
}
]
}
}
```
### Get GPU type details
Query a specific GPU type to see pricing and availability.
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
curl --request POST \
--header 'content-type: application/json' \
--url 'https://api.runpod.io/graphql?api_key=${YOUR_API_KEY}' \
--data '{"query": "query { gpuTypes(input: {id: \"NVIDIA GeForce RTX 3090\"}) { id displayName memoryInGb secureCloud communityCloud lowestPrice(input: {gpuCount: 1}) { uninterruptablePrice } } }"}'
```
```graphql theme={"theme":{"light":"github-light","dark":"github-dark"}}
query {
gpuTypes(input: { id: "NVIDIA GeForce RTX 3090" }) {
id
displayName
memoryInGb
secureCloud
communityCloud
lowestPrice(input: { gpuCount: 1 }) {
uninterruptablePrice
}
}
}
```
```json theme={"theme":{"light":"github-light","dark":"github-dark"}}
{
"data": {
"gpuTypes": [
{
"id": "NVIDIA GeForce RTX 3090",
"displayName": "RTX 3090",
"memoryInGb": 24,
"secureCloud": false,
"communityCloud": true,
"lowestPrice": {
"uninterruptablePrice": 0.3
}
}
]
}
}
```
### Check GPU availability
Use the `stockStatus` field to check availability before creating a Pod. Values include `High`, `Medium`, `Low`, and `None`.
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
curl --request POST \
--header 'content-type: application/json' \
--url 'https://api.runpod.io/graphql?api_key=${YOUR_API_KEY}' \
--data '{"query": "query { gpuTypes(input: { id: \"NVIDIA RTX A4000\" }) { id displayName lowestPrice(input: { gpuCount: 1, secureCloud: true }) { stockStatus uninterruptablePrice availableGpuCounts } } }"}'
```
```graphql theme={"theme":{"light":"github-light","dark":"github-dark"}}
query {
gpuTypes(input: { id: "NVIDIA RTX A4000" }) {
id
displayName
lowestPrice(input: { gpuCount: 1, secureCloud: true }) {
stockStatus
uninterruptablePrice
availableGpuCounts
}
}
}
```
```json theme={"theme":{"light":"github-light","dark":"github-dark"}}
{
"data": {
"gpuTypes": [
{
"id": "NVIDIA RTX A4000",
"displayName": "RTX A4000",
"lowestPrice": {
"stockStatus": "High",
"uninterruptablePrice": 0.35,
"availableGpuCounts": [1, 2, 4]
}
}
]
}
}
```
```json theme={"theme":{"light":"github-light","dark":"github-dark"}}
{
"data": {
"gpuTypes": [
{
"id": "NVIDIA RTX A4000",
"displayName": "RTX A4000",
"lowestPrice": {
"stockStatus": "Low",
"uninterruptablePrice": 0.24,
"availableGpuCounts": [1, 2, 3, 4, 5, 6, 7]
}
}
]
}
}
```
If `stockStatus` is `Low`, there are very few GPUs available. Consider selecting an alternative GPU type or trying again later.
# Batch jobs
Source: https://docs.runpod.io/serverless/batch-jobs
Submit large collections of Serverless inference requests as named batch jobs, then monitor asynchronous processing and retrieve results.
Use batch jobs to run large volumes of inference requests against a serverless endpoint without waiting for each result in real time. Batch jobs run asynchronously on dedicated workers that are separate from your endpoint's standard `/run` traffic, so submitting a batch never delays your interactive requests.
Batch Jobs is currently in beta. Join our [Discord](https://discord.gg/runpod) to provide feedback and get support.
## When to use batch vs /run
| | Batch | `/run` |
| --------------------- | ---------------------------------- | -------------------------------- |
| **Use case** | Bulk, offline workloads | Interactive, real-time inference |
| **Latency** | Multi-hour | Seconds to minutes |
| **Traffic isolation** | Dedicated batch workers | Standard serverless workers |
| **Result delivery** | Poll or subscribe to notifications | Synchronous or async poll |
Choose batch when your workload can tolerate multi-hour latency — for example, nightly dataset processing, pre-computing embeddings, or running evaluations.
## Batch lifecycle
A batch moves through the following states:
```
DRAFT → FINALIZED → FAILED
→ CANCELLED
```
* **DRAFT** — The batch is a draft. You can add, update, or remove individual requests. Batch workers have not started any work.
* **FINALIZED** — The batch is locked; no further requests can be added or removed. Batch workers process the requests while the batch stays in this state, and there is no separate `RUNNING` or `COMPLETED` batch status. Track progress through the `requestTotal`, `requestInProgress`, `requestCompleted`, and `requestFailed` counts — all requests have finished when `requestCompleted + requestFailed` equals `requestTotal`.
* **FAILED** — The batch itself failed before or during execution (distinct from individual request failures in a batch whose other requests finished successfully).
* **CANCELLED** — You cancelled the batch. See [Cancellation](#cancellation) for details.
You must call `/finalize` before the batch begins processing. A DRAFT batch will not be executed.
## API walkthrough
### 1. Create a batch
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
POST /v2/{endpoint_id}/batch
Authorization: Bearer {api_key}
Content-Type: application/json
```
The request body is a top-level JSON array. Send an empty array `[]` to create a batch and add requests later, or send a populated array to include an initial list of requests. Each element uses the same shape as a standard `/run` call — a JSON object with an `input` field.
```json theme={"theme":{"light":"github-light","dark":"github-dark"}}
[
{ "input": { "text": "The quick brown fox" } },
{ "input": { "text": "Jumped over the lazy dog" } }
]
```
**Response:**
```json theme={"theme":{"light":"github-light","dark":"github-dark"}}
{
"id": "batch_01j9abc123",
"status": "DRAFT"
}
```
### 2. Add more requests
While the batch is DRAFT, append additional requests:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
POST /v2/{endpoint_id}/batch/{batch_id}/requests
Authorization: Bearer {api_key}
Content-Type: application/json
```
```json theme={"theme":{"light":"github-light","dark":"github-dark"}}
{
"requests": [
{ "input": { "text": "More text to embed" } },
{ "input": { "text": "Another piece of text" } },
{ "input": { "text": "And another one" } }
]
}
```
Request body size is limited to 10 MiB per call. You can call this endpoint multiple times to build up large batches incrementally.
### 3. Finalize the batch
Once you've added all requests, finalize the batch to make it eligible for execution:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
POST /v2/{endpoint_id}/batch/{batch_id}/finalize
Authorization: Bearer {api_key}
```
After finalization, the batch status transitions to `FINALIZED` and requests are locked. You can no longer add or remove individual requests.
### 4. Poll batch status
Check overall progress by fetching the batch summary:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
GET /v2/{endpoint_id}/batch/{batch_id}
Authorization: Bearer {api_key}
```
**Response:**
```json theme={"theme":{"light":"github-light","dark":"github-dark"}}
{
"id": "batch_01j9abc123",
"endpointId": "abc123xyz",
"status": "FINALIZED",
"requestTotal": 1000,
"requestInProgress": 8,
"requestCompleted": 244,
"requestFailed": 6,
"createdAt": 1783584000000
}
```
Poll this endpoint at whatever interval suits your workflow. A batch that is still processing reports `status: FINALIZED`; there is no `RUNNING` or `COMPLETED` status. All requests have finished when `requestCompleted + requestFailed` equals `requestTotal`. The batch reaches a terminal state only when `status` is `FAILED` or `CANCELLED`. The `createdAt` field is a Unix epoch timestamp in milliseconds.
### 5. Retrieve results
Fetch paginated results for all child requests in the batch:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
GET /v2/{endpoint_id}/batch/{batch_id}/requests
Authorization: Bearer {api_key}
```
**Response:**
```json theme={"theme":{"light":"github-light","dark":"github-dark"}}
{
"requests": [
{
"id": "req_abc001",
"status": "COMPLETED",
"output": { "embedding": [0.12, 0.34, ...] },
"startedAt": "2026-07-09T09:15:00Z",
"completedAt": "2026-07-09T09:15:02Z"
},
{
"id": "req_abc002",
"status": "FAILED",
"error": "Handler raised an exception: timeout exceeded",
"startedAt": "2026-07-09T09:15:01Z",
"completedAt": "2026-07-09T09:15:10Z"
}
],
"total": 1000,
"offset": 0,
"limit": 50,
"hasMore": true
}
```
The results are paginated. Pass the `offset` and `limit` query parameters to page through results. The `hasMore` field indicates whether more pages remain.
## Full API reference
| Method | Path | Description |
| -------- | --------------------------------------------------- | ---------------------------------------------------- |
| `POST` | `/v2/{endpoint_id}/batch` | Create a new batch, optionally with initial requests |
| `POST` | `/v2/{endpoint_id}/batch/{id}/requests` | Append requests to a DRAFT batch |
| `POST` | `/v2/{endpoint_id}/batch/{id}/finalize` | Lock the batch and make it eligible for execution |
| `PUT` | `/v2/{endpoint_id}/batch/{id}` | Update batch attributes (e.g. display name) |
| `DELETE` | `/v2/{endpoint_id}/batch/{id}/requests/{requestId}` | Remove a single request from a DRAFT batch |
| `GET` | `/v2/{endpoint_id}/batch` | List all batches for an endpoint, newest first |
| `GET` | `/v2/{endpoint_id}/batch/{id}` | Batch summary with request counts |
| `POST` | `/v2/{endpoint_id}/batch/{id}/cancel` | Cancel a batch |
| `GET` | `/v2/{endpoint_id}/batch/{id}/requests` | Paginated child request list |
For full request and response schemas, see the [API reference](#full-api-reference).
## Monitoring batches in the console
Open your endpoint in the Runpod console and select the **Batch** tab to see all batches. Each row shows the batch name, status, and progress counts.
Click a batch to open the detail view, which shows:
* Top-level status and progress
* Per-request rows with status, timestamps, and error messages for failed requests
* Links to the full request detail view for each child request
The child request list is sorted by failures first, then in-progress, then queued, then completed.
## Notifications
When a batch reaches a terminal state (`FAILED` or `CANCELLED`), Runpod sends:
* **Console Inbox notification** — includes batch ID, endpoint name, terminal status, and item counts (completed / failed / total)
* **Webhook event** — if your account has a webhook subscription configured for batch events
Notifications are sent once per terminal state transition and are not fired for intermediate progress.
## Cancellation
To cancel a batch:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
POST /v2/{endpoint_id}/batch/{batch_id}/cancel
Authorization: Bearer {api_key}
```
Cancellation behavior:
* **Queued requests** are cancelled immediately and are not billed.
* **In-progress requests** are allowed to finish and are billed normally.
The batch status transitions to `CANCELLED` once all in-progress work has drained.
## Limits
| Limit | Value |
| ---------------------------- | ------ |
| Active batches per endpoint | 10 |
| Requests per batch | 5,000 |
| Queued requests per endpoint | 50,000 |
The maximum queued requests per endpoint can go up to 1,000,000 daily jobs. Limits are configurable for enterprise accounts. Contact support for custom limits.
## Billing
Batch jobs are billed at the same rate as standard serverless requests on your endpoint. For enterprise customers, flex worker discounts apply to batch jobs. Billing is based on the compute time used by each child request, regardless of whether the batch was later cancelled (in-progress requests that completed before cancellation are billed normally).
## Error handling
**Individual request failures** — A failed child request does not fail the entire batch. The batch stays `FINALIZED` and continues processing the remaining requests; overall completion is inferred from the request counts (all requests are done when `requestCompleted + requestFailed` equals `requestTotal`). Inspect failed requests via the console or the `GET .../requests` endpoint; each failed request includes an error message from the handler.
**Batch-level failure** — If the batch itself fails (status `FAILED`), it indicates a systemic problem rather than individual handler errors. Contact support if you see this state and cannot explain it from request-level errors.
**Redis durability** — Batch jobs use the same Redis-backed queue as standard serverless requests. In the event of a Redis failure, queued batch requests may be lost. This is an MVP limitation that applies equally to `/run` traffic.
## Known limitations
* Batch jobs inherit the GPU type configured on your endpoint. You cannot specify a different GPU per batch or per request.
* There is no per-request scheduling or ordering. Requests within a batch are processed in an unspecified order.
* Cost estimation before finalization is not available at launch.
* Runpod schedules batch workers based on global queue urgency and off-peak capacity, so start times aren't guaranteed.
# Aggregate streaming outputs
Source: https://docs.runpod.io/serverless/development/aggregate-outputs
Automatically collect and aggregate yielded results from streaming handler functions. See Runpod Serverless setup and usage details.
When building a streaming that yields results incrementally, you can use the `return_aggregate_stream` feature to automatically collect all yielded outputs into a single aggregated response. This simplifies result handling by eliminating the need to manually collect and format streaming results, making your handlers easier to implement and consume.
This guide shows you how to use output aggregation effectively in your applications.
## Understanding output aggregation
By default, streaming handlers that yield results are only accessible via the `/stream` operation, which returns results as they become available. When you enable `return_aggregate_stream`, Runpod automatically:
1. Collects all yielded results as your handler produces them.
2. Aggregates them into a single list.
3. Makes the complete aggregated results available via `/run` and `/runsync` operations.
This allows clients to choose between streaming individual results as they arrive or waiting for the complete aggregated response.
## Basic aggregation example
Here's a simple handler that processes multiple items and yields results incrementally:
```python handler.py theme={"theme":{"light":"github-light","dark":"github-dark"}}
import runpod
def handler(job):
job_input = job["input"]
items = job_input.get("items", [])
results = []
for item in items:
# Process each item
result = f"Processed: {item}"
# Yield result immediately for streaming
yield result
# Also collect for final return
results.append(result)
# Return complete list
return results
runpod.serverless.start({
"handler": handler,
"return_aggregate_stream": True
})
```
When a client calls this handler with multiple items, they can:
* Use `/stream` to receive each result as it's yielded.
* Use `/run` or `/runsync` to receive all results aggregated into a list.
## Processing multiple items
A common pattern is processing a batch of items and yielding results as each completes. This is useful for tasks like:
* Analyzing sentiment for multiple text samples.
* Generating images from multiple prompts.
* Running inference on multiple inputs.
Here's a practical example:
```python handler.py theme={"theme":{"light":"github-light","dark":"github-dark"}}
import runpod
import time
def analyze_items(items, task_type):
"""Process items based on task type."""
results = []
for item in items:
# Simulate processing time
time.sleep(0.5)
# Process based on type
if task_type == "sentiment":
result = {"text": item, "sentiment": "positive", "score": 0.92}
elif task_type == "classify":
result = {"text": item, "category": "technology", "confidence": 0.88}
else:
result = {"error": f"Unknown task type: {task_type}"}
# Yield each result as it completes
yield result
results.append(result)
return results
def handler(job):
job_input = job["input"]
task_type = job_input.get("task_type", "sentiment")
items = job_input.get("items", [])
# Validate input
if not items:
return {"error": "No items provided"}
# Process items and yield results
return analyze_items(items, task_type)
runpod.serverless.start({
"handler": handler,
"return_aggregate_stream": True
})
```
This handler processes each item sequentially, yielding results immediately while building a complete list to return.
## Local testing with aggregation
When testing locally, generators behave differently than in production. You need to handle the difference between the local test environment and production:
```python handler.py theme={"theme":{"light":"github-light","dark":"github-dark"}}
import runpod
def handler(job):
job_input = job["input"]
items = job_input.get("items", [])
for item in items:
result = f"Processed: {item}"
yield result
def start_handler():
"""Wrapper to handle local testing vs. production."""
def wrapper(job):
generator = handler(job)
# In local testing, convert generator to list
if job.get("id") == "local_test":
return list(generator)
# In production, return the generator
return generator
runpod.serverless.start({
"handler": wrapper,
"return_aggregate_stream": True
})
if __name__ == "__main__":
start_handler()
```
The wrapper function checks if the job ID is `local_test` (indicating local testing) and converts the generator to a list. In production, it returns the generator directly, allowing Runpod to handle the aggregation.
## Testing locally
Create a test input file to verify your aggregation works correctly:
```json test_input.json theme={"theme":{"light":"github-light","dark":"github-dark"}}
{
"input": {
"task_type": "sentiment",
"items": [
"I love this product!",
"The service was okay.",
"Not great, could be better."
]
}
}
```
Run your handler:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
python handler.py --test_input '{"input": {"task_type": "sentiment", "items": ["Item 1", "Item 2", "Item 3"]}}'
```
You should see output showing each result being processed and the final aggregated list:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
--- Starting Serverless Worker | Version 1.6.2 ---
INFO | Using test_input.json as job input.
DEBUG | Retrieved local job: {'input': {'task_type': 'sentiment', 'items': ['Item 1', 'Item 2', 'Item 3']}, 'id': 'local_test'}
INFO | local_test | Started.
DEBUG | local_test | Handler output: ['Processed: Item 1', 'Processed: Item 2', 'Processed: Item 3']
INFO | Job local_test completed successfully.
```
## Understanding the output format
When `return_aggregate_stream` is enabled, the final output structure includes all yielded results in a list:
**Without aggregation** (streaming only):
* Results arrive one at a time via `/stream`.
* No combined output available via `/run` or `/runsync`.
**With aggregation enabled:**
* Individual results still available via `/stream` as they're yielded.
* Complete aggregated list available via `/run` and `/runsync`:
```json theme={"theme":{"light":"github-light","dark":"github-dark"}}
{
"output": [
{"text": "Item 1", "sentiment": "positive", "score": 0.92},
{"text": "Item 2", "sentiment": "neutral", "score": 0.54},
{"text": "Item 3", "sentiment": "negative", "score": 0.78}
]
}
```
## When to use output aggregation
Use `return_aggregate_stream` for:
* **Batch processing**: Processing multiple items and clients need the complete set of results.
* **Progress tracking**: Clients want to see incremental progress but also need the final aggregated results.
* **Flexible consumption**: Supporting both streaming and batch consumption patterns.
* **Simplified integration**: Clients don't want to implement streaming logic but still benefit from incremental processing.
Don't use it for:
* **Large result sets**: Aggregating thousands of results can create memory pressure and large response payloads.
* **True streaming only**: Results should only be consumed as a stream (like real-time chat).
* **Single result**: Handler only returns one result (no need for aggregation).
## Best practices
1. **Memory management**: Be mindful of memory usage when aggregating large numbers of results.
2. **Payload limits**: Remember the payload size limits:
* `/run` operation: 10 MB
* `/runsync` operation: 20 MB
If aggregated results exceed these limits, consider using streaming only or storing results in cloud storage.
3. **Error handling**: Handle errors for individual items without breaking the entire batch:
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
def handler(job):
items = job["input"].get("items", [])
for item in items:
try:
result = process_item(item)
yield {"success": True, "result": result}
except Exception as e:
yield {"success": False, "error": str(e), "item": item}
```
4. **Consistent output structure**: Yield results in a consistent format to simplify client-side processing.
## Combining with async handlers
You can also use aggregation with async handlers for concurrent processing:
```python handler.py theme={"theme":{"light":"github-light","dark":"github-dark"}}
import runpod
import asyncio
async def async_handler(job):
items = job["input"].get("items", [])
for item in items:
# Simulate async processing
await asyncio.sleep(0.5)
result = f"Async processed: {item}"
yield result
runpod.serverless.start({
"handler": async_handler,
"return_aggregate_stream": True
})
```
This combines the benefits of async processing with automatic output aggregation.
## Next steps
* Learn more about [streaming handlers](/serverless/workers/handler-functions#streaming-handlers).
* Explore [async handlers](/serverless/workers/handler-functions#asynchronous-handlers) for concurrent processing.
* Understand [error handling](/serverless/workers/handler-functions#error-handling) for robust batch processing.
* Review [payload limits](/serverless/workers/handler-functions#payload-limits) to avoid oversized responses.
# Benchmark workers and requests
Source: https://docs.runpod.io/serverless/development/benchmarking
Measure the performance of your Serverless workers and identify bottlenecks. Review configuration and operations guidance for Runpod Serverless.
Benchmarking your Serverless workers helps you identify bottlenecks and [optimize your code](/serverless/development/optimization) for performance and cost.
Performance is measured by two key metrics:
* **Delay time**: The time spent waiting for a worker to become available. This includes the cold start time if a new worker needs to be spun up.
* **Execution time**: The time the GPU takes to process the request once the worker has received the job.
## Send a test request
To gather initial metrics, use `curl` to send a request to your endpoint. This will initiate the job and return a request ID that you can use to poll for status.
```sh theme={"theme":{"light":"github-light","dark":"github-dark"}}
curl -X POST https://api.runpod.ai/v2/YOUR_ENDPOINT_ID/run \
-H "Content-Type: application/json" \
-H "Authorization: Bearer YOUR_API_KEY" \
-d '{"input": {"prompt": "Hello, world!"}}'
```
This returns a JSON object containing the request ID. Poll the `/status` endpoint to get the delay time and execution time:
```sh theme={"theme":{"light":"github-light","dark":"github-dark"}}
curl -X GET https://api.runpod.ai/v2/YOUR_ENDPOINT_ID/status/REQUEST_ID \
-H "Content-Type: application/json" \
-H "Authorization: Bearer YOUR_API_KEY"
```
This returns a JSON object:
```json theme={"theme":{"light":"github-light","dark":"github-dark"}}
{
"id": "1234567890",
"status": "COMPLETED",
"delayTime": 1000,
"executionTime": 2000
}
```
### Automate benchmarking
To get a statistically significant view of your worker's performance, you should automate the benchmarking process. The following Python script sends multiple requests and calculates the minimum, maximum, and average times for both delay and execution.
```python benchmark.py theme={"theme":{"light":"github-light","dark":"github-dark"}}
import requests
import time
import statistics
ENDPOINT_ID = "YOUR_ENDPOINT_ID"
API_KEY = "YOUR_API_KEY"
BASE_URL = f"https://api.runpod.ai/v2/{ENDPOINT_ID}"
HEADERS = {
"Content-Type": "application/json",
"Authorization": f"Bearer {API_KEY}"
}
def run_benchmark(num_requests=5):
delay_times = []
execution_times = []
for i in range(num_requests):
# Send request
response = requests.post(
f"{BASE_URL}/run",
headers=HEADERS,
json={"input": {"prompt": f"Test request {i+1}"}}
)
request_id = response.json()["id"]
# Poll for completion
while True:
status_response = requests.get(
f"{BASE_URL}/status/{request_id}",
headers=HEADERS
)
status_data = status_response.json()
if status_data["status"] == "COMPLETED":
delay_times.append(status_data["delayTime"])
execution_times.append(status_data["executionTime"])
break
elif status_data["status"] == "FAILED":
print(f"Request {i+1} failed")
break
time.sleep(1)
# Calculate statistics
print(f"Delay Time - Min: {min(delay_times)}ms, Max: {max(delay_times)}ms, Avg: {statistics.mean(delay_times):.0f}ms")
print(f"Execution Time - Min: {min(execution_times)}ms, Max: {max(execution_times)}ms, Avg: {statistics.mean(execution_times):.0f}ms")
if __name__ == "__main__":
run_benchmark(num_requests=5)
```
# Clean up temporary files
Source: https://docs.runpod.io/serverless/development/cleanup
Manage disk space by automatically removing temporary files. Review setup, configuration, deployment, and operations guidance for Runpod Serverless.
The Runpod SDK's `clean()` function helps maintain the health of your Serverless worker by removing temporary files and folders after processing completes. This is particularly important for workers that download large assets or generate temporary artifacts, as accumulated data can lead to `DiskQuotaExceeded` errors over time.
## Import the `clean()` function
To use the `clean()` function, import it from the `utils.rp_cleanup` module:
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
from runpod.serverless.utils.rp_cleanup import clean
```
## Default behavior
When called without arguments, `clean()` targets a specific set of default directories for removal:
* `input_objects/`
* `output_objects/`
* `job_files/`
* `output.zip`
These are standard locations used by various SDK operations, and cleaning them ensures a fresh state for the next request.
## Custom cleanup
If your handler generates files in non-standard directories, you can override the default behavior by passing a list of folder names to the `folder_list` parameter.
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
clean(folder_list=["temp_images", "cache", "downloads"])
```
## Use `clean()` in your handler
You should integrate cleanup logic into your handler's lifecycle, typically within a `finally` block or right before returning the result.
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
import runpod
from runpod.serverless.utils.rp_cleanup import clean
import requests
import os
def download_image(url, save_path):
response = requests.get(url)
if response.status_code == 200:
with open(save_path, "wb") as file:
file.write(response.content)
return True
return False
def handler(event):
try:
image_url = event["input"]["image_url"]
# Create a temporary directory
os.makedirs("temp_images", exist_ok=True)
image_path = "temp_images/downloaded_image.jpg"
# Download the image
if not download_image(image_url, image_path):
raise Exception("Failed to download image")
# Process the image (your code here)
result = f"Processed image from: {image_url}"
# Cleanup specific folders after processing
clean(folder_list=["temp_images"])
return {"output": result}
except Exception as e:
# Attempt cleanup even if an error occurs
clean(folder_list=["temp_images"])
return {"error": str(e)}
runpod.serverless.start({"handler": handler})
```
## Best practices
To ensure reliability, always call `clean()` at the end of your handler execution. We recommend wrapping your cleanup calls in a `try...except` or `finally` block so that disk space is recovered even if your main processing logic fails.
Be cautious when adding custom folders to the cleanup list to avoid accidentally deleting persistent data, and consider logging cleanup actions during development to verify that the correct paths are being targeted.
# Pod-first development
Source: https://docs.runpod.io/serverless/development/dual-mode-worker
Develop on a Pod before deploying your worker to Serverless for faster iteration. Review configuration and operations guidance for Runpod Serverless.
Developing machine learning applications often requires powerful GPUs, making local development challenging. Instead of repeatedly deploying your worker to Serverless for testing, you can develop on a Pod first and then deploy the same Docker image to Serverless when ready.
This "Pod-first" workflow lets you develop and test interactively in a GPU environment, then seamlessly transition to Serverless for production. You'll use a Pod as your cloud-based development machine with tools like Jupyter Notebooks and SSH, catching issues early before deploying your worker to Serverless.
To get started quickly, you can [clone this repository](https://github.com/justinwlin/Runpod-GPU-And-Serverless-Base) for a pre-configured template for a dual-mode worker.
## Requirements
* You've [created a Runpod account](/accounts-billing/manage-accounts).
* You've installed [Python 3.x](https://www.python.org/downloads/) and [Docker](https://docs.docker.com/get-started/get-docker/) and configured them for your command line.
* Basic understanding of Docker concepts and shell scripting.
## Step 1: Set up your project structure
Create a directory for your project and the necessary files:
```sh theme={"theme":{"light":"github-light","dark":"github-dark"}}
mkdir dual-mode-worker
cd dual-mode-worker
touch handler.py start.sh Dockerfile requirements.txt
```
This creates the following project structure:
## Step 2: Create the handler
This Python script will check for a `MODE_TO_RUN` environment variable to determine whether to run in Pod or Serverless mode.
Add the following code to `handler.py`:
```python handler.py theme={"theme":{"light":"github-light","dark":"github-dark"}}
import os
import asyncio
import runpod
# Use the MODEL environment variable; fallback to a default if not set
mode_to_run = os.getenv("MODE_TO_RUN", "pod")
model_length_default = 25000
print("------- ENVIRONMENT VARIABLES -------")
print("Mode running: ", mode_to_run)
print("------- -------------------- -------")
async def handler(event):
inputReq = event.get("input", {})
return inputReq
if mode_to_run == "pod":
async def main():
prompt = "Hello World"
requestObject = {"input": {"prompt": prompt}}
response = await handler(requestObject)
print(response)
asyncio.run(main())
else:
runpod.serverless.start({
"handler": handler,
"concurrency_modifier": lambda current: 1,
})
```
Key features:
* `MODE_TO_RUN = os.getenv("MODE_TO_RUN", "pod")`: Reads the mode from an environment variable, defaulting to `pod`.
* `async def handler(event)`: Your core logic.
* `if mode_to_run == "pod" ... else`: This conditional controls what happens when the script is executed directly.
* In `pod` mode, it runs a sample test call to your `handler` function, allowing for quick iteration.
* In `serverless`" mode, it starts the Runpod Serverless worker.
## Step 3: Create the `start.sh` script
The `start.sh` script serves as the entrypoint for your Docker container and manages different operational modes. It reads the `MODE_TO_RUN` environment variable and configures the container accordingly.
Add the following code to `start.sh`:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
#!/bin/bash
set -e # Exit the script if any statement returns a non-true return value
# Set workspace directory from env or default
WORKSPACE_DIR="${WORKSPACE_DIR:-/workspace}"
# Start nginx service
start_nginx() {
echo "Starting Nginx service..."
service nginx start
}
# Execute script if exists
execute_script() {
local script_path=$1
local script_msg=$2
if [[ -f ${script_path} ]]; then
echo "${script_msg}"
bash ${script_path}
fi
}
# Setup ssh
setup_ssh() {
if [[ $PUBLIC_KEY ]]; then
echo "Setting up SSH..."
mkdir -p ~/.ssh
echo "$PUBLIC_KEY" >> ~/.ssh/authorized_keys
chmod 700 -R ~/.ssh
# Generate SSH host keys if not present
generate_ssh_keys
service ssh start
echo "SSH host keys:"
cat /etc/ssh/*.pub
fi
}
# Generate SSH host keys
generate_ssh_keys() {
ssh-keygen -A
}
# Export env vars
export_env_vars() {
echo "Exporting environment variables..."
printenv | grep -E '^RUNPOD_|^PATH=|^_=' | awk -F = '{ print "export " $1 "=\"" $2 "\"" }' >> /etc/rp_environment
echo 'source /etc/rp_environment' >> ~/.bashrc
}
# Start jupyter lab
start_jupyter() {
echo "Starting Jupyter Lab..."
mkdir -p "$WORKSPACE_DIR" && \
cd / && \
nohup jupyter lab --allow-root --no-browser --port=8888 --ip=* --NotebookApp.token='' --NotebookApp.password='' --FileContentsManager.delete_to_trash=False --ServerApp.terminado_settings='{"shell_command":["/bin/bash"]}' --ServerApp.allow_origin=* --ServerApp.preferred_dir="$WORKSPACE_DIR" &> /jupyter.log &
echo "Jupyter Lab started without a password"
}
# Call Python handler if mode is serverless or both
call_python_handler() {
echo "Calling Python handler.py..."
python $WORKSPACE_DIR/handler.py
}
# ---------------------------------------------------------------------------- #
# Main Program #
# ---------------------------------------------------------------------------- #
start_nginx
echo "Pod Started"
setup_ssh
case $MODE_TO_RUN in
serverless)
echo "Running in serverless mode"
call_python_handler
;;
pod)
echo "Running in pod mode"
start_jupyter
;;
*)
echo "Invalid MODE_TO_RUN value: $MODE_TO_RUN. Expected 'serverless', 'pod', or 'both'."
exit 1
;;
esac
export_env_vars
echo "Start script(s) finished"
sleep infinity
```
Here are some key features of this script:
* `case $MODE_TO_RUN in ... esac`: This structure directs the startup based on the mode.
* `serverless` mode: Executes `handler.py`, which then starts the Runpod Serverless worker. `exec` replaces the shell process with the Python process.
* `pod` mode: Starts up the JupyterLab server for Pod development, then runs `sleep infinity` to keep the container alive so you can connect to it (e.g., via SSH or `docker exec`). You would then manually run `python /app/handler.py` inside the Pod to test your handler logic.
## Step 4: Create the `Dockerfile`
Create a `Dockerfile` that includes your handler and startup script:
```dockerfile theme={"theme":{"light":"github-light","dark":"github-dark"}}
# Use an official Runpod base image
FROM runpod/pytorch:2.0.1-py3.10-cuda11.8.0-devel-ubuntu22.04
# Environment variables
ENV PYTHONUNBUFFERED=1
# Supported modes: pod, serverless
ARG MODE_TO_RUN=pod
ENV MODE_TO_RUN=$MODE_TO_RUN
# Set up the working directory
ARG WORKSPACE_DIR=/app
ENV WORKSPACE_DIR=${WORKSPACE_DIR}
WORKDIR $WORKSPACE_DIR
# Install dependencies in a single RUN command to reduce layers and clean up in the same layer to reduce image size
RUN apt-get update --yes --quiet && \
DEBIAN_FRONTEND=noninteractive apt-get install --yes --quiet --no-install-recommends \
software-properties-common \
gpg-agent \
build-essential \
apt-utils \
ca-certificates \
curl && \
add-apt-repository --yes ppa:deadsnakes/ppa && \
apt-get update --yes --quiet && \
DEBIAN_FRONTEND=noninteractive apt-get install --yes --quiet --no-install-recommends
# Create and activate a Python virtual environment
RUN python3 -m venv /app/venv
ENV PATH="/app/venv/bin:$PATH"
# Install Python packages
RUN pip install --no-cache-dir \
asyncio \
requests \
runpod
# Install requirements.txt
COPY requirements.txt ./requirements.txt
RUN pip install --no-cache-dir --upgrade pip && \
pip install --no-cache-dir -r requirements.txt
# Delete's the default start.sh file from Runpod (so we can replace it with our own below)
RUN rm ../start.sh
# Copy all of our files into the container
COPY handler.py $WORKSPACE_DIR/handler.py
COPY start.sh $WORKSPACE_DIR/start.sh
# Make sure start.sh is executable
RUN chmod +x start.sh
# Make sure that the start.sh is in the path
RUN ls -la $WORKSPACE_DIR/start.sh
# depot build -t justinrunpod/pod-server-base:1.0 . --push --platform linux/amd64
CMD $WORKSPACE_DIR/start.sh
```
Key features of this `Dockerfile`:
* `FROM runpod/pytorch:2.0.1-py3.10-cuda11.8.0-devel-ubuntu22.04`: Starts with a Runpod base image that comes with nginx, runpodctl, and other helpful base packages.
* `ARG WORKSPACE_DIR=/workspace` and `ENV WORKSPACE_DIR=${WORKSPACE_DIR}`: Allows the workspace directory to be set at build time.
* `WORKDIR $WORKSPACE_DIR`: Sets the working directory to the value of `WORKSPACE_DIR`.
* `COPY requirements.txt ./requirements.txt` and `RUN pip install ...`: Installs Python dependencies.
* `COPY . .`: Copies all application files into the workspace directory.
* `ENV MODE_TO_RUN="pod"`: Sets the default operational mode to "pod". This can be overridden at runtime.
* `CMD ["$WORKSPACE_DIR/start.sh"]`: Specifies `start.sh` as the command to run when the container starts.
## Step 5: Build and push your Docker image
Instead of building and pushing your image via Docker Hub, you can also [deploy your worker from a GitHub repository](/serverless/workers/github-integration).
Now you're ready to build your Docker image and push it to Docker Hub:
Build your Docker image, replacing `YOUR_USERNAME` with your Docker Hub username and choosing a suitable image name:
```sh theme={"theme":{"light":"github-light","dark":"github-dark"}}
docker build --platform linux/amd64 --tag YOUR_USERNAME/dual-mode-worker .
```
The `--platform linux/amd64` flag is important for compatibility with Runpod's infrastructure.
```sh theme={"theme":{"light":"github-light","dark":"github-dark"}}
docker push YOUR_USERNAME/dual-mode-worker:latest
```
You might need to run `docker login` first.
## Step 6: Testing in Pod mode
Now that you've finished building our Docker image, let's explore how you would use the Pod-first development workflow in practice.
Deploy the image to a Pod by following these steps:
1. Navigate to the [Pods page](https://console.runpod.io/pods) in the Runpod console.
2. Click **Deploy**.
3. Select your preferred GPU.
4. Under **Container Image**, enter `YOUR_USERNAME/dual-mode-worker:latest`.
5. Under **Public Environment Variables**, select **Add environment variable** and add:
* Key: `MODE_TO_RUN`
* Value: `pod`
6. Click **Deploy**.
Once your Pod is running, you can:
* [Connect via the web terminal, JupyterLab, or SSH](/pods/connect-to-a-pod) to test your handler interactively.
* Debug and iterate on your code.
* Test GPU-specific operations.
* Edit `handler.py` within the Pod and re-run it for rapid iteration.
## Step 7: Deploy to a Serverless endpoint
Once you're confident with your `handler.py` logic tested in Pod mode, you're ready to deploy your dual-mode worker to a Serverless endpoint.
1. Navigate to the [Serverless page](https://console.runpod.io/serverless) in the Runpod console.
2. Click **New Endpoint**.
3. Click **Import from Docker Registry**.
4. In the **Container Image** field, enter your Docker image URL: `docker.io/YOUR_USERNAME/dual-mode-worker:latest`, then click *Next*\*\*\*.
5. Under **Environment Variables**, add:
* Key: `MODE_TO_RUN`
* Value: `serverless`
6. Configure your endpoint settings (GPU type, workers, etc.).
7. Click **Deploy Endpoint**.
The *same* image will be used for your workers, but `start.sh` will now direct them to run in Serverless mode, using the `runpod.serverless.start()` function to process requests.
## Step 8: Test your endpoint
After deploying your endpoint in to Serverless mode, you can test it by sending API requests to your endpoint.
1. Navigate to your endpoint's detail page in the Runpod console.
2. Click the **Requests** tab.
3. Use the following JSON as test input:
```json theme={"theme":{"light":"github-light","dark":"github-dark"}}
{
"input": {
"prompt": "Hello World!",
}
}
```
4. Click **Run**.
After a few moments for initialization and processing, you should see output similar to this:
```json theme={"theme":{"light":"github-light","dark":"github-dark"}}
{
"delayTime": 12345, // This will vary
"executionTime": 3050, // This will be around 3000ms + overhead
"id": "some-unique-id",
"output": {
"output": "Processed prompt: 'Hello Serverless World!' after 3s in Serverless mode."
},
"status": "COMPLETED"
}
```
## Explore the Pod-first development workflow
Congratulations! You've successfully built, deployed, and tested a dual-mode Serverless worker.
Now, let's explore the recommended iteration process for a Pod-first development workflow:
1. Deploy your initial Docker image to a Runpod Pod, ensuring `MODE_TO_RUN` is set to `pod` (or rely on the Dockerfile default).
2. [Connect to your Pod](/pods/connect-to-a-pod) (via SSH or web terminal).
3. Navigate to the `/app` directory.
4. As you develop, install any necessary Python packages (`pip install PACKAGE_NAME`) or system dependencies (`apt-get install PACKAGE_NAME`).
5. Iterate on your `handler.py` script. Test your changes frequently by running `python handler.py` directly in the Pod's terminal. This will execute the test harness you defined in the `elif MODE_TO_RUN == "pod":` block, giving you immediate feedback.
Once you're satisfied with a set of changes and have new dependencies:
1. Add new Python packages to your `requirements.txt` file.
2. Add system installation commands (e.g., `RUN apt-get update && apt-get install -y PACKAGE_NAME`) to your `Dockerfile`.
3. Ensure your updated `handler.py` is saved.
1. Re-deploy your worker image to a Serverless endpoint using [Docker Hub](/serverless/workers/deploy) or [GitHub](/serverless/workers/github-integration).
2. During deployment, ensure that the `MODE_TO_RUN` environment variable for the endpoint is set to `serverless`.
For instructions on how to set environment variables during deployment, see [Manage endpoints](/serverless/endpoints/overview).
3. After your endpoint is deployed, you can test it by [sending API requests](/serverless/endpoints/send-requests).
This iterative loop (write your handler, update the Docker image, test in Pod mode, then deploy to Serverless) enables you to rapidly develop and debug your Serverless workers.
# Environment variables
Source: https://docs.runpod.io/serverless/development/environment-variables
Configure your Serverless endpoints with environment variables. Review configuration and operations guidance for Runpod Serverless.
Environment variables let you configure your endpoints without hardcoding credentials or settings in your code. They're ideal for managing API keys, service URLs, feature flags, and other configuration that changes between development and production.
## How environment variables work
Environment variables are set in the Runpod console and are available to your handler at runtime through `os.environ`. Your handler can read these variables to configure its behavior.
## Set environment variables
You can set environment variables in the Runpod console when [creating or editing your endpoint](/serverless/endpoints/overview):
1. Navigate to your endpoint in the [Runpod console](https://console.runpod.io/serverless).
2. Click on the **Settings** tab.
3. Scroll to the **Environment Variables** section.
4. Add your variables as key-value pairs.
5. Click **Save** to apply the changes.
These environment variables will be available to all workers running on your endpoint.
### Access environment variables in your handler
You can access environment variables in your handler function at runtime using `os.environ.get("VARIABLE_NAME")`. For example:
```python title="handler.py" theme={"theme":{"light":"github-light","dark":"github-dark"}}
import os
import runpod
def handler(job):
# Read environment variables
api_key = os.environ.get("API_KEY")
service_url = os.environ.get("SERVICE_URL", "https://default-url.com")
# Use the configuration
result = call_external_service(service_url, api_key)
return {"output": result}
runpod.serverless.start({"handler": handler})
```
## Build-time vs runtime variables
There are two types of environment variables:
### Build-time variables
Build-time variables are set in your Dockerfile using the `ENV` instruction. These are baked into your Docker image during the build:
```dockerfile title="Dockerfile" theme={"theme":{"light":"github-light","dark":"github-dark"}}
FROM runpod/base:0.4.0-cuda11.8.0
# Build-time environment variables
ENV MODEL_NAME="llama-2-7b"
ENV DEFAULT_TEMPERATURE="0.7"
COPY handler.py /handler.py
CMD ["python", "-u", "/handler.py"]
```
Build-time variables are useful for setting default configuration values, values that rarely change, and non-sensitive information.
### Runtime variables
Runtime variables are set in the Runpod console and can be changed without rebuilding your image. These override build-time variables with the same name:
Runtime variables are useful for:
* API keys and secrets.
* Environment-specific configuration (dev, staging, prod).
* Values that change frequently.
* Sensitive information that shouldn't be in your image.
## Common use cases
### API keys and secrets
Store sensitive credentials as runtime environment variables:
```python title="handler.py" theme={"theme":{"light":"github-light","dark":"github-dark"}}
import os
import runpod
import requests
def handler(event):
# Read API keys from environment
openai_key = os.environ.get("OPENAI_API_KEY")
anthropic_key = os.environ.get("ANTHROPIC_API_KEY")
# Use them in your code
if not openai_key:
return {"error": "OPENAI_API_KEY not configured"}
# Your API call here
result = call_openai(openai_key, event["input"]["prompt"])
return {"output": result}
runpod.serverless.start({"handler": handler})
```
Never hardcode API keys or secrets in your code. Always use environment variables to keep credentials secure.
### S3 bucket configuration
Configure S3 or S3-compatible storage for uploading results:
```python title="handler.py" theme={"theme":{"light":"github-light","dark":"github-dark"}}
import os
import runpod
from runpod.serverless.utils import rp_upload
def handler(event):
# S3 credentials are read from environment variables:
# - BUCKET_ENDPOINT_URL
# - BUCKET_ACCESS_KEY_ID
# - BUCKET_SECRET_ACCESS_KEY
# Process your input
result_image_path = generate_image(event["input"]["prompt"])
# Upload to S3
image_url = rp_upload.upload_image(event["id"], result_image_path)
return {"output": {"image_url": image_url}}
runpod.serverless.start({"handler": handler})
```
Set these variables in the Runpod console:
* `BUCKET_ENDPOINT_URL`: Your bucket endpoint (e.g., `https://your-bucket.s3.us-west-2.amazonaws.com`)
* `BUCKET_ACCESS_KEY_ID`: Your access key ID
* `BUCKET_SECRET_ACCESS_KEY`: Your secret access key
The `BUCKET_ENDPOINT_URL` should include your bucket name in the URL.
### Feature flags
Use environment variables to enable or disable features:
```python title="handler.py" theme={"theme":{"light":"github-light","dark":"github-dark"}}
import os
import runpod
def handler(event):
# Read feature flags
enable_caching = os.environ.get("ENABLE_CACHING", "false").lower() == "true"
enable_logging = os.environ.get("ENABLE_LOGGING", "true").lower() == "true"
if enable_logging:
print(f"Processing request: {event['id']}")
# Your processing logic
result = process_input(event["input"], use_cache=enable_caching)
return {"output": result}
runpod.serverless.start({"handler": handler})
```
### Model configuration
Configure model parameters without changing code:
```python title="handler.py" theme={"theme":{"light":"github-light","dark":"github-dark"}}
import os
import runpod
def handler(event):
# Read model configuration from environment
model_name = os.environ.get("MODEL_NAME", "default-model")
max_tokens = int(os.environ.get("MAX_TOKENS", "1024"))
temperature = float(os.environ.get("TEMPERATURE", "0.7"))
# Use configuration in your model
result = generate_text(
model=model_name,
prompt=event["input"]["prompt"],
max_tokens=max_tokens,
temperature=temperature
)
return {"output": result}
runpod.serverless.start({"handler": handler})
```
## Best practices
### Use defaults
Always provide default values for non-critical environment variables:
```python title="handler.py" theme={"theme":{"light":"github-light","dark":"github-dark"}}
# Good: Provides a default
service_url = os.environ.get("SERVICE_URL", "https://api.example.com")
# Good: Fails explicitly if missing
api_key = os.environ.get("API_KEY")
if not api_key:
raise ValueError("API_KEY environment variable is required")
```
### Validate on startup
Validate critical environment variables when your handler starts:
```python title="handler.py" theme={"theme":{"light":"github-light","dark":"github-dark"}}
import os
import runpod
# Validate environment variables on startup
required_vars = ["API_KEY", "SERVICE_URL"]
missing_vars = [var for var in required_vars if not os.environ.get(var)]
if missing_vars:
raise ValueError(f"Missing required environment variables: {', '.join(missing_vars)}")
def handler(event):
# Your handler logic here
pass
runpod.serverless.start({"handler": handler})
```
### Document your variables
Document the environment variables your handler expects in your README:
```markdown README.md theme={"theme":{"light":"github-light","dark":"github-dark"}}
## Environment Variables
| Variable | Required | Default | Description |
|----------|----------|---------|-------------|
| `API_KEY` | Yes | N/A | Your API key for the external service |
| `SERVICE_URL` | No | `https://api.example.com` | The service endpoint URL |
| `MAX_WORKERS` | No | `4` | Maximum number of concurrent workers |
```
### Separate secrets from config
Use different approaches for secrets vs configuration:
* **Secrets**: Only set as runtime variables in the Runpod console.
* **Configuration**: Can use build-time defaults with runtime overrides.
# Fitness checks and preflight checks
Source: https://docs.runpod.io/serverless/development/fitness-checks
Validate Serverless worker dependencies at startup with custom and automatic fitness checks, failure handling, configuration, and operations guidance.
Fitness checks validate a worker's environment at startup, before it begins processing jobs. When a worker starts, Runpod runs each registered check in order. If a check fails, the worker logs the error, exits, and is marked unhealthy so Runpod can restart or replace it before any traffic reaches it. This lets you catch problems like a missing GPU, a model that won't load, or absent configuration up front, instead of failing requests one by one in production.
Fitness checks are available in the Runpod Python SDK starting with version 1.9.0. To use them, make sure your worker image installs `runpod>=1.9.0`.
There are two kinds of fitness checks: custom checks that you define for your own workload, and automatic system checks that Runpod runs for you without any code changes.
## Register a custom fitness check
Use the `@runpod.serverless.register_fitness_check` decorator to register your own checks. A check is a function that raises an exception (typically `RuntimeError`) to signal failure. If it returns without raising, the check passes. Both synchronous and asynchronous functions are supported.
```python title="handler.py" theme={"theme":{"light":"github-light","dark":"github-dark"}}
import runpod
import torch
@runpod.serverless.register_fitness_check
def check_gpu():
"""Verify a GPU is available."""
if not torch.cuda.is_available():
raise RuntimeError("GPU not available")
@runpod.serverless.register_fitness_check
def check_disk_space():
"""Verify sufficient disk space."""
import shutil
stat = shutil.disk_usage("/")
free_gb = stat.free / (1024**3)
if free_gb < 10:
raise RuntimeError(f"Insufficient disk space: {free_gb:.2f}GB free")
def handler(job):
return {"output": "success"}
runpod.serverless.start({"handler": handler})
```
### Asynchronous checks
Checks can also be asynchronous functions, which is useful for I/O-bound validation such as confirming connectivity to an external service:
```python title="handler.py" theme={"theme":{"light":"github-light","dark":"github-dark"}}
import runpod
import aiohttp
@runpod.serverless.register_fitness_check
async def check_api_connectivity():
"""Check if an external API is accessible."""
async with aiohttp.ClientSession() as session:
try:
async with session.get("https://api.example.com/health", timeout=5) as resp:
if resp.status != 200:
raise RuntimeError(f"API health check failed: {resp.status}")
except Exception as e:
raise RuntimeError(f"Cannot connect to API: {e}")
def handler(job):
return {"output": "success"}
runpod.serverless.start({"handler": handler})
```
## When checks run
Custom checks run once at worker startup, before the first job, and they execute in the order you register them. They run only on the Runpod Serverless platform, so they're skipped during local development and testing. When a check fails, Runpod logs the check name and exception, the worker exits with code 1, and the container is marked unhealthy so your endpoint can restart it. When all checks pass, the worker starts its heartbeat and begins accepting jobs.
A failed startup looks like this in the logs:
```
ERROR | Fitness check failed: check_gpu | RuntimeError: GPU not available
ERROR | Worker is unhealthy, exiting.
```
A successful startup logs each check as it runs:
```
INFO | Running 2 fitness check(s)...
DEBUG | Executing fitness check: check_gpu
DEBUG | Fitness check passed: check_gpu
DEBUG | Executing fitness check: check_disk_space
DEBUG | Fitness check passed: check_disk_space
INFO | All fitness checks passed.
```
## Custom check examples
The following examples cover common readiness checks you can adapt to your own workload.
Verify that a GPU is available and has enough memory:
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
import runpod
import torch
@runpod.serverless.register_fitness_check
def check_gpu_available():
"""Verify a GPU is available and has sufficient memory."""
if not torch.cuda.is_available():
raise RuntimeError("GPU is not available")
gpu_memory_gb = torch.cuda.get_device_properties(0).total_memory / (1024**3)
if gpu_memory_gb < 8:
raise RuntimeError(f"GPU memory insufficient: {gpu_memory_gb:.1f}GB (need at least 8GB)")
```
Confirm that required model files are present:
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
import runpod
from pathlib import Path
@runpod.serverless.register_fitness_check
def check_model_files():
"""Verify required model files exist."""
required_files = [
Path("/models/model.safetensors"),
Path("/models/config.json"),
Path("/models/tokenizer.model"),
]
for file_path in required_files:
if not file_path.exists():
raise RuntimeError(f"Required file not found: {file_path}")
```
Validate that required environment variables are set:
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
import runpod
import os
@runpod.serverless.register_fitness_check
def check_environment():
"""Verify required environment variables are set."""
required_vars = ["API_KEY", "MODEL_PATH", "CONFIG_URL"]
missing = [var for var in required_vars if not os.environ.get(var)]
if missing:
raise RuntimeError(f"Missing environment variables: {', '.join(missing)}")
```
## Automatic system checks
Along with your custom checks, Runpod automatically runs a set of built-in system checks at startup, with no registration required. These verify that the worker has enough memory, disk space, and network connectivity to run reliably. GPU workers run three additional checks that validate the CUDA version, initialize the CUDA device, and benchmark GPU compute. GPU workers also run a native GPU memory allocation test that exercises actual memory allocation across all detected GPUs; this test is skipped automatically on CPU-only workers.
You can tune the thresholds for these checks using environment variables, or disable them entirely for testing. Your own registered checks always run, regardless of these settings.
| Check | Applies to | What it verifies | Default threshold | Environment variable |
| -------------------------- | ----------- | ------------------------------------------- | ----------------- | ------------------------------ |
| Memory | All workers | Available RAM | 4 GB minimum | `RUNPOD_MIN_MEMORY_GB` |
| Disk space | All workers | Free space on `/` as a percentage of total | 10% free | `RUNPOD_MIN_DISK_PERCENT` |
| Network connectivity | All workers | Internet connectivity via TCP to 8.8.8.8:53 | 5 second timeout | `RUNPOD_NETWORK_CHECK_TIMEOUT` |
| CUDA version | GPU workers | Minimum CUDA driver version | CUDA 11.8+ | `RUNPOD_MIN_CUDA_VERSION` |
| CUDA device initialization | GPU workers | Devices initialize and synchronize | — | — |
| GPU compute benchmark | GPU workers | Compute responsiveness (matrix multiply) | 100 ms maximum | `RUNPOD_GPU_BENCHMARK_TIMEOUT` |
The native GPU memory allocation test can be tuned with these additional variables. The test typically takes 100–500 ms per GPU, so keep that in mind when setting `RUNPOD_GPU_TEST_TIMEOUT`.
| Environment variable | Default | Description |
| ------------------------------- | ---------- | ------------------------------------------- |
| `RUNPOD_GPU_TEST_TIMEOUT` | 30 seconds | Timeout for the GPU memory allocation test. |
| `RUNPOD_BINARY_GPU_TEST_PATH` | — | Override the path to the GPU test binary. |
| `RUNPOD_GPU_MAX_ERROR_MESSAGES` | 10 | Maximum number of error messages to report. |
You can set these variables in your Dockerfile to adjust the thresholds for your workload:
```dockerfile title="Dockerfile" theme={"theme":{"light":"github-light","dark":"github-dark"}}
ENV RUNPOD_MIN_MEMORY_GB=8.0
ENV RUNPOD_MIN_DISK_PERCENT=15.0
ENV RUNPOD_MIN_CUDA_VERSION=12.0
ENV RUNPOD_NETWORK_CHECK_TIMEOUT=10
```
### Disable automatic checks
You can disable the automatic system checks with the following environment variables. This is intended for testing only and is not recommended in production, since the checks help catch unhealthy workers before they take traffic.
| Environment variable | Effect |
| ------------------------------------- | --------------------------------------------------------------------------------------------- |
| `RUNPOD_SKIP_AUTO_SYSTEM_CHECKS=true` | Skips the memory, disk, network, CUDA version, CUDA initialization, and GPU benchmark checks. |
| `RUNPOD_SKIP_GPU_CHECK=true` | Skips the native GPU memory allocation test. |
Your own registered checks still run even when these flags are set.
## Test fitness checks locally
Fitness checks don't run during local testing, so to verify yours before you deploy, call each check function directly. A check signals failure by raising an exception, so you can call it and handle the result yourself.
```python title="test_fitness.py" theme={"theme":{"light":"github-light","dark":"github-dark"}}
import asyncio
import inspect
from handler import check_gpu, check_disk_space
async def run_check(check):
"""Call a single fitness check and report the result."""
try:
if inspect.iscoroutinefunction(check):
await check()
else:
check()
print(f"{check.__name__} passed")
except Exception as e:
print(f"{check.__name__} failed: {e}")
async def main():
for check in (check_gpu, check_disk_space):
await run_check(check)
if __name__ == "__main__":
asyncio.run(main())
```
Don't use the runner, `run_fitness_checks()`, to test for failures. When a check fails, the runner force-kills the worker by calling `os._exit(1)`, which terminates the process immediately. It doesn't raise a catchable exception, and it skips any `except` or `finally` blocks. Call your check functions directly instead, as shown above.
## Best practices
Keep your checks fast and focused on validating readiness, rather than doing heavy work like training models or processing large datasets. Use clear, descriptive error messages so failures are easy to diagnose from the logs. Keep in mind that fitness checks only validate the worker at startup; to catch problems that arise while processing jobs, add health checks and logging inside your handler as well.
# Use Hugging Face models
Source: https://docs.runpod.io/serverless/development/huggingface-models
Integrate pre-trained Hugging Face models into your Serverless handler functions. Review configuration and operations guidance for Runpod Serverless.
Hugging Face provides thousands of pre-trained models for natural language processing, computer vision, audio processing, and more. You can integrate these models into your to deploy AI capabilities without training models from scratch.
This guide shows you how to load and use Hugging Face models in your Serverless handlers, using sentiment analysis as an example that you can adapt for other model types.
This guide covers two approaches:
* [Downloading models at runtime](#load-models-at-runtime) (simpler, good for development).
* [Using cached models](#use-cached-models) (recommended for production).
## Install dependencies
Your handler needs the `transformers` library to load Hugging Face models, and `torch` to run inference. Install both in your development environment:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
pip install torch transformers
```
When deploying to Runpod, you'll need to include these dependencies in your [Dockerfile](/serverless/workers/create-dockerfile) or requirements file.
## Load models at runtime
Create a file named `handler.py` and follow these steps to build a handler that performs sentiment analysis using a Hugging Face model.
Start by importing the necessary libraries:
```python handler.py theme={"theme":{"light":"github-light","dark":"github-dark"}}
import runpod
from transformers import pipeline
```
The `pipeline` function from the `transformers` library provides a simple interface for using pre-trained models. It handles tokenization, model inference, and post-processing automatically.
Load your model outside the handler function to avoid reloading it on every request. This significantly improves performance by initializing the model only once when the starts:
```python handler.py theme={"theme":{"light":"github-light","dark":"github-dark"}}
# Load model once when worker starts
model = pipeline(
"sentiment-analysis",
model="distilbert/distilbert-base-uncased-finetuned-sst-2-english"
)
```
The `pipeline` function takes two arguments: the task type (like `"sentiment-analysis"`, `"text-generation"`, or `"image-classification"`) and the specific model identifier from the Hugging Face model hub.
Create a handler function that extracts input text from the request, validates it, runs inference, and returns results:
```python handler.py theme={"theme":{"light":"github-light","dark":"github-dark"}}
def handler(job):
# Extract input from the job
job_input = job["input"]
text = job_input.get("text")
# Validate input
if not text:
return {"error": "No text provided for analysis."}
# Run inference
result = model(text)[0]
# Return formatted results
return {
"sentiment": result["label"],
"score": float(result["score"])
}
```
The handler follows Runpod's standard pattern: extract input, validate it, process it, and return results. The model returns a list of predictions, so we take the first result with `[0]` and extract the label and confidence score.
Add this line at the end of your file to register the handler and start the worker:
```python handler.py theme={"theme":{"light":"github-light","dark":"github-dark"}}
runpod.serverless.start({"handler": handler})
```
### Complete implementation
Here's the complete code:
```python handler.py theme={"theme":{"light":"github-light","dark":"github-dark"}}
import runpod
from transformers import pipeline
# Load model once when worker starts
model = pipeline(
"sentiment-analysis",
model="distilbert/distilbert-base-uncased-finetuned-sst-2-english"
)
def handler(job):
# Extract input from the job
job_input = job["input"]
text = job_input.get("text")
# Validate input
if not text:
return {"error": "No text provided for analysis."}
# Run inference
result = model(text)[0]
# Return formatted results
return {
"sentiment": result["label"],
"score": float(result["score"])
}
runpod.serverless.start({"handler": handler})
```
### Test locally
Create a test input file to verify your handler works correctly:
```json test_input.json theme={"theme":{"light":"github-light","dark":"github-dark"}}
{
"input": {
"text": "This is absolutely wonderful and amazing!"
}
}
```
Run your handler locally using the Runpod SDK:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
python handler.py --rp_server_api
```
You should see output indicating successful sentiment analysis:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
--- Starting Serverless Worker | Version 1.6.2 ---
INFO | Using test_input.json as job input.
DEBUG | Retrieved local job: {'input': {'text': 'This is absolutely wonderful and amazing!'}, 'id': 'local_test'}
INFO | local_test | Started.
DEBUG | local_test | Handler output: {'sentiment': 'POSITIVE', 'score': 0.999880313873291}
INFO | Job local_test completed successfully.
```
The first time you run this, Hugging Face will download the model files. Subsequent runs will use the cached model.
### Adapt for other models
This pattern works for any Hugging Face model. To use a different model:
1. **Choose your model**: Browse the [Hugging Face model hub](https://huggingface.co/models) to find a model for your task.
2. **Update the pipeline**: Change the task type and model identifier:
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
# Text generation example
model = pipeline("text-generation", model="gpt2")
# Image classification example
model = pipeline("image-classification", model="google/vit-base-patch16-224")
# Translation example
model = pipeline("translation_en_to_fr", model="t5-base")
```
3. **Adjust input/output handling**: Different models expect different input formats and return different output structures. Check the model's documentation on Hugging Face to understand its API.
## Use cached models
The example above downloads models when workers start, which works fine for development and testing.
For production endpoints, we highly recommend using [cached models](/serverless/endpoints/model-caching) instead. Cached models provide faster cold starts (seconds instead of minutes) and eliminate charges for model download time.
### Enable model caching
To enable cached models on your endpoint:
Navigate to the [Serverless section](https://www.console.runpod.io/serverless) of the console. Either create a new endpoint or select **Manage → Edit Endpoint** on an existing one.
Scroll to the **Model** field and enter your Hugging Face model identifier.
For example: `distilbert/distilbert-base-uncased-finetuned-sst-2-english`
Save your endpoint configuration. Runpod will automatically cache the model and make it available to your workers.
### Locate cached models
Cached models are stored at `/runpod-volume/huggingface-cache/hub/` following Hugging Face cache conventions. Add this helper function to your handler to resolve the correct snapshot path:
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
import os
HF_CACHE_ROOT = "/runpod-volume/huggingface-cache/hub"
def resolve_snapshot_path(model_id: str) -> str:
"""
Resolve the local snapshot path for a cached model.
Args:
model_id: The model name from Hugging Face
(e.g., 'distilbert/distilbert-base-uncased-finetuned-sst-2-english')
Returns:
The full path to the cached model snapshot
"""
if "/" not in model_id:
raise ValueError(f"model_id '{model_id}' must be in 'org/name' format")
org, name = model_id.split("/", 1)
model_root = os.path.join(HF_CACHE_ROOT, f"models--{org}--{name}")
refs_main = os.path.join(model_root, "refs", "main")
snapshots_dir = os.path.join(model_root, "snapshots")
# Read the snapshot hash from refs/main
if os.path.isfile(refs_main):
with open(refs_main, "r") as f:
snapshot_hash = f.read().strip()
candidate = os.path.join(snapshots_dir, snapshot_hash)
if os.path.isdir(candidate):
return candidate
# Fall back to first available snapshot
if os.path.isdir(snapshots_dir):
versions = [
d for d in os.listdir(snapshots_dir)
if os.path.isdir(os.path.join(snapshots_dir, d))
]
if versions:
versions.sort()
return os.path.join(snapshots_dir, versions[0])
raise RuntimeError(f"Cached model not found: {model_id}")
```
### Adapt your handler for cached models
Once model caching is enabled, you need to update your handler to load the model from the local cache instead of downloading it. Here's how the code changes:
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
from transformers import pipeline
# Downloads model when worker starts
model = pipeline(
"sentiment-analysis",
model="distilbert/distilbert-base-uncased-finetuned-sst-2-english"
)
```
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
import os
from transformers import pipeline
# Force offline mode to prevent accidental downloads
os.environ["HF_HUB_OFFLINE"] = "1"
os.environ["TRANSFORMERS_OFFLINE"] = "1"
# Resolve the cached model path
LOCAL_PATH = resolve_snapshot_path(
"distilbert/distilbert-base-uncased-finetuned-sst-2-english"
)
# Load from local cache
model = pipeline(
"sentiment-analysis",
model=LOCAL_PATH,
local_files_only=True
)
```
The key differences are:
* **Offline mode**: Setting `HF_HUB_OFFLINE` and `TRANSFORMERS_OFFLINE` prevents accidental downloads if the model isn't cached.
* **Local path**: Instead of a model identifier, you pass the resolved local path to the cached model files.
* **local\_files\_only**: This flag tells the transformers library to only use local files.
### Complete cached implementation
Here's the complete handler using cached models:
```python handler.py theme={"theme":{"light":"github-light","dark":"github-dark"}}
import os
import runpod
from transformers import pipeline
MODEL_ID = "distilbert/distilbert-base-uncased-finetuned-sst-2-english"
HF_CACHE_ROOT = "/runpod-volume/huggingface-cache/hub"
# Force offline mode to use only cached models
os.environ["HF_HUB_OFFLINE"] = "1"
os.environ["TRANSFORMERS_OFFLINE"] = "1"
def resolve_snapshot_path(model_id: str) -> str:
"""Resolve the local snapshot path for a cached model."""
if "/" not in model_id:
raise ValueError(f"model_id '{model_id}' must be in 'org/name' format")
org, name = model_id.split("/", 1)
model_root = os.path.join(HF_CACHE_ROOT, f"models--{org}--{name}")
refs_main = os.path.join(model_root, "refs", "main")
snapshots_dir = os.path.join(model_root, "snapshots")
if os.path.isfile(refs_main):
with open(refs_main, "r") as f:
snapshot_hash = f.read().strip()
candidate = os.path.join(snapshots_dir, snapshot_hash)
if os.path.isdir(candidate):
return candidate
if os.path.isdir(snapshots_dir):
versions = [
d for d in os.listdir(snapshots_dir)
if os.path.isdir(os.path.join(snapshots_dir, d))
]
if versions:
versions.sort()
return os.path.join(snapshots_dir, versions[0])
raise RuntimeError(f"Cached model not found: {model_id}")
# Load model once when worker starts
LOCAL_PATH = resolve_snapshot_path(MODEL_ID)
model = pipeline("sentiment-analysis", model=LOCAL_PATH, local_files_only=True)
def handler(job):
job_input = job["input"]
text = job_input.get("text")
if not text:
return {"error": "No text provided for analysis."}
result = model(text)[0]
return {
"sentiment": result["label"],
"score": float(result["score"])
}
runpod.serverless.start({"handler": handler})
```
For a complete walkthrough including Dockerfile creation and deployment, see the [cached model tutorial](/tutorials/serverless/model-caching-text).
## Other best practices
When deploying Hugging Face models to production endpoints, keep these additional considerations in mind:
* **Model size**: Larger models require more VRAM and take longer to load. Choose the smallest model that meets your accuracy requirements.
* **GPU utilization**: Most Hugging Face models run faster on GPUs. Ensure your endpoint uses GPU workers for optimal performance.
* **Batch processing**: If your model supports batching, process multiple inputs together to improve throughput.
## Next steps
* Learn more about [how cached models work](/serverless/endpoints/model-caching).
* [Create a Dockerfile](/serverless/workers/create-dockerfile) to package your handler with its dependencies.
* [Deploy your worker](/serverless/workers/deploy) to a Runpod endpoint.
* Explore [optimization techniques](/serverless/development/optimization) to improve performance.
# Local testing
Source: https://docs.runpod.io/serverless/development/local-testing
Test your Serverless handlers locally before deploying to production. Review configuration and operations guidance for Runpod Serverless.
Testing your handler locally before deploying saves time and helps you catch issues early. The Runpod SDK provides multiple ways to test your handler function without consuming cloud resources. See the [Serverless development overview](/serverless/development/overview) for the complete development lifecycle.
## Basic testing
The simplest way to test your handler is by running it directly with test input.
### Inline JSON
Pass test input directly via the command line:
```sh theme={"theme":{"light":"github-light","dark":"github-dark"}}
python handler.py --test_input '{"input": {"prompt": "Hello, world!"}}'
```
This runs your handler with the specified input and displays the output in your terminal.
### Test file
For more complex or reusable test inputs, create a `test_input.json` file in the same directory as your handler:
```json test_input.json theme={"theme":{"light":"github-light","dark":"github-dark"}}
{
"input": {
"prompt": "This is a test input from a JSON file"
}
}
```
Run your handler without any arguments:
```sh theme={"theme":{"light":"github-light","dark":"github-dark"}}
python main.py
```
The SDK automatically detects and uses the `test_input.json` file.
If you provide both a `test_input.json` file and the `--test_input` flag, the command-line input takes precedence.
## Local API server
For more comprehensive testing, start a local API server that simulates your Serverless endpoint. This lets you send HTTP requests to test your handler as if it were deployed.
Start the local server:
```sh theme={"theme":{"light":"github-light","dark":"github-dark"}}
python handler.py --rp_serve_api
```
This starts a FastAPI server on `http://localhost:8000`.
### Send requests to the server
Once your local server is running, send HTTP `POST` requests from another terminal to test your function:
```sh theme={"theme":{"light":"github-light","dark":"github-dark"}}
curl -X POST http://localhost:8000/runsync \
-H "Content-Type: application/json" \
-d '{"input": {"prompt": "Hello, world!"}}'
```
The `/run` endpoint only returns a fake request ID without executing your code, since async mode requires communication with Runpod's system. For local testing, use `/runsync` to execute your handler and get results immediately.
## Testing concurrency
To test how your handler performs under parallel execution, use the `--rp_api_concurrency` flag to set the number of concurrent workers.
This command starts your local server with 4 concurrent workers:
```sh theme={"theme":{"light":"github-light","dark":"github-dark"}}
python main.py --rp_serve_api --rp_api_concurrency 4
```
When using `--rp_api_concurrency` with a value greater than 1, your main file must be named `main.py` for proper FastAPI integration. If your file has a different name, rename it to `main.py` before running with multiple workers.
### Testing concurrent requests
Send multiple requests simultaneously to test concurrency:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
for i in {1..10}; do
curl -X POST http://localhost:8000/runsync \
-H "Content-Type: application/json" \
-d '{"input": {}}' &
done
```
### Handling concurrency in your code
If your handler uses shared state (like global variables), use proper synchronization to avoid race conditions:
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
import runpod
from threading import Lock
counter = 0
counter_lock = Lock()
def handler(event):
global counter
with counter_lock:
counter += 1
return {"counter": counter}
runpod.serverless.start({"handler": handler})
```
## Debugging
### Log levels
Control the verbosity of console output with the `--rp_log_level` flag:
```sh theme={"theme":{"light":"github-light","dark":"github-dark"}}
python handler.py --rp_serve_api --rp_log_level DEBUG
```
Available log levels:
* `ERROR`: Only show error messages.
* `WARN`: Show warnings and errors.
* `INFO`: Show general information, warnings, and errors.
* `DEBUG`: Show all messages, including detailed debug information.
### Enable the debugger
Use the `--rp_debugger` flag for detailed troubleshooting:
```sh theme={"theme":{"light":"github-light","dark":"github-dark"}}
python handler.py --rp_serve_api --rp_debugger
```
This enables the Runpod debugger, which provides additional diagnostic information to help you troubleshoot issues.
## Server configuration
Customize the local API server with these flags:
### Port
Set a custom port (default is 8000):
```sh theme={"theme":{"light":"github-light","dark":"github-dark"}}
python handler.py --rp_serve_api --rp_api_port 8080
```
### Host
Set the hostname (default is "localhost"):
```sh theme={"theme":{"light":"github-light","dark":"github-dark"}}
python handler.py --rp_serve_api --rp_api_host 0.0.0.0
```
Setting `--rp_api_host` to `0.0.0.0` allows connections from other devices on the network. This can be useful for testing but may have security implications.
## Flag reference
Here's a complete reference of all available flags for local testing:
| Flag | Description | Default | Example |
| ---------------------- | --------------------------- | ----------- | ------------------------------ |
| `--rp_serve_api` | Starts the local API server | N/A | `--rp_serve_api` |
| `--rp_api_port` | Sets the server port | 8000 | `--rp_api_port 8080` |
| `--rp_api_host` | Sets the server hostname | "localhost" | `--rp_api_host 0.0.0.0` |
| `--rp_api_concurrency` | Sets concurrent workers | 1 | `--rp_api_concurrency 4` |
| `--rp_log_level` | Controls log verbosity | INFO | `--rp_log_level DEBUG` |
| `--rp_debugger` | Enables the debugger | Disabled | `--rp_debugger` |
| `--test_input` | Provides test input as JSON | N/A | `--test_input '{"input": {}}'` |
## Combined example
You can combine multiple flags to create a customized local testing environment:
```sh theme={"theme":{"light":"github-light","dark":"github-dark"}}
python handler.py --rp_serve_api \
--rp_api_port 8080 \
--rp_api_concurrency 4 \
--rp_log_level DEBUG \
--rp_debugger
```
This command:
* Starts the local API server on port 8080.
* Uses 4 concurrent workers.
* Sets the log level to `DEBUG` for maximum information.
* Enables the debugger for troubleshooting.
# Monitor logs
Source: https://docs.runpod.io/serverless/development/logs
View and access logs for Serverless endpoints and workers. Review setup, configuration, deployment, and operations guidance for Runpod Serverless.
Runpod provides comprehensive logging capabilities for Serverless endpoints and workers. You can view real-time and historical logs through the Runpod console to help you monitor, debug, and troubleshoot your applications.
To learn how to write structured logs from your handler functions, see [Write logs](/serverless/development/write-logs).
## Endpoint logs
Endpoint logs are retained for 90 days, after which they are automatically removed from the system. If you need to retain logs indefinitely, you can [write them to a network volume](#persistent-log-storage) or an external service.
Endpoint logs are automatically collected from your worker instances and streamed to Runpod's centralized logging system. These logs include:
* **Standard output (stdout)** from your handler functions.
* **Standard error (stderr)** from your applications.
* **System messages** related to worker lifecycle events.
* **Framework logs** from the Runpod SDK.
To view endpoint logs, navigate to your Serverless endpoint in the [Runpod console](https://console.runpod.io/serverless) and click the **Logs** tab.
If your worker generates excessive output, logs may be throttled and dropped to prevent system overload. See [Log throttling](#log-throttling) for more information.
### Real-time streaming
The Logs tab streams new log entries automatically — no manual page refresh needed. Logs appear within a few seconds of being emitted by the worker.
To scroll back through historical logs without the view jumping to new entries, scroll up in the log viewer. The stream continues in the background and new entries accumulate at the bottom.
### Request filtering
To view logs for a specific job, use the request filter:
1. Click the **Filter** field at the top of the Logs tab.
2. Enter the job ID you want to inspect.
3. The log view updates to show only entries associated with that request.
Filtering by request is useful when debugging a specific failed job in a high-traffic endpoint where logs from many requests are interleaved.
To clear the filter and return to the full log stream, remove the value from the filter field.
### View in context
**View in context** lets you jump to the surrounding log lines for a specific entry. This is useful for seeing what happened immediately before and after an event of interest, such as an error or an unexpected output.
To use View in context:
1. Find the log entry you want to investigate.
2. Click **View in context** next to the entry.
3. The log view scrolls to and highlights that entry with the surrounding lines visible.
## Worker logs
Worker logs are temporary logs that exist only on the specific server where the worker is running. These logs are not throttled, but are not persistent, and are removed when a worker terminates.
To view worker logs:
1. Navigate to your Serverless endpoint in the [Runpod console](https://console.runpod.io/serverless).
2. Click on the **Workers** tab.
3. Click on a worker to view its logs and request history.
4. Use the search and filtering capabilities to find specific log entries.
5. Download logs as text files for offline analysis.
## Stream output to clients
To send progress updates or stream results to clients during job execution, see [Progress updates](/serverless/workers/handler-functions#progress-updates) and [Streaming handlers](/serverless/workers/handler-functions#streaming-handlers).
## Troubleshooting
### Missing logs
If logs are not appearing in the Logs tab:
1. **Check log throttling**: Excessive logging may trigger throttling.
2. **Verify output streams**: Ensure you're writing to stdout/stderr.
3. **Check worker status**: Logs only appear for successfully initialized workers.
4. **Review retention period**: Logs older than 90 days are automatically removed.
### Log throttling
To avoid log throttling, follow these best practices:
1. **Reduce log verbosity** in production environments.
2. **Use structured logging** to make logs more efficient.
3. **Implement log sampling** for high-frequency events.
4. **Store detailed logs** in network volumes instead of console output.
# Optimize your endpoints
Source: https://docs.runpod.io/serverless/development/optimization
Implement strategies to reduce latency and cost for your Serverless endpoints. Review configuration and operations guidance for Runpod Serverless.
Optimization involves measuring performance with [benchmarking](/serverless/development/benchmarking), identifying bottlenecks, and tuning your [endpoint configurations](/serverless/endpoints/endpoint-configurations).
## Quick optimization checklist
| Strategy | Impact | When to use |
| ---------------------------------------------------------------------------------------------- | -------------------------- | ----------------------------------- |
| [Use cached models](/serverless/endpoints/model-caching) | ⬇️ Cold start (major) | Models on Hugging Face |
| [Bake models into image](/serverless/workers/create-dockerfile#including-models-and-files) | ⬇️ Cold start | Private models |
| [Cache files to a network volume](/serverless/development/volume-cache) | ⬇️ Cold start | Downloaded weights, attached volume |
| [Set active workers > 0](/serverless/endpoints/endpoint-configurations#active-workers) | ⬇️ Cold start (eliminates) | Latency-sensitive apps |
| [Select multiple GPU types](/serverless/endpoints/endpoint-configurations#gpu-configuration) | ⬆️ Availability | Production workloads |
| [Increase max workers](/serverless/endpoints/endpoint-configurations#max-workers) | ⬆️ Throughput | High concurrency |
| [Lower queue delay threshold](/serverless/endpoints/endpoint-configurations#auto-scaling-type) | ⬇️ Response time | Traffic spikes |
## Understanding delay time
Two metrics affect request response time:
| Metric | Description | Optimization |
| ------------------ | ------------------------------------------ | -------------------------------- |
| **Delay time** | Waiting for a worker (includes cold start) | Model caching, active workers |
| **Execution time** | GPU processing the request | Code optimization, GPU selection |
**Delay time** breaks down into:
* **Initialization time**: Downloading Docker image
* **Cold start time**: Loading model into GPU memory
Use [benchmarking](/serverless/development/benchmarking) to measure these metrics for your workload.
If cold start exceeds 7 minutes, the worker is marked unhealthy. Extend with `RUNPOD_INIT_TIMEOUT=800` (seconds).
## Reduce cold starts
### Use cached models (recommended)
For models on Hugging Face, [cached models](/serverless/endpoints/model-caching) provide the fastest cold starts and lowest cost.
### Bake models into images
For private models, [embed them in your Docker image](/serverless/workers/create-dockerfile#including-models-and-files). Models load from high-speed local NVMe storage instead of downloading at runtime.
### Maintain active workers
Set [active workers](/serverless/endpoints/endpoint-configurations#active-workers) > 0 to eliminate cold starts entirely.
**Formula**: `Active workers = (Requests/min × Request duration in seconds) / 60`
Example: 6 requests/min × 30 seconds = 3 active workers needed.
## Improve availability
### Select multiple GPU types
Specify multiple [GPU types](/references/gpu-types) in priority order. A single high-end GPU often outperforms multiple lower-tier cards for .
For endpoints with five or more workers, Runpod [distributes workers across your GPU priorities](/serverless/endpoints/endpoint-configurations#gpu-priority-and-worker-distribution) to reduce throttling when your primary GPU type is constrained.
### Add headroom to max workers
Set [max workers](/serverless/endpoints/endpoint-configurations#max-workers) \~20% above expected concurrency to handle load spikes without throttling.
### Tune auto-scaling
Lower the [queue delay threshold](/serverless/endpoints/endpoint-configurations#auto-scaling-type) to 2-3 seconds (default: 4) for faster worker provisioning.
## Architecture considerations
| Choice | Tradeoff |
| ---------------------- | ------------------------------------------------ |
| **Baked models** | Fastest loading, but larger images |
| **Network volumes** | Flexible, but restricts to specific data centers |
| **Multiple GPU types** | Higher availability, variable performance |
# Serverless development
Source: https://docs.runpod.io/serverless/development/overview
Test, debug, and optimize your Serverless applications. Review setup, configuration, deployment, and operations guidance for Runpod Serverless.
When developing for Runpod Serverless, you'll typically start by writing handler functions, test them locally, and then deploy to production. This guide introduces the development workflow and tools that help you test, debug, and optimize your Serverless applications effectively.
## Development lifecycle
The typical workflow starts with writing your handler function. Your handler receives an event object with input data and returns a response. Once you have a handler function, test it locally using the Runpod SDK's testing environment. You can test with inline JSON inputs, use a local API server, or simulate concurrency, all without actually deploying your code and incurring charges.
When your handler is working correctly, package it into a Docker image and deploy it to a Serverless endpoint. Your worker will auto-scale based on demand. Once deployed, use logs, metrics, and SSH access to troubleshoot issues and optimize performance in production.
```mermaid theme={"theme":{"light":"github-light","dark":"github-dark"}}
%%{init: {'theme':'base', 'themeVariables': { 'primaryColor':'#9289FE','primaryTextColor':'#fff','primaryBorderColor':'#9289FE','lineColor':'#5F4CFE','secondaryColor':'#AE6DFF','tertiaryColor':'#FCB1FF','edgeLabelBackground':'#5F4CFE', 'fontSize':'15px','fontFamily':'font-inter'}}}%%
flowchart TD
Start([Write handler function]) --> Test[Test handler locally with the Runpod SDK]
Test --> Check{Tests pass?}
Check -->|" No "| Fix[Fix code & debug]
Fix --> Test
Check -->|" Yes "| Package[Package worker as a Docker image]
Package --> Deploy[Deploy worker image to Runpod Serverless]
subgraph Production [Production Environment]
Deploy --> Running[Workers auto-scale based on demand]
Running --> Monitor[Monitor logs and metrics]
end
Monitor -.-> Start
%% Applied Color Logic from Diagram 2
style Start fill:#5F4CFE,stroke:#5F4CFE,color:#FFFFFF,stroke-width:2px
style Test fill:#4D38F5,stroke:#4D38F5,color:#FFFFFF,stroke-width:2px
style Check fill:#f87171,stroke:#f87171,color:#000000,stroke-width:2px
style Fix fill:#ecc94b,stroke:#ecc94b,color:#000000,stroke-width:2px
style Package fill:#5F4CFE,stroke:#5F4CFE,color:#FFFFFF,stroke-width:2px
style Deploy fill:#5F4CFE,stroke:#5F4CFE,color:#FFFFFF,stroke-width:2px
style Running fill:#5F4CFE,stroke:#5F4CFE,color:#FFFFFF,stroke-width:2px
style Monitor fill:#ecc94b,stroke:#ecc94b,color:#000000,stroke-width:2px
%% Subgraph Styling
style Production fill:#1B0656,stroke:#5F4CFE,color:#FFFFFF,stroke-dasharray: 5 5
linkStyle default stroke-width:2px,stroke:#5F4CFE
```
For faster iteration and debugging of GPU-intensive applications, you can develop on a Pod first before deploying to Serverless. This "Pod-first" workflow gives you direct access to the GPU environment with tools like Jupyter Notebooks and SSH, letting you iterate faster than deploying repeatedly to Serverless. Learn more in [Pod-first development](/serverless/development/dual-mode-worker).
## Local testing
The Runpod SDK provides a comprehensive local testing environment:
* **Basic testing**: Run your handler with inline JSON or test files.
* **Local API server**: Simulate HTTP requests to your Serverless endpoint.
* **Concurrency testing**: Test how your handler performs under parallel execution.
* **Debug mode**: Enable detailed logging and troubleshooting output.
Learn more in [Local testing](/serverless/development/local-testing).
## Error handling
Implement robust error handling to ensure your workers remain stable and return useful error messages.
Learn more in [Error handling](/serverless/workers/handler-functions#error-handling).
## SDK utilities
The Runpod SDK includes helper functions to make your handlers more robust:
* **Input validation**: Validate request data against a schema.
* **Cleanup utilities**: Automatically remove temporary files after processing.
Learn more in [Validate inputs](/serverless/development/validation) and [Clean up files](/serverless/development/cleanup).
## Benchmarking and optimization
Optimize your workers for performance and cost:
* **Benchmark response times**: Measure cold start and execution time.
* **Optimize your workers**: Reduce startup and execution times.
Learn more in the [Benchmarking](/serverless/development/benchmarking) and [Optimization](/serverless/development/optimization) guides.
## Pod-first development
For faster iteration and debugging of GPU-intensive applications, develop on a Pod first, then deploy the same Docker image to Serverless. This workflow provides:
* Interactive development with Jupyter Notebooks.
* Direct SSH access to the GPU environment.
* Faster iteration compared to deploying repeatedly to Serverless.
Learn more in [Pod-first development](/serverless/development/dual-mode-worker).
## Debugging and observability
Runpod provides several tools for debugging and monitoring:
* **Logs**: View real-time and historical logs from your workers.
* **Metrics**: Monitor execution time, delay time, and resource usage.
* **SSH access**: Connect directly to running workers for live debugging.
Learn more in [Logs and monitoring](/serverless/development/logs) and [Connect to workers with SSH](/serverless/development/ssh-into-workers).
## Environment variables
Use to configure your workers without hardcoding credentials or settings in your code. Environment variables are set in the Runpod console and are available to your handler at runtime.
Learn more in [Environment variables](/serverless/development/environment-variables).
# Connect to workers with SSH
Source: https://docs.runpod.io/serverless/development/ssh-into-workers
SSH into running workers for debugging and troubleshooting. Review setup, configuration, deployment, and operations guidance for Runpod Serverless.
You can connect directly to running workers via SSH for debugging and troubleshooting. By connecting to a worker, you can inspect logs, file systems, and environment variables in real-time.
## Generate an SSH key and add it to your Runpod account
Before you can connect to a worker, you'll need to generate an SSH key and add it to your Runpod account.
Run this command on your local terminal to generate an SSH key, replacing `YOUR_EMAIL@DOMAIN.COM` with your actual email:
```sh theme={"theme":{"light":"github-light","dark":"github-dark"}}
ssh-keygen -t ed25519 -C "YOUR_EMAIL@DOMAIN.COM"
```
This saves a public/private key pair on your local machine to `~/.ssh/id_ed25519.pub` and `~/.ssh/id_ed25519` respectively.
If you are using Command Prompt on Windows instead of the Linux terminal or WSL, your public and private key pair will be saved to `C:\Users\YOUR_USER_ACCOUNT\.ssh\id_ed25519.pub` and `C:\Users\YOUR_USER_ACCOUNT\.ssh\id_ed25519`, respectively.
Run this command on your local terminal to retrieve the public SSH key you just generated:
```sh theme={"theme":{"light":"github-light","dark":"github-dark"}}
cat ~/.ssh/id_ed25519.pub
```
This will output something similar to this:
```sh theme={"theme":{"light":"github-light","dark":"github-dark"}}
ssh-ed25519 AAAAC4NzaC1lZDI1JTE5AAAAIGP+L8hnjIcBqUb8NRrDiC32FuJBvRA0m8jLShzgq6BQ YOUR_EMAIL@DOMAIN.COM
```
Copy and paste your public key from the previous step into the **SSH Public Keys** field in your [Runpod user account settings](https://www.console.runpod.io/user/settings).
If you need to add multiple SSH keys to your Runpod account, make sure that each key pair is on its own line in the **SSH Public Keys** field.
## SSH into a worker
Before you can connect, you need at least one worker running. To guarantee a worker is available:
1. Navigate to the [Serverless section](https://www.console.runpod.io/serverless) of the Runpod console.
2. Select your endpoint from the list.
3. Go to the **Configuration** tab.
4. Under **Worker configuration**, set **Active workers** to 1 or more.
5. Click **Save** to apply the changes.
This ensures at least one worker remains running at all times, and allowing you to SSH in without your worker being automatically scaled down.
Select the **Workers** tab in your endpoint's details page to view all running workers for this endpoint.
Here you'll see a list of all workers associated with your endpoint. Find a worker with a status of **Running** and click on it to open its detail pane.
In the worker's detail pane:
1. Select the **Connect** tab.
2. Under the **SSH** section, copy the provided SSH command.
The command will look similar to this:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
ssh root@worker-id-xyz -i ~/.ssh/id_ed25519
```
If you saved your SSH key to a custom location, update the path after the `-i` flag to match your key's location.
Open your local terminal and paste the SSH command you copied. Press Enter to connect to the worker.
Once connected, you can:
* Inspect logs and debug output.
* Check environment variables with `env`.
* Verify file systems and mounted volumes.
* Test your worker's behavior in the production environment.
* Run diagnostic commands to troubleshoot issues.
## Troubleshooting SSH key authentication
If you're asked for a password when connecting to your worker via SSH, this means something is not set up correctly. Runpod does not require a password for SSH connections, as authentication is handled entirely through your SSH key pair.
Here are some common reasons why this might happen:
* If you copy and paste the key *fingerprint* (which starts with `SHA256:`) into your Runpod user settings instead of the actual public key (the contents of your `id_ed25519.pub` file), authentication will fail.
* If you omit the encryption type at the beginning of your public key when pasting it into your Runpod user settings (for example, leaving out `ssh-ed25519`), the key will not be recognized.
* If you add multiple public keys to your Runpod user settings but do not separate them with a newline, only the first key will work. Each key must be on its own line.
* If you specify the wrong file path to your private key when connecting, SSH will not be able to find the correct key (`No such file or directory` error).
* If your private key file is accessible by other users on your machine, SSH may refuse to use it for security reasons (`bad permissions` error).
* If your SSH configuration file (`~/.ssh/config`) points to the wrong private key, you will also be prompted for a password. Make sure the `IdentityFile` entry in your config file matches the private key that corresponds to the public key you added to your Runpod account.
# Validate inputs
Source: https://docs.runpod.io/serverless/development/validation
Validate handler inputs using the Runpod SDK schema validator. Review setup, configuration, deployment, and operations guidance for Runpod Serverless.
The Runpod SDK includes a built-in validation utility that ensures your handler receives data in the correct format before processing begins. Validating inputs early helps catch errors immediately and prevents your worker from crashing due to unexpected or malformed data types.
## Import the validator
To use the validation features, import the `validate` function from the utils module:
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
from runpod.serverless.utils.rp_validator import validate
```
## Define a schema
You define your validation rules using a dictionary where each key represents an expected input field. This schema dictates the data types, necessity, and constraints for the incoming data.
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
schema = {
"text": {
"type": str,
"required": True,
},
"max_length": {
"type": int,
"required": False,
"default": 100,
"constraints": lambda x: x > 0,
},
}
```
The schema supports several configuration keys:
* `type` (required): Expected input type (e.g., `str`, `int`, `float`, `bool`).
* `required` (default: `False`): Whether the field is required.
* `default` (default: `None`): Default value if input is not provided.
* `constraints` (optional): A lambda function that returns `True` or `False` to validate the value.
## Validate input in your handler
When implementing validation in your handler, pass the input object and your schema to the `validate` function. The function returns a dictionary containing either an `errors` key or a `validated_input` key.
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
import runpod
from runpod.serverless.utils.rp_validator import validate
schema = {
"text": {
"type": str,
"required": True,
},
"max_length": {
"type": int,
"required": False,
"default": 100,
"constraints": lambda x: x > 0,
},
}
def handler(event):
try:
# Validate the input against the schema
validated_input = validate(event["input"], schema)
# Check for validation errors
if "errors" in validated_input:
return {"error": validated_input["errors"]}
# Access the sanitized inputs
text = validated_input["validated_input"]["text"]
max_length = validated_input["validated_input"]["max_length"]
result = text[:max_length]
return {"output": result}
except Exception as e:
return {"error": str(e)}
runpod.serverless.start({"handler": handler})
```
## Test the validator
You can test your validation logic locally without deploying. Save your handler code and run it via the command line with the `--test_input` flag.
```sh theme={"theme":{"light":"github-light","dark":"github-dark"}}
python your_handler.py --test_input '{"input": {"text": "Hello, world!", "max_length": 5}}'
```
Alternatively, you can define your test case in a JSON file and pass it to the handler to simulate a real request.
```json test_input.json theme={"theme":{"light":"github-light","dark":"github-dark"}}
{
"input": {
"text": "The quick brown fox jumps over the lazy dog",
"max_length": 50
}
}
```
# Cache files to a network volume
Source: https://docs.runpod.io/serverless/development/volume-cache
Persist model weights and other files to an attached network volume to speed up Serverless worker cold starts.
The Runpod Python SDK includes `VolumeCache`, a helper that warms local directories across Serverless workers using an attached network volume. It keeps a browsable mirror of your cache directories on the volume: on cold start it restores previously cached files into place, and after your code runs it copies newly written files back. This turns a repeated multi-GB model download on every cold start into a one-time cost per endpoint, which reduces times.
`VolumeCache` is best-effort and self-contained, and it never affects the outcome of a job. If any part of the cache fails, the worker falls back to a normal cold start. You add it explicitly by wrapping the code that populates your cache, so nothing runs until you opt in.
## Requirements
* A [network volume](/storage/network-volumes#network-volumes-for-serverless) attached to your endpoint, mounted at `/runpod-volume`. The mirror is stored on the volume, so every operation is a safe no-op when no volume is mounted.
* The Runpod Python SDK installed in your worker image.
* An endpoint namespace to scope the mirror. On Serverless this defaults to `RUNPOD_ENDPOINT_ID`, which Runpod sets automatically.
## Usage
`VolumeCache` is a context manager. Wrapping it around your model load hydrates the cache before the block runs and syncs any changes back afterward:
```python title="handler.py" theme={"theme":{"light":"github-light","dark":"github-dark"}}
import runpod
from runpod.serverless import VolumeCache
def handler(job):
...
# Hydrate from the volume, download only what's missing, then sync back
with VolumeCache(dirs=["/root/.cache/huggingface"]):
model = load_model() # downloads land in the cached directory
runpod.serverless.start({"handler": handler})
```
When you enter the block, `hydrate()` copies files that are missing or newer on the volume mirror into the container. When you exit, `sync()` copies files that are missing or newer in the container back onto the mirror. By default the sync runs on a background daemon thread and returns immediately, so the `with` block doesn't wait on it. A process-exit hook completes any outstanding syncs, so short-lived processes still finish syncing before they exit.
You can also call the phases directly when they happen at different points in your worker's lifecycle:
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
vc = VolumeCache(dirs=["/data/models"], namespace="my-model-cache")
vc.hydrate() # restore cached files (for example, at startup)
model = load_model() # populate the cache
vc.sync(background=False) # persist new files back to the volume, inline
```
## Constructor
`VolumeCache` accepts the following arguments:
| Argument | Default | Description |
| ------------- | ------------------------------------ | ---------------------------------------------------------------------------------------------------------------------- |
| `dirs` | Required | A list of local directories to cache. |
| `namespace` | `RUNPOD_ENDPOINT_ID` | Isolation key for the on-volume mirror. Must be a single safe path component. |
| `volume_path` | `/runpod-volume` | Mount point of the network volume. |
| `best_effort` | `True` | When `True`, cache errors are logged and swallowed instead of raised. Set to `False` while debugging. |
| `max_workers` | `min(32, (os.cpu_count() or 4) * 4)` | Thread count for the parallel copy of large files. The work is I/O-bound, so the default oversubscribes the CPU count. |
## How it works
`VolumeCache` adapts its transport to the size of your files, which keeps both many-small-file caches and multi-GB weight files fast.
* **Size-bucketed mirror:** Cached files live at `{volume_path}/.cache/{namespace}` on the volume. Files below 256 KiB are packed into a single `small.tar` archive, which collapses the per-file metadata round-trips that make many small files slow on a network volume. Larger files are copied unpacked into a `big/` subdirectory, which preserves their original relative paths so the large-file subtree stays browsable. A versioned `manifest.json`, written last, records the size and modification time of every cached file and marks the mirror as complete. A mirror without a valid, current manifest is treated as absent, so a sync self-heals.
* **Incremental large files, whole-archive small files:** Large-file transfers are diffed per file against the manifest, so unchanged files are skipped. The `small.tar` archive is repacked whole whenever any small file changes, since unpacking and re-diffing many tiny files individually is slower than the volume's per-file overhead.
* **Parallel copy:** Large-file transfers run across a thread pool sized by `max_workers`. The work is I/O-bound, so the default oversubscribes the CPU count.
* **Safety:** Symbolic links are never followed or copied. Every archive member and every large-file destination is checked to resolve inside one of your configured `dirs` before it is written, so a mirror entry can't write outside your cached directories.
## Limitations
* **Concurrent cold-start write amplification:** If several workers cold-start at the same time, each may miss the still-empty mirror, download the model, and sync a full copy back. There's no coordination between concurrent syncs, so the mirror reflects whichever worker synced most recently.
* **Background sync on short-lived processes:** `sync()` schedules the copy on a daemon thread. If the process exits without a normal interpreter shutdown (for example, `os._exit` or `SIGKILL`), the exit hook never runs and the sync may not complete.
* **Orphaned large files aren't pruned:** If you delete or rename a large file locally, its copy under `big/` stays on the volume. Hydration is manifest-driven and ignores it, but volume usage grows as you swap model versions.
## Next steps
* [Optimize your endpoints](/serverless/development/optimization): Combine the volume cache with other strategies to reduce cold starts.
* [Cached models](/serverless/endpoints/model-caching): Use Runpod's platform-level model cache for Hugging Face models.
* [Storage options](/serverless/storage/overview): Compare container disks, network volumes, and S3-compatible storage.
# Write logs
Source: https://docs.runpod.io/serverless/development/write-logs
Write application logs from your handler functions to the console or persistent storage. See Runpod Serverless setup and usage details.
Writing logs from your handler functions helps you debug, monitor, and troubleshoot your Serverless applications. You can write logs to the Runpod console for real-time monitoring or to persistent storage for long-term retention.
## Logging levels
Runpod supports standard logging levels to help you control the verbosity and importance of log messages. Using appropriate logging levels makes it easier to filter and analyze logs.
The logging levels available are:
* **DEBUG**: Detailed information, typically of interest only when diagnosing problems.
* **INFO**: Confirmation that things are working as expected.
* **WARNING**: Used for unexpected events or warnings of problems in the near future (e.g., `disk space low`).
* **ERROR**: Used for more serious problems, where the application has not been able to perform some function.
* **FATAL**: Used for very serious errors, indicating that the program itself may be unable to continue running.
## Writing logs to the console
The easiest way to write logs is using Python's `logging` library. Logs written to stdout or stderr are automatically captured by Runpod and displayed in the console.
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
import logging
import os
import runpod
import logging.handlers
def setup_logger(log_level=logging.DEBUG):
"""
Configures and returns a logger that writes to the console.
This function should be called once when the worker initializes.
"""
# Define the format for log messages. We include a placeholder for 'request_id'
# which will be added contextually for each job.
log_format = logging.Formatter(
'%(asctime)s - %(levelname)s - [Request: %(request_id)s] - %(message)s',
datefmt='%Y-%m-%d %H:%M:%S'
)
# Get the root logger
logger = logging.getLogger("runpod_worker")
logger.setLevel(log_level)
# --- Console Handler ---
# This handler sends logs to standard output, which Runpod captures as worker logs.
console_handler = logging.StreamHandler()
console_handler.setFormatter(log_format)
# Add the console handler to the logger
# Check if handlers are already added to avoid duplication on hot reloads
if not logger.handlers:
logger.addHandler(console_handler)
return logger
# --- Global Logger Initialization ---
# Set up the logger when the script is first loaded by the worker.
logger = setup_logger(log_level=logging.DEBUG)
logger = logging.LoggerAdapter(logger, {"request_id": "N/A"})
logger.info("Logger initialized. Ready to process jobs.")
def handler(job):
"""
Main handler function for the Serverless worker.
"""
# Extract the request ID from the job payload for traceability.
request_id = job.get('id', 'unknown')
# Create a new logger adapter for this specific job.
job_logger = logging.LoggerAdapter(logging.getLogger("runpod_worker"), {"request_id": request_id})
job_logger.info(f"Received job. Now demonstrating all log levels.")
try:
# Demonstrate all log levels
job_logger.debug("Debug message for detailed diagnostics.")
job_logger.info("Info message for general execution flow.")
job_logger.warning("Warning message for unexpected events.")
job_logger.error("Error message for serious issues.")
job_logger.critical("Critical message for unrecoverable issues.")
result = "Successfully demonstrated all log levels."
job_logger.info(f"Job completed successfully.")
return {"output": result}
except Exception as e:
job_logger.error(f"Job failed with an unexpected exception.", exc_info=True)
return {"error": f"An unexpected error occurred: {str(e)}"}
# Start the Serverless worker
if __name__ == "__main__":
runpod.serverless.start({"handler": handler})
```
## Persistent log storage
Endpoint logs are retained for 90 days, after which they are automatically removed. Worker logs are removed when a worker terminates. If you need to retain logs beyond these periods, you can write logs to a [network volume](/storage/network-volumes) or an external service like Elasticsearch or Datadog.
### Writing logs to a network volume
Write logs to a network volume attached to your endpoint for long-term retention.
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
import logging
import os
import runpod
import logging.handlers
def setup_logger(log_dir="/runpod-volume/logs", log_level=logging.DEBUG):
"""
Configures a logger that writes to both console and a network volume.
"""
# Ensure the log directory exists on the network volume
os.makedirs(log_dir, exist_ok=True)
log_format = logging.Formatter(
'%(asctime)s - %(levelname)s - [Request: %(request_id)s] - %(message)s',
datefmt='%Y-%m-%d %H:%M:%S'
)
logger = logging.getLogger("runpod_worker")
logger.setLevel(log_level)
# Console Handler
console_handler = logging.StreamHandler()
console_handler.setFormatter(log_format)
# File Handler - writes to network volume
log_file_path = os.path.join(log_dir, "worker.log")
file_handler = logging.FileHandler(log_file_path)
file_handler.setFormatter(log_format)
# Add both handlers
if not logger.handlers:
logger.addHandler(console_handler)
logger.addHandler(file_handler)
return logger
logger = setup_logger(log_level=logging.DEBUG)
logger = logging.LoggerAdapter(logger, {"request_id": "N/A"})
logger.info("Logger initialized with persistent storage.")
def handler(job):
"""
Main handler function with persistent logging.
"""
request_id = job.get('id', 'unknown')
job_logger = logging.LoggerAdapter(logging.getLogger("runpod_worker"), {"request_id": request_id})
job_logger.info(f"Received job.")
try:
job_logger.debug("Processing request with persistent logs.")
result = "Job completed with logs saved to network volume."
job_logger.info(f"Job completed successfully.")
return {"output": result}
except Exception as e:
job_logger.error(f"Job failed.", exc_info=True)
return {"error": f"An error occurred: {str(e)}"}
if __name__ == "__main__":
runpod.serverless.start({"handler": handler})
```
### Accessing stored logs
To access logs stored in network volumes:
* Use the [S3-compatible API](/storage/s3-api) to programmatically access log files.
* Connect to a Pod with the same network volume attached using [SSH](/pods/configuration/use-ssh).
## Structured logging
Structured logging outputs logs in a machine-readable format (typically JSON) that makes it easier to parse, search, and analyze logs programmatically. This is especially useful when exporting logs to external services or analyzing large volumes of logs.
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
import logging
import json
import runpod
def setup_structured_logger():
"""
Configure a logger that outputs JSON-formatted logs.
"""
logger = logging.getLogger("runpod_worker")
logger.setLevel(logging.DEBUG)
handler = logging.StreamHandler()
logger.addHandler(handler)
return logger
logger = setup_structured_logger()
def log_json(level, message, **kwargs):
"""
Log a structured JSON message.
"""
log_entry = {
"level": level,
"message": message,
**kwargs
}
print(json.dumps(log_entry))
def handler(event):
request_id = event.get("id", "unknown")
try:
log_json("INFO", "Processing request", request_id=request_id, input_keys=list(event.get("input", {}).keys()))
# Replace with your processing logic
result = process_input(event["input"])
log_json("INFO", "Request completed", request_id=request_id, execution_time_ms=123)
return {"output": result}
except Exception as e:
log_json("ERROR", "Request failed", request_id=request_id, error=str(e), error_type=type(e).__name__)
return {"error": str(e)}
runpod.serverless.start({"handler": handler})
```
This produces logs in this format:
```json theme={"theme":{"light":"github-light","dark":"github-dark"}}
{"level": "INFO", "message": "Processing request", "request_id": "abc123", "input_keys": ["prompt", "max_length"]}
{"level": "INFO", "message": "Request completed", "request_id": "abc123", "execution_time_ms": 123}
```
### Benefits of structured logging
Structured logging provides several advantages:
* **Easier parsing**: JSON logs can be easily parsed by log aggregation tools.
* **Better search**: Search for specific fields like `request_id` or `error_type`.
* **Analytics**: Analyze trends, patterns, and metrics from log data.
* **Integration**: Export to external services like Datadog, Splunk, or Elasticsearch.
## Best practices
Follow these best practices when writing logs:
1. **Use request IDs**: Include the job ID or request ID in log entries for traceability.
2. **Choose appropriate levels**: Use DEBUG for diagnostics, INFO for normal operations, WARNING for potential issues, and ERROR for failures.
3. **Structure your logs**: Use JSON format for easier parsing and analysis.
4. **Implement log rotation**: Rotate log files to prevent disk space issues when using persistent storage.
5. **Avoid excessive logging**: Excessive console logging may trigger throttling. Use persistent storage for detailed logs.
# Endpoint settings
Source: https://docs.runpod.io/serverless/endpoints/endpoint-configurations
Reference guide for all Serverless endpoint settings and parameters. Review configuration and operations guidance for Runpod Serverless.
This guide details the configuration options available for Runpod Serverless endpoints.
Some settings can only be updated after deploying your endpoint. See [Edit an endpoint](/serverless/endpoints/overview#edit-an-endpoint).
## Quick reference
| Setting | Default | Description |
| --------------------- | ------------- | ------------------------------------------ |
| **Active workers** | 0 | Always-on workers (eliminates cold starts) |
| **Max workers** | 3 | Maximum concurrent workers |
| **GPUs per worker** | 1 | GPU count per worker instance |
| **Idle timeout** | 5s | Time before idle worker shuts down |
| **Execution timeout** | 600s (10 min) | Max job duration |
| **Job TTL** | 24h | Total job lifespan in system |
| **FlashBoot** | Enabled | Faster cold starts via state retention |
## General configuration
### Endpoint name
Display name for identifying your endpoint in the console. Does not affect the endpoint ID used for API requests.
### Endpoint type
**Queue-based endpoints** use a built-in queueing system with guaranteed execution and automatic retries. Ideal for async tasks, batch processing, and long-running jobs. Implemented using [handler functions](/serverless/workers/handler-functions).
**Load balancing endpoints** route traffic directly to workers, bypassing the queue. Designed for low-latency applications like real-time or custom REST APIs. See [Load balancing endpoints](/serverless/load-balancing/overview).
### GPU configuration
Determines the hardware tier for your workers. Select multiple GPU categories to create a prioritized fallback list. If your first choice is unavailable, Runpod automatically uses the next option. Selecting multiple types improves availability during high demand.
| **GPU type(s)** | **Memory** | **Cost per second** | **Description** |
| ----------------------- | ---------- | ------------------- | ----------------------------------------------------- |
| A4000, A4500, RTX 4000 | 16 GB | \$0.00016 | The most cost-effective for small models. |
| L4, A5000, 3090 | 24 GB | \$0.00019 | Great for small-to-medium sized inference workloads. |
| 4090 PRO | 24 GB | \$0.00031 | Extreme throughput for small-to-medium models. |
| A6000, A40 | 48 GB | \$0.00034 | A cost-effective option for running big models. |
| L40, L40S, 6000 Ada PRO | 48 GB | \$0.00053 | Extreme inference throughput on LLMs like Llama 3 7B. |
| A100 | 80 GB | \$0.00076 | High throughput GPU, yet still very cost-effective. |
| H100 PRO | 80 GB | \$0.00116 | Extreme throughput for big models. |
| 6000s PRO | 96 GB | \$0.00111 | High throughput for large model inference workloads. |
| H200 PRO | 141 GB | \$0.00155 | Extreme throughput for huge models. |
| B200 | 180 GB | \$0.00240 | Maximum throughput for huge models. |
#### GPU priority and worker distribution
Specify up to three GPU types in priority order when configuring an endpoint. Runpod uses this ranking to distribute workers across available GPUs, improving availability during high demand.
For endpoints with five or more workers, Runpod distributes workers across your selected GPU priorities. Most workers run on your primary GPU type, with fewer assigned to secondary and tertiary selections. This reduces throttling when your primary GPU is constrained.
For endpoints with fewer than five workers, all workers use the highest-priority GPU type available.
## Worker scaling
### Active workers
Minimum number of workers that remain warm and ready at all times. Setting this to 1+ eliminates cold starts. Active workers incur charges continuously, including when idle.
### Max workers
Maximum concurrent instances your endpoint can scale to. Acts as a cost safety limit and concurrency cap. Set \~20% higher than expected max concurrency to handle traffic spikes smoothly.
### GPUs per worker
Number of GPUs assigned to each worker instance. Default is 1. Generally prioritize fewer high-end GPUs over multiple lower-tier GPUs.
### Auto-scaling type
**Queue delay**: Adds workers when requests wait longer than the threshold (default: 4 seconds). Best when slight delays are acceptable for higher utilization.
**Request count**: More aggressive scaling based on pending + active work. Formula: `Math.ceil((requestsInQueue + requestsInProgress) / scalerValue)`. Use scaler value of 1 for max responsiveness. Recommended for LLM workloads or frequent short requests.
## Lifecycle and timeouts
### Idle timeout
How long a worker stays active after completing a request before shutting down. You're billed during idle time, but the worker remains warm for immediate processing. Default: 5 seconds.
### Idle endpoint scale-down
Runpod automatically scales down endpoints that go a long time without any requests, so unused endpoints don't keep consuming your account balance.
* After 3 days with no requests, the endpoint's max workers is reduced to 2 and Runpod sends you an email notification.
* After 7 days with no requests, max workers is set to 0.
This scale-down is automatic and system-driven, and the timer is based on request activity, so any incoming request resets it.
Once an endpoint has been scaled down this way, it stays at its reduced max workers until you raise the value yourself. To use the endpoint again, increase its max workers in the Runpod console. To prevent an endpoint from scaling down in the first place, make sure it continues to receive requests.
### Execution timeout
Maximum duration for a single job. When exceeded, the job fails and the worker stops. Keep enabled to prevent runaway jobs. Default: 600s (10 min). Range: 5s to 7 days.
Configure in **Advanced** settings, or override per-request via `executionTimeout` in the [job policy](/serverless/endpoints/send-requests#execution-policies).
### Job TTL (time-to-live)
Total lifespan of a job in the system. When TTL expires, job data is deleted regardless of state (queued, running, or completed). Default: 24 hours. Range: 10s to 7 days.
The timer starts at submission, not execution. If a job queues for 45 minutes with a 1-hour TTL, only 15 minutes remain for execution.
TTL is a hard limit. If it expires while a job is running, the job is immediately removed and status checks return 404. Set TTL to cover both expected queue time and execution time.
Override per-request via `ttl` in the [job policy](/serverless/endpoints/send-requests#execution-policies).
### Result retention
| Request type | Retention | Notes |
| ----------------- | --------- | ----------------------------------------------------------- |
| Async (`/run`) | 30 min | Retrieve via `/status/{job_id}` |
| Sync (`/runsync`) | 1 min | Returned in response; also available via `/status/{job_id}` |
Results are permanently deleted after retention expires.
## Performance features
### FlashBoot
Reduces cold starts by retaining worker state after spin-down, allowing faster "revival" than fresh boots. Most effective on endpoints with consistent traffic where workers frequently cycle between active and idle.
Both new GPU and CPU endpoints will have FlashBoot enabled by default, and you can edit existing endpoints to enable or disable FlashBoot.
### Model
Select from [cached models](/serverless/endpoints/model-caching) to schedule workers on with model files pre-loaded. Significantly reduces model loading time during initialization.
## Advanced settings
### Data centers
Restrict your endpoint to specific regions. For maximum availability, allow all data centers:restricting decreases the available GPU pool.
### Network volumes
[Network volumes](/storage/network-volumes) provide persistent storage across worker restarts. Tradeoffs: adds network latency and restricts your endpoint to the volume's data center. Use only when you need shared persistence or datasets exceeding container limits.
### CUDA version selection
Ensures workers run on with compatible drivers. Select your required version plus all newer versions, since CUDA is backward compatible and a wider range increases available hardware.
### Expose HTTP/TCP ports
Exposes the worker's public IP and port for direct external communication. Required for persistent connections like WebSockets.
# Job states and metrics
Source: https://docs.runpod.io/serverless/endpoints/job-states
Monitor your endpoints effectively by understanding job states and key metrics. Review configuration and operations guidance for Runpod Serverless.
Understanding job states and metrics is essential for effectively managing your Serverless endpoints. This documentation covers the different states your jobs can be in and the key metrics available to monitor endpoint performance and health.
## Request job states
Understanding job states helps you track the progress of individual and identify where potential issues might occur in your workflow.
* `IN_QUEUE`: The job is waiting in the endpoint queue for an available worker to process it.
* `IN_PROGRESS`: The job has been picked up by a worker and is currently being processed. The request is no longer waiting in the queue.
* `RUNNING`: A worker has picked up the job and is actively processing it.
* `COMPLETED`: The job has finished processing successfully and returned a result.
* `FAILED`: The job encountered an error during execution and did not complete successfully.
* `CANCELLED`: The job was manually cancelled using the `/cancel/job_id` endpoint before completion.
* `TIMED_OUT`: The job either expired before it was picked up by a worker or the worker failed to report back before reaching the timeout threshold.
## Endpoint metrics
You can find endpoint metrics in the **Metrics** tab of the Serverless endpoint details page in the [Runpod web interface](https://www.console.runpod.io/serverless).
* **Requests**: Displays the total number of requests received by your endpoint, along with the number of completed, failed, and retried requests.
* **Execution time**: Displays the P70, P90, and P98 execution times for requests on your endpoint. These percentiles help analyze execution time distribution and identify potential performance bottlenecks.
* **Delay time**: Delay time is the duration a request spends waiting in the queue before it is picked up by a worker. Displays the P70, P90, and P98 delay times for requests on your endpoint. These percentiles help assess whether your endpoint is scaling efficiently.
* **Cold start time**: Cold start time measures how long it takes to wake up a worker. This includes the time needed to start the container, load the model into GPU VRAM, and get the worker ready to process a job. Displays the P70, P90, and P98 cold start times for your endpoint.
* **Cold start count**: Displays the number of cold starts your endpoint has during a given period. The fewer, the better, as fewer cold starts mean faster response times.
* **WebhookRequest responses**: Displays the number of webhook requests sent and their corresponding responses, including success and failure counts.
* **Worker states**: Displays the number of workers that are [running, idle, throttled, etc.](/serverless/workers/overview#worker-states) across the selected time interval.
# Cached models
Source: https://docs.runpod.io/serverless/endpoints/model-caching
Accelerate worker cold starts and reduce costs by using cached models. Review configuration and operations guidance for Runpod Serverless.
To learn how to use cached models with the Hugging Face Transformers library, see [Use Hugging Face models](/serverless/development/huggingface-models#use-cached-models). For a complete end-to-end deployment walkthrough, see the [cached model tutorial](/tutorials/serverless/model-caching-text).
Enabling cached models on your endpoints can reduce times and dramatically reduce the cost for loading large models.
## Why use cached models?
* **Faster cold starts:** Using cached models can reduce times to just a few seconds, even for large models.
* **Reduced costs:** You aren't billed for worker time while your model is being downloaded. This is especially impactful for large models that can take several minutes to load.
* **Accelerated deployment:** You can deploy cached models instantly without waiting for external downloads or transfers.
* **Smaller container images:** By decoupling models from your container image, you can create smaller, more focused images that contain only your application logic.
* **Shared across workers:** Multiple running on the same host can reference the same cached model, eliminating redundant downloads and saving disk space.
## Cached model compatibility
Cached models work with any model hosted on Hugging Face, including:
* **Public models:** Any publicly available model on Hugging Face.
* **Gated models:** Models that require you to accept terms (provide a Hugging Face access token).
* **Private models:** Private models your Hugging Face token has access to.
Cached models aren't suitable if your model is private and not hosted on Hugging Face. In that case, [bake it into your Docker image](/serverless/workers/deploy#including-models-and-external-files) instead.
## How it works
When you select a cached model for your endpoint, Runpod automatically tries to start your workers on hosts that already contain the selected model.
If no cached host are available, the system delays starting your workers until the model is downloaded onto the machine where your workers will run, ensuring you still won't be charged for the download time.
```mermaid theme={"theme":{"light":"github-light","dark":"github-dark"}}
%%{init: {'theme':'base', 'themeVariables': { 'primaryColor':'#9289FE','primaryTextColor':'#fff','primaryBorderColor':'#9289FE','lineColor':'#5F4CFE','secondaryColor':'#AE6DFF','tertiaryColor':'#FCB1FF','edgeLabelBackground':'#5F4CFE', 'fontSize':'15px','fontFamily':'font-inter'}}}%%
flowchart TD
Start([Request received]) --> CheckWorkers{Worker ready?}
CheckWorkers -->|" Yes "| Process[Process request]
CheckWorkers -->|" No "| CheckCache{Cached model host available?}
CheckCache -->|" Yes "| FastStart[Start worker on cached host]
FastStart --> Ready1[Worker ready in seconds]
Ready1 --> Process
CheckCache -->|" No "| WaitForCache[Wait for model download on target host]
WaitForCache --> Ready2[Worker ready after download]
Ready2 --> Process
Process --> Response([Return response])
style Start fill:#5F4CFE,stroke:#5F4CFE,color:#FFFFFF,stroke-width:2px
style Response fill:#5F4CFE,stroke:#5F4CFE,color:#FFFFFF,stroke-width:2px
style CheckWorkers fill:#f87171,stroke:#f87171,color:#000000,stroke-width:2px
style CheckCache fill:#fb923c,stroke:#fb923c,color:#000000,stroke-width:2px
style Process fill:#22C55E,stroke:#22C55E,color:#000000,stroke-width:2px
style FastStart fill:#22C55E,stroke:#22C55E,color:#000000,stroke-width:2px
style Ready1 fill:#22C55E,stroke:#22C55E,color:#000000,stroke-width:2px
style WaitForCache fill:#ecc94b,stroke:#ecc94b,color:#000000,stroke-width:2px
style Ready2 fill:#ecc94b,stroke:#ecc94b,color:#000000,stroke-width:2px
linkStyle default stroke-width:2px,stroke:#5F4CFE
```
## Enable cached models
Follow these steps to select and add a cached model to your endpoint:
Navigate to the [Serverless section](https://www.console.runpod.io/serverless) of the console and click **New Endpoint**. Choose your deployment type (Hugging Face, Docker, GitHub, or Hub).
If you select **Hugging Face**, the model field is the primary input—model caching is pre-configured automatically.
In the **Endpoint Configuration** step, scroll down to **Model** and add the link or path for the model you want to use.
For example, `Qwen/qwen3-32b-awq`.
The console displays the model size and estimated GPU requirements to help you choose the right GPU configuration.
If you're using a gated model, you'll need to enter a [Hugging Face access token](https://huggingface.co/docs/hub/en/security-tokens).
Complete your endpoint configuration and click **Deploy Endpoint** .
You can add a cached model to an existing endpoint by selecting **Manage → Edit Endpoint** in the endpoint details page and updating the **Model** field.
Once it's deployed, your workers will all have access to the cached model for .
## Using cached models in your workers
When using [vLLM workers](/serverless/vllm/overview) or other official Runpod worker images, you can usually just set the **Model** field as shown above (or use the `MODEL_NAME` environment variable), and your workers will automatically use the cached model for inference.
To use cached models with [custom workers](/serverless/quickstart), you'll need to manually locate the cached model path and integrate it into your worker code.
### Where cached models are stored
Cached models are available to your workers at `/runpod-volume/huggingface-cache/hub/` following Hugging Face cache conventions. The directory structure replaces forward slashes (`/`) from the original model name with double dashes (`--`), and includes a version hash subdirectory.
While cached models use the same mount path as network volumes (`/runpod-volume/`), the model loaded from the cache will load significantly faster than the same model loaded from a network volume.
For example, here is how the model `gensyn/qwen2.5-0.5b-instruct` would be stored:
### Locate cached models in your handler
To use a cached model in your , you need to resolve the local path to the model files. The path follows a predictable pattern based on the model identifier:
```
/runpod-volume/huggingface-cache/hub/models--{org}--{name}/snapshots/{hash}/
```
For example, `Qwen/Qwen2.5-0.5B-Instruct` would be stored at:
```
/runpod-volume/huggingface-cache/hub/models--Qwen--Qwen2.5-0.5B-Instruct/snapshots/abc123.../
```
For complete implementation details including a helper function to resolve these paths dynamically, see [Use Hugging Face models](/serverless/development/huggingface-models#use-cached-models).
### Examples and resources
Learn how to adapt your Transformers code to use cached models
End-to-end walkthrough deploying Phi-3 with model caching
Sample worker using cached models for LLM inference
Pre-built workers with automatic cached model support
## Current limitations
* Each endpoint is currently limited to one cached model at a time.
* If a Hugging Face repository contains multiple quantization versions of a model (for example, 4-bit AWQ and 8-bit GPTQ versions), the system currently downloads all quantization versions. The ability to select specific quantizations will be available in a future update.
# Operation reference
Source: https://docs.runpod.io/serverless/endpoints/operation-reference
Detailed API reference for all queue-based endpoint operations. Review configuration and operations guidance for Runpod Serverless.
This reference covers all operations available for queue-based endpoints. For conceptual information and advanced options, see [Send API requests](/serverless/endpoints/send-requests).
## Setup
Before running these examples, install the Runpod SDK:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
# Python
python -m pip install runpod
# JavaScript
npm install --save runpod-sdk
# Go
go get github.com/runpod/go-sdk && go mod tidy
```
Set your [API key](/get-started/api-keys) and endpoint ID as environment variables:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
export RUNPOD_API_KEY="YOUR_API_KEY"
export ENDPOINT_ID="YOUR_ENDPOINT_ID"
```
You can also send requests using standard HTTP libraries like `fetch` (JavaScript) and `requests` (Python).
## /runsync
Synchronous jobs wait for completion and return the complete result in a single response. Best for shorter tasks, interactive applications, and simpler client code without status polling.
* **Maximum payload size**: 20 MB
* **Result retention**: 1 minute after completion
* **Default wait time**: 90 seconds (adjustable via `?wait=x` parameter, 1000-300000 ms)
```sh theme={"theme":{"light":"github-light","dark":"github-dark"}}
https://api.runpod.ai/v2/$ENDPOINT_ID/runsync?wait=120000
```
The `?wait` parameter controls how long the request waits for job completion, not how long results are retained.
```sh theme={"theme":{"light":"github-light","dark":"github-dark"}}
curl --request POST \
--url https://api.runpod.ai/v2/$ENDPOINT_ID/runsync \
-H "accept: application/json" \
-H "authorization: $RUNPOD_API_KEY" \
-H "content-type: application/json" \
-d '{ "input": { "prompt": "Hello, world!" }}'
```
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
import runpod
import os
runpod.api_key = os.getenv("RUNPOD_API_KEY")
endpoint = runpod.Endpoint(os.getenv("ENDPOINT_ID"))
try:
run_request = endpoint.run_sync(
{"prompt": "Hello, world!"},
timeout=60, # Client timeout in seconds
)
print(run_request)
except TimeoutError:
print("Job timed out.")
```
```javascript theme={"theme":{"light":"github-light","dark":"github-dark"}}
const { RUNPOD_API_KEY, ENDPOINT_ID } = process.env;
import runpodSdk from "runpod-sdk";
const runpod = runpodSdk(RUNPOD_API_KEY);
const endpoint = runpod.endpoint(ENDPOINT_ID);
const result = await endpoint.runSync({
"input": {
"prompt": "Hello, World!",
},
timeout: 60000, // Client timeout in milliseconds
});
console.log(result);
```
```go theme={"theme":{"light":"github-light","dark":"github-dark"}}
package main
import (
"encoding/json"
"fmt"
"log"
"os"
"github.com/runpod/go-sdk/pkg/sdk"
"github.com.runpod/go-sdk/pkg/sdk/config"
rpEndpoint "github.com/runpod/go-sdk/pkg/sdk/endpoint"
)
func main() {
apiKey := os.Getenv("RUNPOD_API_KEY")
baseURL := os.Getenv("RUNPOD_BASE_URL")
endpoint, err := rpEndpoint.New(
&config.Config{ApiKey: &apiKey},
&rpEndpoint.Option{EndpointId: &baseURL},
)
if err != nil {
log.Fatalf("Failed to create endpoint: %v", err)
}
jobInput := rpEndpoint.RunSyncInput{
JobInput: &rpEndpoint.JobInput{
Input: map[string]interface{}{
"prompt": "Hello World",
},
},
Timeout: sdk.Int(60), // Client timeout in seconds
}
output, err := endpoint.RunSync(&jobInput)
if err != nil {
panic(err)
}
data, _ := json.Marshal(output)
fmt.Printf("output: %s\n", data)
}
```
**Response:**
```json theme={"theme":{"light":"github-light","dark":"github-dark"}}
{
"delayTime": 824,
"executionTime": 3391,
"id": "sync-79164ff4-d212-44bc-9fe3-389e199a5c15",
"output": [
{
"image": "https://image.url",
"seed": 46578
}
],
"status": "COMPLETED"
}
```
## /run
Asynchronous jobs process in the background and return immediately with a job ID. Best for longer-running tasks, operations requiring significant processing time, and managing multiple concurrent jobs.
* **Maximum payload size**: 10 MB
* **Result retention**: 30 minutes after completion
```sh theme={"theme":{"light":"github-light","dark":"github-dark"}}
curl --request POST \
--url https://api.runpod.ai/v2/$ENDPOINT_ID/run \
-H "accept: application/json" \
-H "authorization: $RUNPOD_API_KEY" \
-H "content-type: application/json" \
-d '{"input": {"prompt": "Hello, world!"}}'
```
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
import runpod
import os
runpod.api_key = os.getenv("RUNPOD_API_KEY")
endpoint = runpod.Endpoint(os.getenv("ENDPOINT_ID"))
# Submit asynchronous job
run_request = endpoint.run({"prompt": "Hello, World!"})
# Check initial status
status = run_request.status()
print(f"Initial job status: {status}")
if status != "COMPLETED":
# Poll for results with timeout
output = run_request.output(timeout=60)
else:
output = run_request.output()
print(f"Job output: {output}")
```
```javascript theme={"theme":{"light":"github-light","dark":"github-dark"}}
const { RUNPOD_API_KEY, ENDPOINT_ID } = process.env;
import runpodSdk from "runpod-sdk";
const runpod = runpodSdk(RUNPOD_API_KEY);
const endpoint = runpod.endpoint(ENDPOINT_ID);
const result = await endpoint.run({
"input": {
"prompt": "Hello, World!",
},
});
console.log(result);
```
```go theme={"theme":{"light":"github-light","dark":"github-dark"}}
package main
import (
"encoding/json"
"fmt"
"log"
"os"
"github.com/runpod/go-sdk/pkg/sdk"
"github.com/runpod/go-sdk/pkg/sdk/config"
rpEndpoint "github.com/runpod/go-sdk/pkg/sdk/endpoint"
)
func main() {
client := sdk.New(&config.Config{
ApiKey: os.Getenv("RUNPOD_API_KEY"),
BaseURL: os.Getenv("RUNPOD_BASE_URL"),
})
endpoint, err := client.NewEndpoint("YOUR_ENDPOINT_ID")
if err != nil {
log.Fatalf("Failed to create endpoint: %v", err)
}
jobInput := rpEndpoint.RunInput{
JobInput: &rpEndpoint.JobInput{
Input: map[string]interface{}{
"prompt": "Hello World",
},
},
RequestTimeout: sdk.Int(120),
}
output, err := endpoint.Run(&jobInput)
if err != nil {
panic(err)
}
data, _ := json.Marshal(output)
fmt.Printf("output: %s\n", data)
}
```
**Response:**
```json theme={"theme":{"light":"github-light","dark":"github-dark"}}
{
"id": "eaebd6e7-6a92-4bb8-a911-f996ac5ea99d",
"status": "IN_QUEUE"
}
```
Retrieve results using the `/status` operation.
## /status
Check the current state, execution statistics, and results of previously submitted jobs.
Configure time-to-live (TTL) for individual jobs by appending `?ttl=x` to the request URL. For example, `?ttl=6000` sets the TTL to 6 seconds.
Replace `YOUR_JOB_ID` with the job ID from your `/run` response.
```sh theme={"theme":{"light":"github-light","dark":"github-dark"}}
curl --request GET \
--url https://api.runpod.ai/v2/$ENDPOINT_ID/status/YOUR_JOB_ID \
-H "authorization: $RUNPOD_API_KEY" \
```
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
import runpod
runpod.api_key = os.getenv("RUNPOD_API_KEY")
endpoint = runpod.Endpoint(os.getenv("ENDPOINT_ID"))
input_payload = {"input": {"prompt": "Hello, World!"}}
run_request = endpoint.run(input_payload)
# Initial check without blocking, useful for quick tasks
status = run_request.status()
print(f"Initial job status: {status}")
if status != "COMPLETED":
# Polling with timeout for long-running tasks
output = run_request.output(timeout=60)
else:
output = run_request.output()
print(f"Job output: {output}")
```
```javascript theme={"theme":{"light":"github-light","dark":"github-dark"}}
const { RUNPOD_API_KEY, ENDPOINT_ID } = process.env;
import runpodSdk from "runpod-sdk";
async function main() {
try {
const runpod = runpodSdk(RUNPOD_API_KEY);
const endpoint = runpod.endpoint(ENDPOINT_ID);
const result = await endpoint.run({
input: {
prompt: "Hello, World!",
},
});
const { id } = result;
if (!id) {
console.error("No ID returned from endpoint.run");
return;
}
const status = await endpoint.status(id);
console.log(status);
} catch (error) {
console.error("An error occurred:", error);
}
}
main();
```
Replace `YOUR_JOB_ID` with the job ID from your `/run` response.
```go theme={"theme":{"light":"github-light","dark":"github-dark"}}
package main
import (
"encoding/json"
"fmt"
"log"
"os"
"github.com/runpod/go-sdk/pkg/sdk"
"github.com/runpod/go-sdk/pkg/sdk/config"
rpEndpoint "github.com/runpod/go-sdk/pkg/sdk/endpoint"
)
func main() {
apiKey := os.Getenv("RUNPOD_API_KEY")
baseURL := os.Getenv("RUNPOD_BASE_URL")
endpoint, err := rpEndpoint.New(
&config.Config{ApiKey: &apiKey},
&rpEndpoint.Option{EndpointId: &baseURL},
)
if err != nil {
log.Fatalf("Failed to create endpoint: %v", err)
}
input := rpEndpoint.StatusInput{
Id: sdk.String("YOUR_JOB_ID"),
}
output, err := endpoint.Status(&input)
if err != nil {
panic(err)
}
dt, _ := json.Marshal(output)
fmt.Printf("output:%s\n", dt)
}
```
**Response:**
Returns job status (`IN_QUEUE`, `IN_PROGRESS`, `COMPLETED`, `FAILED`) with optional `output` field:
```json theme={"theme":{"light":"github-light","dark":"github-dark"}}
{
"delayTime": 31618,
"executionTime": 1437,
"id": "60902e6c-08a1-426e-9cb9-9eaec90f5e2b-u1",
"output": {
"input_tokens": 22,
"output_tokens": 16,
"text": ["Hello! How can I assist you today?\nUSER: I'm having"]
},
"status": "COMPLETED"
}
```
## /stream
Receive incremental results as they become available from jobs that generate output progressively. Best for text generation, long-running jobs where you want to show progress, and large outputs that benefit from incremental processing.
Your handler must support streaming. See [Streaming handlers](/serverless/workers/handler-functions#streaming-handlers) for implementation details.
Replace `YOUR_JOB_ID` with the job ID from your `/run` response.
```sh theme={"theme":{"light":"github-light","dark":"github-dark"}}
curl --request GET \
--url https://api.runpod.ai/v2/$ENDPOINT_ID/stream/YOUR_JOB_ID \
-H "accept: application/json" \
-H "authorization: $RUNPOD_API_KEY" \
```
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
import runpod
runpod.api_key = os.getenv("RUNPOD_API_KEY")
endpoint = runpod.Endpoint(os.getenv("ENDPOINT_ID"))
run_request = endpoint.run(
{
"input": {
"prompt": "Hello, world!",
}
}
)
for output in run_request.stream():
print(output)
```
```javascript theme={"theme":{"light":"github-light","dark":"github-dark"}}
const { RUNPOD_API_KEY, ENDPOINT_ID } = process.env;
import runpodSdk from "runpod-sdk";
async function main() {
const runpod = runpodSdk(RUNPOD_API_KEY);
const endpoint = runpod.endpoint(ENDPOINT_ID);
const result = await endpoint.run({
input: {
prompt: "Hello, World!",
},
});
console.log(result);
const { id } = result;
for await (const result of endpoint.stream(id)) {
console.log(`${JSON.stringify(result, null, 2)}`);
}
console.log("done streaming");
}
main();
```
```go theme={"theme":{"light":"github-light","dark":"github-dark"}}
package main
import (
"encoding/json"
"fmt"
"github.com/runpod/go-sdk/pkg/sdk/config"
rpEndpoint "github.com/runpod/go-sdk/pkg/sdk/endpoint"
)
func main() {
apiKey := os.Getenv("RUNPOD_API_KEY")
baseURL := os.Getenv("RUNPOD_BASE_URL")
endpoint, err := rpEndpoint.New(
&config.Config{ApiKey: &apiKey},
&rpEndpoint.Option{EndpointId: &baseURL},
)
if err != nil {
panic(err)
}
request, err := endpoint.Run(&rpEndpoint.RunInput{
JobInput: &rpEndpoint.JobInput{
Input: map[string]interface{}{
"prompt": "Hello World",
},
},
})
if err != nil {
panic(err)
}
streamChan := make(chan rpEndpoint.StreamResult, 100)
err = endpoint.Stream(&rpEndpoint.StreamInput{Id: request.Id}, streamChan)
if err != nil {
// timeout reached, if we want to get the data that has been streamed
if err.Error() == "ctx timeout reached" {
for data := range streamChan {
dt, _ := json.Marshal(data)
fmt.Printf("output:%s\n", dt)
}
}
panic(err)
}
for data := range streamChan {
dt, _ := json.Marshal(data)
fmt.Printf("output:%s\n", dt)
}
}
```
Maximum size for a single streamed payload chunk is 1 MB. Larger outputs are split across multiple chunks.
**Response:**
```json theme={"theme":{"light":"github-light","dark":"github-dark"}}
[
{
"metrics": {
"avg_gen_throughput": 0,
"avg_prompt_throughput": 0,
"cpu_kv_cache_usage": 0,
"gpu_kv_cache_usage": 0.0016722408026755853,
"input_tokens": 0,
"output_tokens": 1,
"pending": 0,
"running": 1,
"scenario": "stream",
"stream_index": 2,
"swapped": 0
},
"output": {
"input_tokens": 0,
"output_tokens": 1,
"text": [" How"]
}
}
]
```
## /cancel
Stop jobs that are no longer needed or taking too long. Stops in-progress jobs, removes queued jobs before they start, and returns immediately with the canceled status.
Replace `YOUR_JOB_ID` with the job ID from your `/run` response.
```sh theme={"theme":{"light":"github-light","dark":"github-dark"}}
curl --request POST \
--url https://api.runpod.ai/v2/$ENDPOINT_ID/cancel/YOUR_JOB_ID \
-H "authorization: $RUNPOD_API_KEY" \
```
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
import time
import runpod
runpod.api_key = os.getenv("RUNPOD_API_KEY")
endpoint = runpod.Endpoint(os.getenv("ENDPOINT_ID"))
run_request = endpoint.run(
{
"input": {
"prompt": "Hello, world!",
}
}
)
try:
while True:
status = run_request.status()
print(f"Current job status: {status}")
if status == "COMPLETED":
output = run_request.output()
print("Job output:", output)
break
elif status in ["FAILED", "ERROR"]:
print("Job failed to complete successfully.")
break
else:
time.sleep(10)
except KeyboardInterrupt: # Catch KeyboardInterrupt
print("KeyboardInterrupt detected. Canceling the job...")
if run_request: # Check if a job is active
run_request.cancel()
print("Job canceled.")
```
```javascript theme={"theme":{"light":"github-light","dark":"github-dark"}}
const { RUNPOD_API_KEY, ENDPOINT_ID } = process.env;
import runpodSdk from "runpod-sdk";
async function main() {
try {
const runpod = runpodSdk(RUNPOD_API_KEY);
const endpoint = runpod.endpoint(ENDPOINT_ID);
const result = await endpoint.run({
input: {
prompt: "Hello, World!",
},
});
const { id } = result;
if (!id) {
console.error("No ID returned from endpoint.run");
return;
}
const cancel = await endpoint.cancel(id);
console.log(cancel);
} catch (error) {
console.error("An error occurred:", error);
}
}
main();
```
```go theme={"theme":{"light":"github-light","dark":"github-dark"}}
package main
import (
"encoding/json"
"fmt"
"github.com/runpod/go-sdk/pkg/sdk"
"github.com/runpod/go-sdk/pkg/sdk/config"
rpEndpoint "github.com/runpod/go-sdk/pkg/sdk/endpoint"
)
func main() {
apiKey := os.Getenv("RUNPOD_API_KEY")
baseURL := os.Getenv("RUNPOD_BASE_URL")
endpoint, err := rpEndpoint.New(
&config.Config{ApiKey: &apiKey},
&rpEndpoint.Option{EndpointId: &baseURL},
)
if err != nil {
panic(err)
}
cancelInput := rpEndpoint.CancelInput{
Id: sdk.String("00edfd03-8094-46da-82e3-ea47dd9566dc-u1"),
}
output, err := endpoint.Cancel(&cancelInput)
if err != nil {
panic(err)
}
healthData, _ := json.Marshal(output)
fmt.Printf("health output: %s\n", healthData)
}
```
**Response:**
```json theme={"theme":{"light":"github-light","dark":"github-dark"}}
{
"id": "724907fe-7bcc-4e42-998d-52cb93e1421f-u1",
"status": "CANCELLED"
}
```
## /retry
Requeue jobs that have failed or timed out without submitting a new request. Maintains the same job ID, requeues with original input parameters, and removes previous output. Only works for jobs with `FAILED` or `TIMED_OUT` status.
Replace `YOUR_JOB_ID` with the job ID from your `/run` response.
```sh theme={"theme":{"light":"github-light","dark":"github-dark"}}
curl --request POST \
--url https://api.runpod.ai/v2/$ENDPOINT_ID/retry/YOUR_JOB_ID \
-H "authorization: $RUNPOD_API_KEY"
```
**Response:**
```json theme={"theme":{"light":"github-light","dark":"github-dark"}}
{
"id": "60902e6c-08a1-426e-9cb9-9eaec90f5e2b-u1",
"status": "IN_QUEUE"
}
```
Job results expire after a set period. Async jobs (`/run`) results are available for 30 minutes, sync jobs (`/runsync`) for 1 minute (up to 5 minutes with `?wait=t`). Once expired, jobs cannot be retried.
## /purge-queue
Remove all pending jobs from the queue. Useful for error recovery, clearing outdated requests, and resetting after configuration changes.
```sh theme={"theme":{"light":"github-light","dark":"github-dark"}}
curl --request POST \
--url https://api.runpod.ai/v2/$ENDPOINT_ID/purge-queue \
-H "authorization: $RUNPOD_API_KEY"
```
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
import runpod
import os
runpod.api_key = os.getenv("RUNPOD_API_KEY")
endpoint = runpod.Endpoint(os.getenv("ENDPOINT_ID"))
endpoint.purge_queue(timeout=3)
```
```javascript theme={"theme":{"light":"github-light","dark":"github-dark"}}
const { RUNPOD_API_KEY, ENDPOINT_ID } = process.env;
import runpodSdk from "runpod-sdk";
async function main() {
try {
const runpod = runpodSdk(RUNPOD_API_KEY);
const endpoint = runpod.endpoint(ENDPOINT_ID);
await endpoint.run({
input: {
prompt: "Hello, World!",
},
});
const purgeQueue = await endpoint.purgeQueue();
console.log(purgeQueue);
} catch (error) {
console.error("An error occurred:", error);
}
}
main();
```
This operation only affects jobs waiting in the queue. Jobs already in progress continue to run.
**Response:**
```json theme={"theme":{"light":"github-light","dark":"github-dark"}}
{
"removed": 2,
"status": "completed"
}
```
## /health
Get a quick overview of your endpoint's operational status including worker availability and job queue status.
```sh theme={"theme":{"light":"github-light","dark":"github-dark"}}
curl --request GET \
--url https://api.runpod.ai/v2/$ENDPOINT_ID/health \
-H "authorization: $RUNPOD_API_KEY"
```
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
import runpod
import json
import os
runpod.api_key = os.getenv("RUNPOD_API_KEY")
endpoint = runpod.Endpoint(os.getenv("ENDPOINT_ID"))
endpoint_health = endpoint.health()
print(json.dumps(endpoint_health, indent=2))
```
```javascript theme={"theme":{"light":"github-light","dark":"github-dark"}}
const { RUNPOD_API_KEY, ENDPOINT_ID } = process.env;
import runpodSdk from "runpod-sdk";
const runpod = runpodSdk(RUNPOD_API_KEY);
const endpoint = runpod.endpoint(ENDPOINT_ID);
const health = await endpoint.health();
console.log(health);
```
```go theme={"theme":{"light":"github-light","dark":"github-dark"}}
package main
import (
"encoding/json"
"fmt"
"log"
"os"
"github.com/runpod/go-sdk/pkg/sdk/config"
rpEndpoint "github.com/runpod/go-sdk/pkg/sdk/endpoint"
)
func main() {
apiKey := os.Getenv("RUNPOD_API_KEY")
endpointId := os.Getenv("ENDPOINT_ID")
endpoint, err := rpEndpoint.New(
&config.Config{ApiKey: &apiKey},
&rpEndpoint.Option{EndpointId: &endpointId},
)
if err != nil {
log.Fatalf("Failed to create endpoint: %v", err)
}
health, err := endpoint.Health()
if err != nil {
log.Fatalf("Failed to get health: %v", err)
}
data, _ := json.Marshal(health)
fmt.Printf("Health: %s\n", data)
}
```
**Response:**
```json theme={"theme":{"light":"github-light","dark":"github-dark"}}
{
"jobs": {
"completed": 1,
"failed": 5,
"inProgress": 0,
"inQueue": 2,
"retried": 0
},
"workers": {
"idle": 0,
"running": 0
}
}
```
# Overview
Source: https://docs.runpod.io/serverless/endpoints/overview
Deploy and manage Serverless endpoints using the Runpod console or REST API. Review configuration and operations guidance for Runpod Serverless.
Endpoints are the foundation of Runpod Serverless, serving as the gateway for deploying and managing your [Serverless workers](/serverless/workers/overview). Each endpoint provides a unique URL that accepts [HTTP requests](/serverless/endpoints/send-requests), processes them using your [handler function](/serverless/workers/handler-functions), and returns results.
Learn how to send requests to your endpoints.
Configure scaling, timeouts, and GPU selection.
Monitor job status and metrics.
Reduce cold starts with cached models.
## Endpoint types
| | Queue-based | Load balancing |
| --------------------- | ------------------------------------------ | ----------------------------- |
| **Processing** | Requests queued and processed sequentially | Direct HTTP access to workers |
| **Execution modes** | Async (`/run`) or sync (`/runsync`) | Custom HTTP endpoints |
| **Retries** | Automatic retries on failure | No automatic retries |
| **Handler required?** | Yes | No (use any HTTP framework) |
| **Best for** | Batch jobs, guaranteed execution | Real-time apps, streaming |
Learn more about [load balancing endpoints](/serverless/load-balancing/overview).
## Create an endpoint
Before creating an endpoint, ensure you have a [handler function](/serverless/workers/handler-functions) and [Dockerfile](/serverless/workers/create-dockerfile).
1. Navigate to the [Serverless section](https://www.console.runpod.io/serverless) and click **New Endpoint**.
2. Choose your deployment path:
* **Hello World**: Runpod forks a starter worker template into a new GitHub repo in your account. Choose Queue-based or Load balancing, then click **Deploy**.
* **Hugging Face LLM**: Search for any text-generation model on Hugging Face (for example, type "Gemma" to find Gemma 4), select it, and click **Create Endpoint**. Runpod deploys a vLLM endpoint for you.
* **Docker**: Deploy from a container image. Select a saved Serverless template to fill in the container configuration automatically, or skip the template and enter an image name manually. See [Deploy from Docker](/serverless/workers/deploy).
* **GitHub**: Select a repository, filtering by code owner if needed. Runpod checks for a Dockerfile and runs a background check on your handler: queue-based endpoints check for handler files, and load balancing endpoints check for a `/ping` path. See [Deploy from GitHub](/serverless/workers/github-integration).
* **Hub**: Opens the Hub browser, where you can browse and deploy prebuilt workers. This replaces the previous "Ready-to-Deploy Repos" option. See [Hub overview](/hub/overview).
* **Flash**: A guided setup flow for [Flash](/flash/overview) that walks you through installing the SDK, initializing your project, and sending your first command. Steps complete automatically as you progress.
3. For the GitHub, Docker, and Hello World paths, configure your endpoint before deploying:
* **Endpoint name** and **type** ([Queue-based](/flash/create-endpoints#queue-based-endpoints) or [Load balancing](/flash/create-endpoints#load-balanced-endpoints))
* **GPU** configuration and worker scaling
* **Model** (optional): Enter a Hugging Face URL for [cached models](/serverless/endpoints/model-caching)
* **Environment variables** and container configuration. See [environment variables](/serverless/development/environment-variables).
4. Click **Deploy Endpoint**.
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
curl --request POST \
--url https://rest.runpod.io/v1/endpoints \
--header 'Authorization: Bearer RUNPOD_API_KEY' \
--header 'Content-Type: application/json' \
--data '{
"name": "my-endpoint",
"templateId": "30zmvf89kd",
"gpuTypeIds": ["NVIDIA GeForce RTX 4090"],
"workersMin": 0,
"workersMax": 3,
"idleTimeout": 5
}'
```
See the [Endpoint API reference](/api-reference/endpoints/POST/endpoints) for all parameters.
Optimize cost and availability by specifying multiple GPU types in priority order. Runpod allocates your first choice if available, otherwise uses the next in your list.
After deployment, your endpoint displays a unique API URL: `https://api.runpod.ai/v2/{endpoint_id}/`
## Edit an endpoint
1. Navigate to the [Serverless section](https://www.console.runpod.io/serverless).
2. Click the three dots on your endpoint → **Edit Endpoint**.
3. Update [endpoint settings](/serverless/endpoints/endpoint-configurations) and click **Save Endpoint**.
Changes to GPU types or worker counts may require restarting active workers.
## Delete an endpoint
1. Navigate to the [Serverless section](https://www.console.runpod.io/serverless).
2. Click the three dots on your endpoint → **Delete Endpoint**.
3. Type the endpoint name to confirm.
Deleting an endpoint permanently removes all configuration, logs, and job history.
# Rolling releases
Source: https://docs.runpod.io/serverless/endpoints/rolling-releases
Understand how Runpod rolls out Serverless endpoint updates without downtime, including worker replacement, version tracking, and rollback behavior.
When you update an endpoint's Docker image or configuration, Runpod deploys the change as a rolling release. Workers are replaced progressively rather than all at once, so your endpoint continues to handle requests throughout the update.
## When a rolling release happens
A rolling release starts automatically whenever you save a change to an endpoint that has active workers. This includes:
* Updating the Docker image (for example, pushing a new image version and saving the endpoint)
* Changing any endpoint configuration setting that requires workers to restart
## How it works
When a rolling release starts, Runpod distinguishes between idle workers and workers that are actively processing a job.
**Idle workers** (not currently processing a job) on the old version are terminated immediately and replaced with new workers running the updated image.
**Running workers** (actively processing a job) are allowed to finish their current job. Once the job completes, the worker is replaced with a new one on the updated image.
During the rollout, your endpoint continues to accept and process requests. New requests are routed to workers on the updated image when available.
## Best practices
**Use versioned image tags, not `:latest`.** If you deploy with `:latest`, Runpod may serve the cached version on existing workers rather than your updated image. Workers only pick up a new image when they are replaced. Using an explicit version tag (for example, `v1.2.0` or a SHA digest) ensures a clean rollout.
```sh theme={"theme":{"light":"github-light","dark":"github-dark"}}
# Tag with a version
docker build -t yourusername/your-worker:v1.2.0 .
docker push yourusername/your-worker:v1.2.0
```
Then update your endpoint to reference the new tag. This triggers a rolling release and guarantees all new workers use the correct image.
**Expect a short overlap period.** During a rolling release, some workers run the old image and some run the new one. If your update changes request or response behavior in a breaking way, plan for this transition. For strict version consistency, scale your endpoint to zero workers before updating and then scale back up.
To scale an endpoint to zero workers before updating, set **Max workers** to 0, wait for all running jobs to complete and workers to terminate, then update the image and restore your max workers setting.
## Checking rollout status
You can monitor which workers are running during a rollout from the **Workers** tab on your endpoint's detail page in the Runpod console. Workers on the previous image version will show as terminating as they finish their jobs and are replaced.
# Send API requests
Source: https://docs.runpod.io/serverless/endpoints/send-requests
Submit and manage jobs for your queue-based endpoints by sending HTTP requests. Review configuration and operations guidance for Runpod Serverless.
After creating a [Serverless endpoint](/serverless/endpoints/overview), you can start sending HTTP requests to submit jobs and retrieve results:
```sh theme={"theme":{"light":"github-light","dark":"github-dark"}}
curl -x POST https://api.runpod.ai/v2/ENDPOINT_ID/runsync \
-H "authorization: Bearer RUNPOD_API_KEY" \
-H "content-type: application/json" \
-d '{ "input": { "prompt": "Hello, world!" }}'
```
This guide is for . If you're building a , the request structure and endpoints depend on how you define your HTTP servers.
## How requests work
A **job** is a unit of work containing the input data from the request, packaged for processing by your [workers](/serverless/workers/overview). If no worker is immediately available, the job is queued. Once a worker is available, the job is processed using your worker's [handler function](/serverless/workers/handler-functions).
## Sync vs. async
* `/runsync` submits a **synchronous** job.
* Client waits for the job to complete before returning the result.
* Results are available for 1 minute (5 minutes max).
* Ideal for quick responses and interactive applications.
* `/run` submits an **asynchronous** job.
* The job processes in the background; retrieve results via `/status`.
* Results are available for 30 minutes after completion.
* Ideal for long-running tasks and batch processing.
## Request input structure
When submitting a job with `/runsync` or `/run`, your request must include a JSON object with the key `input` containing the parameters required by your worker's [handler function](/serverless/workers/handler-functions):
```json theme={"theme":{"light":"github-light","dark":"github-dark"}}
{
"input": {
"prompt": "Your input here"
}
}
```
The exact parameters depend on your specific worker implementation. Check your worker's documentation for required and optional parameters.
## Send requests from the console
The quickest way to test your endpoint is in the Runpod console. Navigate to [Serverless](https://www.console.runpod.io/serverless), select your endpoint, and click the **Requests** tab.
Modify the default test request as needed, then click **Run**. On first execution, workers need to initialize, which may take a moment.
## Operation overview
Queue-based endpoints support these operations for job lifecycle management:
| Operation | Method | Description |
| -------------- | ------ | ----------------------------------------------------------------- |
| `/runsync` | POST | Submit a synchronous job and wait for complete results. |
| `/run` | POST | Submit an asynchronous job that processes in the background. |
| `/status` | GET | Check status, execution details, and results of a submitted job. |
| `/stream` | GET | Receive incremental results as they become available. |
| `/cancel` | POST | Stop a job in progress or waiting in the queue. |
| `/retry` | POST | Requeue a failed or timed-out job with the same job ID and input. |
| `/purge-queue` | POST | Clear all pending jobs from the queue. |
| `/health` | GET | Monitor endpoint status, including worker and job statistics. |
See the [operation reference](/serverless/endpoints/operation-reference) for detailed examples using cURL and the Runpod SDK.
For custom API paths, use [load balancing endpoints](/serverless/load-balancing/overview).
## Advanced options
Beyond the required `input` object, you can include optional top-level parameters for additional functionality.
### Webhook notifications
Receive notifications when jobs complete by specifying a webhook URL:
```json theme={"theme":{"light":"github-light","dark":"github-dark"}}
{
"input": { "prompt": "Your input here" },
"webhook": "https://your-webhook-url.com"
}
```
Your webhook should return a `200` status code. If the call fails, Runpod retries up to 2 more times with a 10-second delay.
### Execution policies
Control job execution behavior with custom policies:
```json theme={"theme":{"light":"github-light","dark":"github-dark"}}
{
"input": { "prompt": "Your input here" },
"policy": {
"executionTimeout": 900000,
"lowPriority": false,
"ttl": 3600000
}
}
```
| Option | Description | Default | Constraints |
| ------------------ | ------------------------------------------------ | ------------------- | ---------------------- |
| `executionTimeout` | Maximum time a job can run while being processed | 600000 (10 minutes) | Min 5 sec, max 7 days |
| `lowPriority` | When true, job won't trigger worker scaling | false | - |
| `ttl` | Total lifespan of the job before deletion | 86400000 (24 hours) | Min 10 sec, max 7 days |
Setting `executionTimeout` in a request overrides the default endpoint setting for that specific job only.
#### TTL vs. execution timeout
* **`ttl`**: Total lifespan of the job. Timer starts when submitted and covers queue time, execution time, and everything in between. When TTL expires, the job is deleted regardless of state.
* **`executionTimeout`**: Maximum time the job can actively run once a worker picks it up. Only enforced during execution.
TTL is a hard limit. If TTL expires while a job is running, the job is immediately removed and status checks return 404, even if the job would have completed successfully.
#### Long-running jobs
For jobs that need to run longer than the default 24-hour TTL:
1. Set `executionTimeout` to your desired maximum runtime.
2. Set `ttl` to cover **both expected queue time and execution time**.
```json theme={"theme":{"light":"github-light","dark":"github-dark"}}
{
"input": { "prompt": "Long running task" },
"policy": {
"executionTimeout": 172800000,
"ttl": 259200000
}
}
```
This allows up to 48 hours of active runtime with 72 hours total lifespan (24 hours headroom for queue time).
Both `ttl` and `executionTimeout` have a maximum of 7 days. A job with 7-day TTL that queues for 2 days only has 5 days remaining for execution.
#### Result retention
After completion, results are retained for a fixed period separate from TTL:
| Request type | Retention period |
| ----------------- | ---------------- |
| `/run` (async) | 30 minutes |
| `/runsync` (sync) | 1 minute |
### S3-compatible storage
Configure S3-compatible storage for endpoints working with large files:
```json theme={"theme":{"light":"github-light","dark":"github-dark"}}
{
"input": { "prompt": "Your input here" },
"s3Config": {
"accessId": "BUCKET_ACCESS_KEY_ID",
"accessSecret": "BUCKET_SECRET_ACCESS_KEY",
"bucketName": "BUCKET_NAME",
"endpointUrl": "BUCKET_ENDPOINT_URL"
}
}
```
Your worker must contain logic to use this information for storage operations. Works with any S3-compatible provider including MinIO, Backblaze B2, and DigitalOcean Spaces.
## Rate limits
Runpod enforces rate limits per endpoint and operation:
| Operation | Method | Rate Limit | Concurrent Limit |
| -------------- | ------ | ---------------------------- | ---------------- |
| `/runsync` | POST | 2000 requests per 10 seconds | 400 concurrent |
| `/run` | POST | 1000 requests per 10 seconds | 200 concurrent |
| `/status` | GET | 2000 requests per 10 seconds | 400 concurrent |
| `/stream` | GET | 2000 requests per 10 seconds | 400 concurrent |
| `/cancel` | POST | 100 requests per 10 seconds | 20 concurrent |
| `/purge-queue` | POST | 2 requests per 10 seconds | N/A |
| `/openai/*` | POST | 2000 requests per 10 seconds | 400 concurrent |
| `/requests` | GET | 10 requests per 10 seconds | 2 concurrent |
### Dynamic rate limiting
Rate limits scale with your endpoint's worker count. The system uses whichever is higher between:
1. **Base limit**: Fixed rate limit per user per endpoint (shown above)
2. **Worker-based limit**: `number_of_running_workers × requests_per_worker`
Requests exceeding the effective limit return `429 (Too Many Requests)`. Implement retry logic with exponential backoff to handle rate limiting gracefully.
## Error handling
Common errors and solutions:
| HTTP Status | Meaning | Solution |
| ----------- | --------------------- | ------------------------------------------------- |
| 400 | Bad Request | Check your request format and parameters |
| 401 | Unauthorized | Verify your API key is correct and has permission |
| 404 | Not Found | Check your endpoint ID |
| 429 | Too Many Requests | Implement backoff and retry logic |
| 500 | Internal Server Error | Check endpoint logs; worker may have crashed |
| Issue | Possible Causes | Solutions |
| ------------------ | ----------------------------------------- | ----------------------------------------------------------------- |
| Job stuck in queue | No available workers, max workers reached | Increase max workers, check endpoint health |
| Timeout errors | Job takes longer than execution timeout | Increase timeout in job policy, optimize processing |
| Failed jobs | Worker errors, input validation issues | Check [endpoint logs](/serverless/development/logs), verify input |
| Missing results | Results expired | Retrieve within expiration window (30 min async, 1 min sync) |
See [error handling](/serverless/workers/handler-functions#error-handling) for implementation details.
# Build a load balancing worker
Source: https://docs.runpod.io/serverless/load-balancing/build-a-worker
Learn how to implement and deploy a load balancing worker with FastAPI. Review configuration and operations guidance for Runpod Serverless.
This tutorial shows how to build a load balancing worker using FastAPI and deploy it as a Serverless endpoint on Runpod.
## Requirements
Before you begin you'll need:
* A Runpod account.
* Basic familiarity with Python and REST APIs.
* Docker installed on your local machine.
## Step 1: Create a basic FastAPI application
You can download a preconfigured repository containing the completed code for this tutorial [on GitHub](https://github.com/runpod-workers/worker-load-balancing/).
First, let's create a simple FastAPI application that will serve as our API.
Create a file named `app.py`:
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
import os
from fastapi import FastAPI, HTTPException
from pydantic import BaseModel
# Create FastAPI app
app = FastAPI()
# Define request models
class GenerationRequest(BaseModel):
prompt: str
max_tokens: int = 100
temperature: float = 0.7
class GenerationResponse(BaseModel):
generated_text: str
# Global variable to track requests
request_count = 0
# Health check endpoint; required for Runpod to monitor worker health
@app.get("/ping")
async def health_check():
return {"status": "healthy"}
# Our custom generation endpoint
@app.post("/generate", response_model=GenerationResponse)
async def generate(request: GenerationRequest):
global request_count
request_count += 1
# A simple mock implementation; we'll replace this with an actual model later
generated_text = f"Response to: {request.prompt} (request #{request_count})"
return {"generated_text": generated_text}
# A simple endpoint to show request stats
@app.get("/stats")
async def stats():
return {"total_requests": request_count}
# Run the app when the script is executed
if __name__ == "__main__":
import uvicorn
port = int(os.getenv("PORT", 80))
print(f"Starting server on port {port}")
# Start the server
uvicorn.run(app, host="0.0.0.0", port=port)
```
This simple application defines the following endpoints:
* A health check endpoint at `/ping`
* A text generation endpoint at `/generate`
* A statistics endpoint at `/stats`
## Step 2: Create a Dockerfile
Now, let's create a `Dockerfile` to package our application:
```
FROM nvidia/cuda:12.1.0-base-ubuntu22.04
RUN apt-get update -y \
&& apt-get install -y python3-pip
RUN ldconfig /usr/local/cuda-12.1/compat/
# Install Python dependencies
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
# Copy application code
COPY app.py .
# Start the handler
CMD ["python3", "app.py"]
```
You'll also need to create a `requirements.txt` file:
```
fastapi==0.95.1
uvicorn==0.22.0
pydantic==1.10.7
```
## Step 3: Build and push the Docker image
Build and push your Docker image to a container registry:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
# Build the image
docker build --platform linux/amd64 -t YOUR_DOCKER_USERNAME/loadbalancer-example:v1.0 .
# Push to Docker Hub
docker push YOUR_DOCKER_USERNAME/loadbalancer-example:v1.0
```
## Step 4: Deploy to Runpod
Now, let's deploy our application to a Serverless endpoint:
1. Go to the [Serverless page](https://console.runpod.io/serverless) in the Runpod console.
2. Click **New Endpoint**
3. Click **Import from Docker Registry**.
4. In the **Container Image** field, enter your Docker image URL:
```
YOUR_DOCKER_USERNAME/loadbalancer-example:v1.0
```
Then click **Next**.
5. Give your endpoint a name.
6. Under **Endpoint Type**, select **Load Balancer**.
7. Under **GPU Configuration**, select at least one GPU type (16 GB or 24 GB GPUs are fine for this example).
8. Leave all other settings at their defaults.
9. Click **Deploy Endpoint**.
## Step 5: Access your custom API
Once your endpoint is created, you can access your custom APIs at:
```
https://ENDPOINT_ID.api.runpod.ai/PATH
```
For example, the load balancing worker we defined in step 1 exposes these endpoints:
* Health check: `https://ENDPOINT_ID.api.runpod.ai/ping`
* Generate text: `https://ENDPOINT_ID.api.runpod.ai/generate`
* Get request count: `https://ENDPOINT_ID.api.runpod.ai/stats`
Try running one or more of these commands, replacing `ENDPOINT_ID` and `RUNPOD_API_KEY` with your actual endpoint ID and API key:
```bash generate theme={"theme":{"light":"github-light","dark":"github-dark"}}
curl -X POST "https://ENDPOINT_ID.api.runpod.ai/generate" \
-H 'Authorization: Bearer RUNPOD_API_KEY' \
-H "Content-Type: application/json" \
-d '{"prompt": "Hello, world!"}'
```
```bash ping theme={"theme":{"light":"github-light","dark":"github-dark"}}
curl -X GET "https://ENDPOINT_ID.api.runpod.ai/ping" \
-H 'Authorization: Bearer RUNPOD_API_KEY' \
-H "Content-Type: application/json" \
```
```bash stats theme={"theme":{"light":"github-light","dark":"github-dark"}}
curl -X GET "https://ENDPOINT_ID.api.runpod.ai/stats" \
-H 'Authorization: Bearer RUNPOD_API_KEY' \
-H "Content-Type: application/json" \
```
After sending a request, your workers will take some time to initialize. You can track their progress by checking the logs in the **Workers** tab of your endpoint page.
If you see: `{"error":"no workers available"}%` after running the request, this means your workers did not initialize in time to process it. If you try running the request again, this will usually resolve the issue.
For production applications, implement a health check with retries before sending requests. See [Handling cold start errors](/serverless/load-balancing/overview#handling-cold-start-errors) for a complete code example.
Congratulations! You've successfully deployed and tested a load balancing endpoint. If you want to use a real model, you can follow the [vLLM worker](/serverless/load-balancing/vllm-worker) tutorial.
## (Optional) Advanced endpoint definitions
For a more complex API, you can define multiple endpoints and organize them logically. Here's an example of how to structure a more complex API:
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
from fastapi import FastAPI, HTTPException, Depends, Query
from pydantic import BaseModel
import os
app = FastAPI()
# --- Authentication middleware ---
def verify_api_key(api_key: str = Query(None, alias="api_key")):
if api_key != os.getenv("API_KEY", "test_key"):
raise HTTPException(401, "Invalid API key")
return api_key
# --- Models ---
class TextRequest(BaseModel):
text: str
max_length: int = 100
class ImageRequest(BaseModel):
prompt: str
width: int = 512
height: int = 512
# --- Text endpoints ---
@app.post("/v1/text/summarize")
async def summarize(request: TextRequest, api_key: str = Depends(verify_api_key)):
# Implement text summarization
return {"summary": f"Summary of: {request.text[:30]}..."}
@app.post("/v1/text/translate")
async def translate(request: TextRequest, target_lang: str, api_key: str = Depends(verify_api_key)):
# Implement translation
return {"translation": f"Translation to {target_lang}: {request.text[:30]}..."}
# --- Image endpoints ---
@app.post("/v1/image/generate")
async def generate_image(request: ImageRequest, api_key: str = Depends(verify_api_key)):
# Implement image generation
return {"image_url": f"https://example.com/images/{hash(request.prompt)}.jpg"}
# --- Health check ---
@app.get("/ping")
async def health_check():
return {"status": "healthy"}
```
## (Optional) WebSocket support
Load balancing endpoints also support WebSocket connections. This section shows how to add a WebSocket endpoint to your worker and connect to it from a client.
You can clone the [worker-lb-websocket repository](https://github.com/runpod-workers/worker-lb-websocket) for a complete working example, including scaling tests.
### Add a WebSocket endpoint
WebSocket endpoints in FastAPI use the `@app.websocket()` decorator. Add the following to your `app.py`:
```python app.py theme={"theme":{"light":"github-light","dark":"github-dark"}}
from fastapi import FastAPI, WebSocket, WebSocketDisconnect
import asyncio
app = FastAPI()
# Track active connections
active_ws_connections: list[WebSocket] = []
@app.websocket("/ws")
async def websocket_endpoint(websocket: WebSocket):
"""Streaming WebSocket endpoint.
Clients send JSON messages like: {"prompt": "Hello", "max_tokens": 50}
Server streams responses back and sends {"done": true} when complete.
"""
await websocket.accept()
active_ws_connections.append(websocket)
try:
while True:
data = await websocket.receive_json()
prompt = data.get("prompt", "")
if not prompt:
await websocket.send_json({"error": "prompt is required"})
continue
# Simulate streaming response (replace with your model)
words = f"Response to: {prompt}".split()
for i, word in enumerate(words):
await websocket.send_json({"token": word, "index": i})
await asyncio.sleep(0.05) # Simulate inference latency
await websocket.send_json({"done": True})
except WebSocketDisconnect:
pass
finally:
active_ws_connections.remove(websocket)
```
### Connect from a client
When connecting to a WebSocket endpoint on a load balancing worker, you must set the `open_timeout` parameter to allow time for workers to scale up. The default timeout of 5 seconds is usually not enough.
```python client.py theme={"theme":{"light":"github-light","dark":"github-dark"}}
import asyncio
import json
import websockets
async def connect_to_worker():
url = "wss://ENDPOINT_ID.api.runpod.ai/ws"
headers = [("Authorization", "Bearer RUNPOD_API_KEY")]
# Set open_timeout to allow workers time to scale up (default is ~5s)
async with websockets.connect(
url,
additional_headers=headers,
open_timeout=60.0, # Wait up to 60 seconds for connection
) as ws:
# Send a request
await ws.send(json.dumps({"prompt": "Hello, world!", "max_tokens": 50}))
# Receive streaming response
while True:
response = json.loads(await ws.recv())
if response.get("done"):
print("Generation complete")
break
print(response.get("token", ""), end=" ")
asyncio.run(connect_to_worker())
```
If you don't set `open_timeout`, connections will fail with a timeout error when workers need to scale up from zero. A value of 60 seconds works for most use cases.
### Update requirements.txt
Add the `websockets` library to your client's dependencies:
```
websockets==14.2
```
## Troubleshooting
Here are some common issues and methods for troubleshooting:
* **No workers available**: If your request returns `{"error":"no workers available"}%`, this means means your workers did not initialize in time to process the request. Running the request again will usually fix this issue.
* **Worker unhealthy**: Check your health endpoint implementation and ensure it's returning proper status codes.
* **API not accessible**: If your request returns `{"error":"not allowed for QB API"}`, verify that your endpoint type is set to "Load Balancer".
* **Port issues**: Make sure the environment variable for `PORT` matches what your application is using, and that the `PORT_HEALTH` variable is set to a different port.
* **Model errors**: Check your model's requirements and whether it's compatible with your GPU.
* **WebSocket timeout**: If WebSocket connections fail with timeout errors, increase the `open_timeout` parameter in your client code to allow workers time to scale up. See [(Optional) WebSocket support](#optional-websocket-support) for details.
## Next steps
Now that you've learned how to build a basic load balancing worker, you can try [implementing a real model with vLLM](/serverless/load-balancing/vllm-worker).
# Overview
Source: https://docs.runpod.io/serverless/load-balancing/overview
Deploy custom direct-access REST APIs with load balancing Serverless endpoints. Review configuration and operations guidance for Runpod Serverless.
Load balancing endpoints route incoming traffic directly to available workers, bypassing the queueing system. Unlike that process requests sequentially, load balancing distributes requests across your worker pool for lower latency.
You can create custom REST endpoints accessible via a unique URL:
```
https://ENDPOINT_ID.api.runpod.ai/YOUR_CUSTOM_PATH
```
Create and deploy a load balancing worker.
Deploy vLLM with load balancing.
## Load balancing vs. queue-based endpoints
### Queue-based endpoints
With queue-based endpoints, are placed in a queue and processed in order. They use the standard handler pattern (`def handler(job)`) and are accessed through fixed endpoints like `/run` and `/runsync`.
These endpoints are better for tasks that can be processed asynchronously and guarantee request processing, similar to how TCP guarantees packet delivery in networking.
### Load balancing endpoints (new)
Load balancing endpoints send requests directly to workers without queuing. You can use any HTTP framework such as FastAPI or Flask, and define custom URL paths and API contracts to suit your specific needs.
These endpoints are ideal for real-time applications and streaming, but provide no queuing mechanism for request backlog, similar to UDP's behavior in networking.
## Endpoint type comparison table
| Aspect | Load balancing | Queue-based |
| ------------------- | ----------------------------------------- | ------------------------------------- |
| **Request flow** | Direct to worker HTTP server | Through queueing system |
| **Implementation** | Custom HTTP server (FastAPI, Flask, etc.) | Handler function |
| **API flexibility** | Custom URL paths, any HTTP capability | Fixed `/run` and `/runsync` endpoints |
| **Backpressure** | Drops requests when overloaded | Queue buffering |
| **Latency** | Lower (single-hop) | Higher (queue + worker) |
| **Error handling** | No built-in retry | Automatic retries |
## Worker comparison
**Queue-based worker** (traditional):
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
import runpod
def handler(job):
prompt = job["input"].get("prompt", "Hello world")
return {"generated_text": f"Generated text for: {prompt}"}
runpod.serverless.start({"handler": handler})
```
**Load balancing worker** (custom HTTP server):
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
from fastapi import FastAPI
import os
app = FastAPI()
@app.get("/ping")
async def health_check():
return {"status": "healthy"}
@app.post("/generate")
async def generate(request: dict):
return {"generated_text": f"Generated text for: {request['prompt']}"}
if __name__ == "__main__":
import uvicorn
uvicorn.run(app, host="0.0.0.0", port=int(os.getenv("PORT", "80")))
```
This exposes custom endpoints: `https://ENDPOINT_ID.api.runpod.ai/ping` and `https://ENDPOINT_ID.api.runpod.ai/generate`
## Health checks
Each worker exposes a health check endpoint on the `PORT_HEALTH` port, and the load balancer polls it periodically to decide whether the worker is healthy enough to receive traffic. By default the load balancer polls `/ping`, but you can point it at any path by setting the `HEALTH_CHECK_PATH` environment variable. This is useful when you deploy a public image whose server already exposes a health check at a different path, such as a `llama.cpp` image that serves `/health`, so you don't need to build a custom image just to satisfy the health check.
You can also set the health check path directly in the Runpod console when creating a new endpoint. The **Health check endpoint** field appears on the Configure image step. Leave it empty to use the default `/ping`.
The load balancer interprets the response code from the health check endpoint as follows:
| Response code | Status |
| ------------- | ------------ |
| `200` | Healthy |
| `204` | Initializing |
| Other | Unhealthy |
Unhealthy workers are automatically removed from the routing pool.
When calculating endpoint metrics, Runpod calculates the cold start time for load balancing workers by measuring the time it takes between the health check endpoint first returning `204` until it first returns `200`.
## Environment variables
| Variable | Default | Description |
| ------------------- | -------------- | --------------------------------------------------- |
| `PORT` | `80` | Main application server port |
| `PORT_HEALTH` | Same as `PORT` | Health check endpoint port |
| `HEALTH_CHECK_PATH` | `/ping` | Path the load balancer polls to check worker health |
If using a custom port, add it to your endpoint's environment variables and expose it in container configuration (under **Expose HTTP Ports (Max 10)**).
## Timeouts and limits
| Limit | Value |
| ---------------------- | ---------------------------- |
| **Request timeout** | 2 min (no worker available) |
| **Processing timeout** | 5.5 min (per request) |
| **Payload limit** | 30 MB (request and response) |
For payloads larger than 30 MB, use [network volumes](/storage/network-volumes) or implement chunking.
If your server ports are misconfigured, workers stay up for 8 minutes before terminating, returning `502` errors.
## Handling cold starts
When workers are initializing, you may get "no workers available" errors. Implement retry logic to handle this:
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
import requests
import time
def health_check_with_retry(base_url, api_key, max_retries=3, delay=5):
headers = {"Authorization": f"Bearer {api_key}"}
for attempt in range(max_retries):
try:
response = requests.get(f"{base_url}/ping", headers=headers, timeout=10)
if response.status_code == 200:
return True
except Exception:
pass
if attempt < max_retries - 1:
time.sleep(delay)
return False
# Usage
if health_check_with_retry("https://ENDPOINT_ID.api.runpod.ai", "RUNPOD_API_KEY"):
# Worker ready, send requests
pass
```
Use at least 3 retries with 5-10 second delays.
## When to use queue-based endpoints
Use queue-based endpoints when you need:
* Job-based or long-running workloads that run for extended periods. Load balancing is not suited for long-running tasks and would time out or drop the request.
* Guaranteed execution where every request is queued and processed, even during traffic spikes. No requests are dropped when workers are busy.
* Batch or offline workloads processed asynchronously where latency is not critical, such as nightly dataset processing, pre-computing embeddings, or running evaluations.
* Automatic retries on failure without any client-side logic.
* Configurable concurrency that supports both low and high concurrency workloads depending on your needs.
Choose queue-based when your workload can tolerate higher latency in exchange for reliability and guaranteed delivery. If your use case is a short, real-time request/response transaction, use a load balancing endpoint instead.
## When to use load balancing endpoints
Use load balancing endpoints when you need:
* Direct access to your model's HTTP server without queueing overhead.
* Internal batching systems like vLLM that manage their own request batching internally.
* Non-JSON payloads such as binary data or multipart uploads.
* Multiple endpoints within a single worker using custom URL paths and any HTTP framework.
* Low-latency real-time applications where responses are immediate and ephemeral, such as serving a file download or returning page stats.
Choose load balancing when your workload is latency-sensitive and responses are immediate. If your use case involves long-running jobs or requires guaranteed delivery, use a queue-based endpoint instead.
# Build a load balancing vLLM endpoint
Source: https://docs.runpod.io/serverless/load-balancing/vllm-worker
Learn how to deploy a custom vLLM server to a load balancing Serverless endpoint. Review configuration and operations guidance for Runpod Serverless.
This tutorial shows how to build a vLLM application using FastAPI and deploy it as a load balancing Serverless endpoint on Runpod.
To get a basic understanding of how to build a load balancing worker (or for more general use cases), see [Build a load balancing worker](/serverless/load-balancing/build-a-worker).
## Requirements
Before you begin you'll need:
* A Runpod account.
* Basic familiarity with Python, REST APIs, and vLLM.
* Docker installed on your local machine.
## Step 1: Create your project files
You can download a preconfigured repository containing the completed code for this tutorial [on GitHub](https://github.com/runpod-workers/vllm-loadbalancer-ep/).
Start by creating a new directory for your project:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
mkdir vllm_worker
cd vllm_worker
```
Then, create the following files and directories:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
touch Dockerfile
touch requirements.txt
mkdir src
touch src/handler.py
touch src/models.py
touch src/utils.py
```
Your project structure should now look like this:
## Step 2: Define data models
We'll start by creating the data models that define the structure of your API. These models specify what data your endpoints expect to receive and what they'll return.
Add the following code to `src/models.py`:
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
from typing import Optional, List, Union, Literal
from pydantic import BaseModel, Field
class ChatMessage(BaseModel):
role: Literal["system", "user", "assistant"]
content: str
class GenerationRequest(BaseModel):
prompt: str
max_tokens: int = Field(default=512, ge=1, le=4096)
temperature: float = Field(default=0.7, ge=0.0, le=2.0)
top_p: float = Field(default=0.9, ge=0.0, le=1.0)
top_k: int = Field(default=-1, ge=-1)
frequency_penalty: float = Field(default=0.0, ge=-2.0, le=2.0)
presence_penalty: float = Field(default=0.0, ge=-2.0, le=2.0)
stop: Optional[Union[str, List[str]]] = None
stream: bool = Field(default=False)
class GenerationResponse(BaseModel):
text: str
finish_reason: str
prompt_tokens: int
completion_tokens: int
total_tokens: int
class ChatCompletionRequest(BaseModel):
messages: List[ChatMessage]
max_tokens: int = Field(default=512, ge=1, le=4096)
temperature: float = Field(default=0.7, ge=0.0, le=2.0)
top_p: float = Field(default=0.9, ge=0.0, le=1.0)
stop: Optional[Union[str, List[str]]] = None
stream: bool = Field(default=False)
class ErrorResponse(BaseModel):
error: str
detail: str
request_id: Optional[str] = None
```
The `GenerationRequest` and `ChatCompletionRequest` models specify what data clients need to send, while `GenerationResponse` and `ErrorResponse` define what they'll receive back.
Each data model includes validation rules using Pydantic's `Field` function to ensure parameters stay within acceptable ranges.
## Step 3: Create utility functions
Next, we'll create a few helper functions to support the main application. These utilities handle common tasks like formatting chat prompts and creating standardized error responses.
Add the following code to `src/utils.py`:
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
from typing import List
from transformers import AutoTokenizer
from .models import ChatMessage, ErrorResponse
def get_tokenizer(model_name: str):
"""Get tokenizer for the given model"""
return AutoTokenizer.from_pretrained(model_name, trust_remote_code=True)
def format_chat_prompt(messages: List[ChatMessage], model_name: str) -> str:
"""Format messages using the model's chat template"""
tokenizer = get_tokenizer(model_name)
# Use model's built-in chat template if available
if hasattr(tokenizer, 'apply_chat_template'):
message_dicts = [{"role": msg.role, "content": msg.content} for msg in messages]
return tokenizer.apply_chat_template(
message_dicts,
tokenize=False,
add_generation_prompt=True
)
# Fallback to common format
formatted_prompt = ""
for message in messages:
if message.role == "system":
formatted_prompt += f"System: {message.content}\n\n"
elif message.role == "user":
formatted_prompt += f"Human: {message.content}\n\n"
elif message.role == "assistant":
formatted_prompt += f"Assistant: {message.content}\n\n"
formatted_prompt += "Assistant: "
return formatted_prompt
def create_error_response(error: str, detail: str, request_id: str = None) -> ErrorResponse:
return ErrorResponse(error=error, detail=detail, request_id=request_id)
```
The `format_chat_prompt` function converts chat-style conversations into the text format expected by language models. It first tries to use the model's built-in chat template, then falls back to a generic format if that's not available.
The `create_error_response` function provides a consistent way to generate error messages throughout your application.
## Step 4: Build the main FastAPI application
Now we'll build the main application file, `src/handler.py`. This file acts as the orchestrator, bringing together the models and utilities we just created. It uses FastAPI to create the server, defines the API endpoints, and manages the vLLM engine's lifecycle.
Add the following code to `src/handler.py`:
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
from fastapi import FastAPI, HTTPException, status
from fastapi.responses import StreamingResponse, JSONResponse
from contextlib import asynccontextmanager
from typing import Optional, AsyncGenerator
import json
import logging
import os
import uvicorn
from vllm import AsyncLLMEngine
from vllm.engine.arg_utils import AsyncEngineArgs
from vllm.sampling_params import SamplingParams
from vllm.utils import random_uuid
from utils import format_chat_prompt, create_error_response
from .models import GenerationRequest, GenerationResponse, ChatCompletionRequest
# Configure logging
logging.basicConfig(
level=logging.INFO,
format='%(asctime)s - %(name)s - %(levelname)s - %(message)s',
handlers=[
logging.StreamHandler(),
]
)
logger = logging.getLogger(__name__)
@asynccontextmanager
async def lifespan(_: FastAPI):
"""Initialize the vLLM engine on startup and cleanup on shutdown"""
# Startup
await create_engine()
yield
# Shutdown cleanup
global engine, engine_ready
if engine:
logger.info("Shutting down vLLM engine...")
# vLLM AsyncLLMEngine doesn't have an explicit shutdown method,
# but we can clean up our references
engine = None
engine_ready = False
logger.info("vLLM engine shutdown complete")
app = FastAPI(title="vLLM Load Balancing Server", version="1.0.0", lifespan=lifespan)
# Global variables
engine: Optional[AsyncLLMEngine] = None
engine_ready = False
async def create_engine():
"""Initialize the vLLM engine"""
global engine, engine_ready
try:
# Get model name from environment variable
model_name = os.getenv("MODEL_NAME", "microsoft/DialoGPT-medium")
# Configure engine arguments
engine_args = AsyncEngineArgs(
model=model_name,
tensor_parallel_size=int(os.getenv("TENSOR_PARALLEL_SIZE", "1")),
dtype=os.getenv("DTYPE", "auto"),
trust_remote_code=os.getenv("TRUST_REMOTE_CODE", "true").lower() == "true",
max_model_len=int(os.getenv("MAX_MODEL_LEN")) if os.getenv("MAX_MODEL_LEN") else None,
gpu_memory_utilization=float(os.getenv("GPU_MEMORY_UTILIZATION", "0.9")),
enforce_eager=os.getenv("ENFORCE_EAGER", "false").lower() == "true",
)
# Create the engine
engine = AsyncLLMEngine.from_engine_args(engine_args)
engine_ready = True
logger.info(f"vLLM engine initialized successfully with model: {model_name}")
except Exception as e:
logger.error(f"Failed to initialize vLLM engine: {str(e)}")
engine_ready = False
raise
@app.get("/ping")
async def health_check():
"""Health check endpoint required by Runpod load balancer"""
if not engine_ready:
logger.debug("Health check: Engine initializing")
# Return 503 when initializing
return JSONResponse(
content={"status": "initializing"},
status_code=status.HTTP_204_NO_CONTENT
)
logger.debug("Health check: Engine healthy")
# Return 200 when healthy
return {"status": "healthy"}
@app.get("/")
async def root():
"""Root endpoint with basic info"""
return {
"message": "vLLM Load Balancing Server",
"status": "ready" if engine_ready else "initializing",
"endpoints": {
"health": "/ping",
"generate": "/v1/completions",
"chat": "/v1/chat/completions"
}
}
@app.post("/v1/completions", response_model=GenerationResponse)
async def generate_completion(request: GenerationRequest):
"""Generate text completion"""
logger.info(f"Received completion request: max_tokens={request.max_tokens}, temperature={request.temperature}, stream={request.stream}")
if not engine_ready or engine is None:
logger.warning("Completion request rejected: Engine not ready")
error_response = create_error_response("ServiceUnavailable", "Engine not ready")
raise HTTPException(status_code=503, detail=error_response.model_dump())
try:
# Create sampling parameters
sampling_params = SamplingParams(
max_tokens=request.max_tokens,
temperature=request.temperature,
top_p=request.top_p,
top_k=request.top_k,
frequency_penalty=request.frequency_penalty,
presence_penalty=request.presence_penalty,
stop=request.stop,
)
# Generate request ID
request_id = random_uuid()
if request.stream:
return StreamingResponse(
stream_completion(request.prompt, sampling_params, request_id),
media_type="text/event-stream",
)
else:
# Non-streaming generation
results = engine.generate(request.prompt, sampling_params, request_id)
final_output = None
async for output in results:
final_output = output
if final_output is None:
request_id = random_uuid()
error_response = create_error_response("GenerationError", "No output generated", request_id)
raise HTTPException(status_code=500, detail=error_response.model_dump())
generated_text = final_output.outputs[0].text
finish_reason = final_output.outputs[0].finish_reason
# Calculate token counts using actual token IDs when available
if hasattr(final_output, 'prompt_token_ids') and final_output.prompt_token_ids is not None:
prompt_tokens = len(final_output.prompt_token_ids)
else:
# Fallback to approximate word count
prompt_tokens = len(request.prompt.split())
completion_tokens = len(final_output.outputs[0].token_ids)
logger.info(f"Completion generated: {completion_tokens} tokens, finish_reason={finish_reason}")
return GenerationResponse(
text=generated_text,
finish_reason=finish_reason,
prompt_tokens=prompt_tokens,
completion_tokens=completion_tokens,
total_tokens=prompt_tokens + completion_tokens
)
except Exception as e:
request_id = random_uuid()
logger.error(f"Generation failed (request_id={request_id}): {str(e)}", exc_info=True)
error_response = create_error_response("GenerationError", f"Generation failed: {str(e)}", request_id)
raise HTTPException(status_code=500, detail=error_response.model_dump())
async def stream_completion(prompt: str, sampling_params: SamplingParams, request_id: str) -> AsyncGenerator[str, None]:
"""Stream completion generator"""
try:
results = engine.generate(prompt, sampling_params, request_id)
async for output in results:
for output_item in output.outputs:
yield f"data: {json.dumps({'text': output_item.text, 'finish_reason': output_item.finish_reason})}\n\n"
yield "data: [DONE]\n\n"
except Exception as e:
yield f"data: {json.dumps({'error': str(e)})}\n\n"
@app.post("/v1/chat/completions")
async def chat_completions(request: ChatCompletionRequest):
"""OpenAI-compatible chat completions endpoint"""
logger.info(f"Received chat completion request: {len(request.messages)} messages, max_tokens={request.max_tokens}, temperature={request.temperature}")
if not engine_ready or engine is None:
logger.warning("Chat completion request rejected: Engine not ready")
error_response = create_error_response("ServiceUnavailable", "Engine not ready")
raise HTTPException(status_code=503, detail=error_response.model_dump())
try:
# Extract messages and convert to prompt
messages = request.messages
if not messages:
error_response = create_error_response("ValidationError", "No messages provided")
raise HTTPException(status_code=400, detail=error_response.model_dump())
# Use proper chat template formatting
model_name = os.getenv("MODEL_NAME", "microsoft/DialoGPT-medium")
prompt = format_chat_prompt(messages, model_name)
# Create sampling parameters from request
sampling_params = SamplingParams(
max_tokens=request.max_tokens,
temperature=request.temperature,
top_p=request.top_p,
stop=request.stop,
)
# Generate
request_id = random_uuid()
results = engine.generate(prompt, sampling_params, request_id)
final_output = None
async for output in results:
final_output = output
if final_output is None:
error_response = create_error_response("GenerationError", "No output generated", request_id)
raise HTTPException(status_code=500, detail=error_response.model_dump())
generated_text = final_output.outputs[0].text
completion_tokens = len(final_output.outputs[0].token_ids)
logger.info(f"Chat completion generated: {completion_tokens} tokens, finish_reason={final_output.outputs[0].finish_reason}")
# Return OpenAI-compatible response
return {
"id": request_id,
"object": "chat.completion",
"model": os.getenv("MODEL_NAME", "unknown"),
"choices": [{
"index": 0,
"message": {
"role": "assistant",
"content": generated_text
},
"finish_reason": final_output.outputs[0].finish_reason
}],
"usage": {
"prompt_tokens": len(final_output.prompt_token_ids) if hasattr(final_output, 'prompt_token_ids') and final_output.prompt_token_ids is not None else len(prompt.split()),
"completion_tokens": len(final_output.outputs[0].token_ids),
"total_tokens": (len(final_output.prompt_token_ids) if hasattr(final_output, 'prompt_token_ids') and final_output.prompt_token_ids is not None else len(prompt.split())) + len(final_output.outputs[0].token_ids)
}
}
except Exception as e:
request_id = random_uuid()
logger.error(f"Chat completion failed (request_id={request_id}): {str(e)}", exc_info=True)
error_response = create_error_response("ChatCompletionError", f"Chat completion failed: {str(e)}", request_id)
raise HTTPException(status_code=500, detail=error_response.model_dump())
if __name__ == "__main__":
# Get ports from environment variables
port = int(os.getenv("PORT", 80))
logger.info(f"Starting vLLM server on port {port}")
# If health port is different, you'd need to run a separate health server
# For simplicity, we're using the same port here
uvicorn.run(
app,
host="0.0.0.0",
port=port,
log_level="info"
)
```
This file creates a FastAPI server that manages the vLLM engine and exposes three API endpoints:
* A health check at `/ping` that tells the load balancer when your worker is ready.
* A text completion endpoint at `/v1/completions`.
* An OpenAI-compatible chat endpoint at `/v1/chat/completions`.
The application handles both streaming and non-streaming responses, manages the language model lifecycle, and includes comprehensive error handling and logging.
## Step 5: Set up dependencies and build steps
With the application code complete, we still need to define its dependencies and create a Dockerfile to package it into a container image.
1. Add the following dependencies to `requirements.txt`:
```
ray
pandas
pyarrow
runpod~=1.7.0
huggingface-hub
packaging
typing-extensions==4.7.1
pydantic
pydantic-settings
hf-transfer
transformers<4.54.0
```
2. Add the following build steps to your `Dockerfile`:
```dockerfile theme={"theme":{"light":"github-light","dark":"github-dark"}}
FROM nvidia/cuda:12.1.0-base-ubuntu22.04
RUN apt-get update -y \
&& apt-get install -y python3-pip
RUN ldconfig /usr/local/cuda-12.1/compat/
# Install Python dependencies
RUN --mount=type=cache,target=/root/.cache/pip \
python3 -m pip install --upgrade pip && \
python3 -m pip install --upgrade -r /requirements.txt
# Pin vLLM version for stability - 0.9.1 is latest stable as of 2024-07
# FlashInfer provides optimized attention for better performance
ARG VLLM_VERSION=0.9.1
ARG CUDA_VERSION=cu121
ARG TORCH_VERSION=torch2.3
RUN python3 -m pip install vllm==${VLLM_VERSION} && \
python3 -m pip install flashinfer -i https://flashinfer.ai/whl/${CUDA_VERSION}/${TORCH_VERSION}
ENV PYTHONPATH="/:/vllm-workspace"
COPY src /src
WORKDIR /src
CMD ["python3", "handler.py"]
```
## Step 6: Build and push your Docker image
Build and push your Docker image to a container registry:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
# Build the image
docker build --platform linux/amd64 -t YOUR_DOCKER_USERNAME/vllm-loadbalancer:v1.0 .
# Push to Docker Hub
docker push YOUR_DOCKER_USERNAME/vllm-loadbalancer:v1.0
```
## Step 7: Deploy to Runpod
Now, let's deploy our application to a Serverless endpoint:
1. Go to the [Serverless page](https://console.runpod.io/serverless) in the Runpod console.
2. Click **New Endpoint**
3. Click **Import from Docker Registry**.
4. In the **Container Image** field, enter your Docker image URL:
```
YOUR_DOCKER_USERNAME/vllm-loadbalancer:v1.0
```
Then click **Next**.
5. Give your endpoint a name.
6. Under **Endpoint Type**, select **Load Balancer**.
7. Under **GPU Configuration**, select at least one GPU type (16 GB or 24 GB GPUs are fine for this example).
8. Leave all other settings at their defaults.
9. Click **Create Endpoint**.
## Step 8: Test your endpoints
You can find a Python script to test your vLLM load balancer locally [on GitHub](https://github.com/runpod-workers/vllm-loadbalancer-ep/blob/main/example.py).
Once your endpoint has finished deploying, you can access your vLLM APIs at:
```
https://ENDPOINT_ID.api.runpod.ai/PATH
```
For example, the vLLM application we defined in step 4 exposes these endpoints:
* Health check: `https://ENDPOINT_ID.api.runpod.ai/ping`
* Generate text: `https://ENDPOINT_ID.api.runpod.ai/v1/completions`
* Chat completions: `https://ENDPOINT_ID.api.runpod.ai/v1/chat/completions`
Use the curl commands below to make test requests to your vLLM load balancer, replacing `ENDPOINT_ID` and `RUNPOD_API_KEY` with your actual values.
To run a health check:
```bash ping theme={"theme":{"light":"github-light","dark":"github-dark"}}
curl -X GET "https://ENDPOINT_ID.api.runpod.ai/ping" \
-H 'Authorization: Bearer RUNPOD_API_KEY' \
-H "Content-Type: application/json" \
```
For text completions:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
curl -X POST "https://ENDPOINT_ID.api.runpod.ai/v1/completions" \
-H 'Authorization: Bearer RUNPOD_API_KEY' \
-H 'Content-Type: application/json' \
-d '{"prompt": "Once upon a time", "max_tokens": 50, "temperature": 0.8}'
```
For chat completions:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
curl -X POST "https://ENDPOINT_ID.api.runpod.ai/v1/chat/completions" \
-H 'Authorization: Bearer RUNPOD_API_KEY' \
-H 'Content-Type: application/json' \
-d '{
"messages": [
{"role": "user", "content": "Tell me a short story"}
],
"max_tokens": 100,
"temperature": 0.8
}'
```
After sending a request, your workers will take some time to initialize. You can track their progress by checking the logs in the **Workers** tab of your endpoint page.
If you see: `{"error":"no workers available"}%` after running the request, this means your workers did not initialize in time to process it. If you try running the request again, this will usually resolve the issue.
For production applications, implement a health check with retries before sending requests. See [Handling cold start errors](/serverless/load-balancing/overview#handling-cold-start-errors) for a complete code example.
Congratulations! You've created a load balancing vLLM endpoint and used it to serve a large language model.
## Next steps
Now that you've deployed a load balancing vLLM endpoint, you can try:
* Experimenting with different models and frameworks.
* Adding authentication to your API.
* Exploring advanced FastAPI features like background tasks and WebSockets.
* Optimizing your application for performance and reliability.
# Worker affinity
Source: https://docs.runpod.io/serverless/load-balancing/worker-affinity
Pin follow-up requests to a specific worker using the X-Runpod-Worker-Id header. Review configuration and operations guidance for Runpod Serverless.
Worker affinity lets clients route follow-up requests to the same worker that served an earlier request. This is useful for stateful workloads where a worker holds session state in memory and re-routing to a different worker would require reloading it.
## How it works
Every response from the load balancer includes an `X-Runpod-Worker-Id` header that identifies the worker that handled the request:
```
X-Runpod-Worker-Id:
```
To pin a subsequent request to that worker, send the header back with one of the following values:
| Header value | Behavior |
| --------------------------- | -------------------------------------------------------------------------------------------------------------------------------- |
| *(omitted or empty)* | Normal routing — normal worker selection, no preference |
| `` | **Soft affinity**: prefer the specified worker; fall back to normal worker selection if the worker is unavailable or at capacity |
| `strict ` | **Strict affinity**: only use the specified worker; wait if the worker is at capacity, return `404` if the worker is gone |
| `strict-resume ` | **Strict-resume affinity**: same as strict, but also resumes the worker pod if it has been scaled down |
## Affinity modes
### Soft affinity
```
X-Runpod-Worker-Id: pod-abc123
```
* Routes to the specified worker if available.
* Falls back to normal worker selection if the worker is unavailable, at capacity, or not found.
* Never fails a request due to affinity — this mode is best-effort.
### Strict affinity
```
X-Runpod-Worker-Id: strict pod-abc123
```
* Routes only to the specified worker.
* If the worker is at capacity, the request waits up to \~5 minutes.
* If the worker does not free up in time, returns `400` with reason `timed out waiting for worker`.
* If the worker is gone, returns `404` with reason `affinity_worker_gone`.
* Use this mode when falling back to a different worker would produce incorrect results.
### Strict-resume affinity
```
X-Runpod-Worker-Id: strict-resume pod-abc123
```
* Same as strict, but also resumes a scaled-down worker before routing.
* If the resume cannot complete immediately, it keeps retrying while the request waits.
* Returns `400` on timeout, or `404` if the pod is truly gone (terminated or redeployed under a new ID).
* Cross-endpoint access is not possible.
`strict-resume` keeps a worker running as long as requests keep coming in. Workers scale down after being idle (no requests) for a set period. If your client sends a request before that idle period expires, the timer resets and the worker stays up. This means a worker receiving regular traffic, with or without strict-resume, will never scale down.
When your session is done, send your next request without the `X-Runpod-Worker-Id` header so the worker is no longer pinned and can go idle.
## Worker ID on responses
The load balancer always sets `X-Runpod-Worker-Id` on the response to reflect the worker that actually served the request. Use the value from the response header for pinning rather than tracking it separately.
## Error reference
| Status | Reason | Description |
| ------ | ---------------------- | ------------------------------------------------------------------------------------------------------------ |
| `404` | `affinity_worker_gone` | Strict or strict-resume: the requested worker is not in the active worker set |
| `400` | `worker_timeout` | Strict or strict-resume: the pinned worker did not become available within the request timeout (\~5 minutes) |
# Overview
Source: https://docs.runpod.io/serverless/overview
Pay-as-you-go compute for AI models and compute-intensive workloads. Review configuration and operations guidance for Runpod Serverless.
Runpod Serverless is a cloud computing platform that lets you serve AI models for and run other compute-intensive workloads without managing servers. You only pay for the actual compute time you use, with no idle costs when your application isn't processing requests.
## Get started
Write a handler function, build a worker image, create an endpoint, and send your first request.
Deploy a ComfyUI worker and generate images using JSON workflows.
Use Runpod's worker templates on GitHub as a starting point.
## Concepts
### [Endpoints](/serverless/endpoints/overview)
The access point for your Serverless application. Endpoints provide a URL where users or applications can send requests to run your code. Each endpoint can be configured with different compute resources, scaling settings, and other parameters to suit your specific needs.
### [Workers](/serverless/workers/overview)
The container instances that execute your code when requests arrive at your endpoint. Each worker runs your custom Docker container with your application code and dependencies. Runpod automatically manages worker lifecycle, starting them when needed and stopping them when idle to optimize resource usage.
### [Handler functions](/serverless/workers/handler-functions)
The core of your Serverless application. These functions define how a worker processes incoming requests and returns results. They follow a simple pattern:
```Python theme={"theme":{"light":"github-light","dark":"github-dark"}}
import runpod # Required
def handler(event):
# Extract input data from the request
input_data = event["input"]
# Process the input (replace this with your own code)
result = process_data(input_data)
# Return the result
return result
runpod.serverless.start({"handler": handler}) # Required
```
Handler functions are only used for (i.e. traditional endpoints). If you're using a , the request structure and endpoints will depend on how you define your HTTP servers.
### [Requests](/serverless/endpoints/send-requests)
An HTTP request that you send to an endpoint, which can include parameters, payloads, and headers that define what the endpoint should process. For example, you can send a `POST` request to submit a , or a `GET` request to check status of a job, retrieve results, or check endpoint health.
When a user/client sends a request to your endpoint:
1. If no workers are active, Runpod automatically starts one (cold start).
2. The request is queued until a worker is available.
3. A worker processes the request using your handler function.
4. The result is returned to the user/client after they call `/status` (or automatically if you used `/runsync`).
5. Workers remain active for a period to handle additional requests.
6. Idle workers eventually shut down if no new requests arrive.
```mermaid theme={"theme":{"light":"github-light","dark":"github-dark"}}
%%{init: {'theme':'base', 'themeVariables': { 'primaryColor':'#9289FE','primaryTextColor':'#fff','primaryBorderColor':'#9289FE','lineColor':'#5F4CFE','secondaryColor':'#AE6DFF','tertiaryColor':'#FCB1FF','edgeLabelBackground':'#5F4CFE', 'fontSize':'15px','fontFamily':'font-inter'}}}%%
flowchart TD
Start([Endpoint receives a request]) --> Queue[Request queued]
Queue --> CheckActive{Worker ready?}
CheckActive -->|" No "| ColdStart[Start new worker if workers < max_workers]
ColdStart --> WaitWorker[Wait for available worker]
WaitWorker --> Process[Worker processes request using handler function]
CheckActive -->|" Yes "| Process
Process --> End(["Result is stored (/run) or returned immediately (/runsync)"])
style Start fill:#5F4CFE,stroke:#5F4CFE,color:#FFFFFF,stroke-width:2px
style Queue fill:#4D38F5,stroke:#4D38F5,color:#FFFFFF,stroke-width:2px
style CheckActive fill:#f87171,stroke:#f87171,color:#000000,stroke-width:2px
style ColdStart fill:#fb923c,stroke:#fb923c,color:#000000,stroke-width:2px
style WaitWorker fill:#ecc94b,stroke:#ecc94b,color:#000000,stroke-width:2px
style Process fill:#22C55E,stroke:#22C55E,color:#000000,stroke-width:2px
style End fill:#5F4CFE,stroke:#5F4CFE,color:#FFFFFF,stroke-width:2px
linkStyle default stroke-width:2px
```
### Cold starts
A "cold start" refers to the time between when an endpoint with no running workers receives a request, and when a worker is fully "warmed up" and ready to handle the request. This generally involves starting the container, loading models into GPU memory, and initializing runtime environments. Larger models take longer to load into memory, increasing cold start time, and request response time by extension.
Minimizing cold start times is key to creating a responsive and cost-effective endpoint. You can reduce cold starts by using [cached models](/serverless/endpoints/model-caching), enabling [FlashBoot](/serverless/endpoints/endpoint-configurations#flashboot), setting [active worker counts](/serverless/endpoints/endpoint-configurations#active-min-workers) above zero.
### [Load balancing endpoints](/serverless/load-balancing/overview)
These endpoints route incoming traffic directly to available workers, distributing requests across the worker pool. Unlike , they provide no queuing mechanism for request backlog.
When using load balancing endpoints, you can define your own custom API endpoints without a handler function, using any HTTP framework of your choice (like FastAPI or Flask).
### Fitness checks and preflight checks
Validate your worker's environment at startup before it takes traffic. Fitness checks, also known as preflight checks, run registered checks in order before a worker begins processing jobs, catching issues like missing GPUs, unloaded models, or bad configuration before any request reaches your worker.
[Learn more about fitness checks and preflight checks](/serverless/development/fitness-checks)
## Development workflow
Here's a typical Serverless development workflow:
1. [Write a handler function](/serverless/workers/handler-functions) to process API requests.
2. [Test it locally](/serverless/development/local-testing) using the Runpod SDK.
3. [Create a Dockerfile](/serverless/workers/create-dockerfile) to package the handler function and all its dependencies.
4. [Build and push](/serverless/workers/deploy) the worker image to Docker Hub (or another container registry).
* ... or [deploy directly from a GitHub repository](/serverless/workers/github-integration).
5. Deploy the worker image to a [Serverless endpoint](/serverless/endpoints/overview).
6. [Monitor logs](/serverless/development/logs), debug running workers [with SSH](/serverless/development/ssh-into-workers).
7. Adjust your [endpoint settings](/serverless/endpoints/endpoint-configurations) to [optimize performance and cost](/serverless/development/optimization).
8. To update your endpoint logic, go back to step 1 and repeat the process.
```mermaid theme={"theme":{"light":"github-light","dark":"github-dark"}}
%%{init: {'theme':'base', 'themeVariables': { 'primaryColor':'#9289FE','primaryTextColor':'#fff','primaryBorderColor':'#9289FE','lineColor':'#5F4CFE','secondaryColor':'#AE6DFF','tertiaryColor':'#FCB1FF','edgeLabelBackground':'#5F4CFE', 'fontSize':'15px','fontFamily':'font-inter'}}}%%
flowchart TD
Start([Write handler function]) --> Test[Test handler locally with the Runpod SDK]
Test --> Check{Tests pass?}
Check -->|" No "| Fix[Fix code & debug]
Fix --> Test
Check -->|" Yes "| Package[Package worker as a Docker image]
Package --> Deploy[Deploy worker image to Runpod Serverless]
subgraph Production [Production Environment]
Deploy --> Running[Workers auto-scale based on demand]
Running --> Monitor[Monitor logs and metrics]
end
Monitor -.-> Start
%% Applied Color Logic from Diagram 2
style Start fill:#5F4CFE,stroke:#5F4CFE,color:#FFFFFF,stroke-width:2px
style Test fill:#4D38F5,stroke:#4D38F5,color:#FFFFFF,stroke-width:2px
style Check fill:#f87171,stroke:#f87171,color:#000000,stroke-width:2px
style Fix fill:#ecc94b,stroke:#ecc94b,color:#000000,stroke-width:2px
style Package fill:#5F4CFE,stroke:#5F4CFE,color:#FFFFFF,stroke-width:2px
style Deploy fill:#5F4CFE,stroke:#5F4CFE,color:#FFFFFF,stroke-width:2px
style Running fill:#5F4CFE,stroke:#5F4CFE,color:#FFFFFF,stroke-width:2px
style Monitor fill:#ecc94b,stroke:#ecc94b,color:#000000,stroke-width:2px
%% Subgraph Styling
style Production fill:#1B0656,stroke:#5F4CFE,color:#FFFFFF,stroke-dasharray: 5 5
linkStyle default stroke-width:2px,stroke:#5F4CFE
```
# Pricing
Source: https://docs.runpod.io/serverless/pricing
Learn how Serverless billing works to optimize your costs. Review setup, configuration, deployment, and operations guidance for Runpod Serverless.
Runpod offers custom pricing plans for large scale and enterprise workloads. [Contact our sales team](https://ecykq.share.hsforms.com/2MZdZATC3Rb62Dgci7knjbA) to learn more.
Serverless offers pay-per-second pricing with no upfront costs. You're billed from when a worker starts until it fully stops, rounded up to the nearest second.
## Worker types
| | Flex workers | Active workers |
| ------------ | ------------------------------------- | -------------------------------------------- |
| **Behavior** | Scale to zero when idle | Always running (24/7) |
| **Pricing** | Standard per-second rate | Discounts available through sales inquiry |
| **Best for** | Variable workloads, cost optimization | Consistent traffic, low-latency requirements |
## What you're billed for
Your total cost includes compute time and storage:
| Cost component | Description | Rate |
| ------------------ | -------------------------------- | ------------------------------------------------------------ |
| **Compute** | GPU time while workers run | See the [Runpod pricing page](https://www.runpod.io/pricing) |
| **Container disk** | Worker storage (5-min intervals) | \~\$0.10/GB/month |
| **Network volume** | Shared persistent storage | \$0.07/GB/month (\< 1TB), \$0.05/GB/month (> 1TB) |
### Compute cost breakdown
Workers incur charges during three phases:
1. **Start time**: Initializing the container and loading models into GPU memory. Minimize with [FlashBoot](/serverless/endpoints/endpoint-configurations#flashboot) or [model caching](/serverless/endpoints/model-caching).
2. **Execution time**: Processing requests. Set [execution timeouts](/serverless/endpoints/endpoint-configurations#execution-timeout) to prevent runaway jobs.
3. **Idle timeout duration**: The time a worker remains active (running) after completing a request, waiting for additional requests before scaling down (default: 5 seconds). Configure in [endpoint settings](/serverless/endpoints/endpoint-configurations#idle-timeout).
For high-volume workloads with significant storage needs, use [network volumes](/storage/network-volumes) to share data across workers and reduce per-worker storage costs.
## Account limits
**Spend limit**: Default limit of \$80/hour across all resources. [Contact support](https://www.runpod.io/contact) to increase.
## Billing support
If you believe you've been billed incorrectly, [contact support](https://www.runpod.io/contact), including the following information in your ticket:
* Endpoint ID
* Request ID (if applicable)
* Approximate time of the issue
# Quickstart
Source: https://docs.runpod.io/serverless/quickstart
Write a handler function, build a worker image, create an endpoint, and send your first request. See Runpod Serverless setup and usage details.
For an even faster start, clone or download the [worker-basic](https://github.com/runpod-workers/worker-basic) repository for a pre-configured template for building and deploying Serverless workers. After cloning the repository, skip to [step 6 of this tutorial](#step-6%3A-build-and-push-your-docker-image) to deploy and test the endpoint.
## Requirements
* You've [created a Runpod account](/accounts-billing/manage-accounts).
* You've installed [Python 3.10 or higher](https://www.python.org/downloads/) and [Docker](https://docs.docker.com/get-started/get-docker/) on your local machine and configured them for your command line.
## Step 1: Create project files
Create a new directory with empty files for your project:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
mkdir serverless-quickstart && cd serverless-quickstart
touch handler.py Dockerfile requirements.txt test_input.json
```
## Step 2: Install the Serverless SDK
Create a virtual environment and install the Serverless SDK
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
python3 -m venv .venv
source .venv/bin/activate
pip install runpod
```
## Step 3: Create a handler function
Add the following code to `handler.py`:
```python handler.py theme={"theme":{"light":"github-light","dark":"github-dark"}}
import runpod
import time
def handler(event):
# This function processes incoming requests to your Serverless endpoint.
#
# Args:
# event (dict): Contains the input data and request metadata
#
# Returns:
# Any: The result to be returned to the client
# Extract input data
print(f"Worker Start")
input = event['input']
prompt = input.get('prompt')
seconds = input.get('seconds', 0)
print(f"Received prompt: {prompt}")
print(f"Sleeping for {seconds} seconds...")
# You can replace this sleep call with your own Python code
time.sleep(seconds)
return prompt
# Start the Serverless function when the script is run
if __name__ == '__main__':
runpod.serverless.start({'handler': handler })
```
This is a bare-bones handler that processes a JSON object and outputs a `prompt` string contained in the `input` object.
You can replace the `time.sleep(seconds)` call with your own Python code for generating images, text, or running any AI/ML workload.
## Step 4: Create a test input file
Add the following code to `test_input.json` to properly test your handler locally:
```json test_input.json theme={"theme":{"light":"github-light","dark":"github-dark"}}
{
"input": {
"prompt": "Hey there!"
}
}
```
## Step 5: Test your handler function locally
Run your handler function using your local terminal:
```sh theme={"theme":{"light":"github-light","dark":"github-dark"}}
python handler.py
```
You should see output similar to this:
```text theme={"theme":{"light":"github-light","dark":"github-dark"}}
--- Starting Serverless Worker | Version 1.7.9 ---
INFO | Using test_input.json as job input.
DEBUG | Retrieved local job: {'input': {'prompt': 'Hey there!'}, 'id': 'local_test'}
INFO | local_test | Started.
Worker Start
Received prompt: Hey there!
Sleeping for 0 seconds...
DEBUG | local_test | Handler output: Hey there!
DEBUG | local_test | run_job return: {'output': 'Hey there!'}
INFO | Job local_test completed successfully.
INFO | Job result: {'output': 'Hey there!'}
INFO | Local testing complete, exiting.
```
## Step 6: Create a Dockerfile
Add the following content to `Dockerfile`:
New to Dockerfiles? Learn the fundamentals with our [introduction to containers](/tutorials/introduction/containers) tutorial series.
```dockerfile Dockerfile theme={"theme":{"light":"github-light","dark":"github-dark"}}
FROM python:3.10-slim
WORKDIR /
# Install dependencies
RUN pip install --no-cache-dir runpod
# Copy your handler file
COPY handler.py /
# Start the container
CMD ["python3", "-u", "handler.py"]
```
## Step 7: Build and push your worker image
Instead of building and pushing your image via Docker Hub, you can also [deploy your worker from a GitHub repository](/serverless/workers/github-integration).
Before you can deploy your worker on Runpod Serverless, you need to push it to Docker Hub:
Build your Docker image, specifying the platform for Runpod deployment, replacing `[YOUR_USERNAME]` with your Docker username:
```sh theme={"theme":{"light":"github-light","dark":"github-dark"}}
docker build --platform linux/amd64 --tag [YOUR_USERNAME]/serverless-test .
```
```sh theme={"theme":{"light":"github-light","dark":"github-dark"}}
docker push [YOUR_USERNAME]/serverless-test:latest
```
## Step 8: Deploy your worker using the Runpod console
To deploy your worker to a Serverless endpoint:
1. Go to the [Serverless section](https://www.console.runpod.io/serverless) of the Runpod console.
2. Click **New Endpoint**.
3. Click **Import from Docker Registry**
4. In the **Container Image** field, enter your Docker image URL: `docker.io/yourusername/serverless-test:latest`.
5. Click **Next** to proceed to endpoint configuration.
6. Configure your endpoint settings:
* (Optional) Enter a custom name for your endpoint, or use the randomly generated name.
* Make sure the **Endpoint Type** is set to **Queue**.
* Under **GPU Configuration**, check the box for **16 GB** GPUs.
* Leave the rest of the settings at their defaults.
7. Click **Deploy Endpoint**.
The system will redirect you to a dedicated detail page for your new endpoint.
## Step 9: Test your endpoint
To test your endpoint, click the **Requests** tab in the endpoint detail page:
On the left you should see the default test request:
```json theme={"theme":{"light":"github-light","dark":"github-dark"}}
{
"input": {
"prompt": "Hello World"
}
}
```
Leave the default input as is and click **Run**. The system will take a few minutes to initialize your workers.
When the workers finish processing your request, you should see output on the right side of the page similar to this:
```json theme={"theme":{"light":"github-light","dark":"github-dark"}}
{
"delayTime": 15088,
"executionTime": 60,
"id": "04f01223-4aa2-40df-bdab-37e5caa43cbe-u1",
"output": "Hello World",
"status": "COMPLETED",
"workerId": "uhbbfre73gqjwh"
}
```
Congratulations! You've successfully deployed and tested your first Serverless endpoint.
## Next steps
Create more advanced handler functions.
Add AI/ML models and other dependencies to your worker.
Learn how to structure and send requests to your endpoint.
Configure and manage your Serverless endpoints.
# Install the Runpod Serverless SDK
Source: https://docs.runpod.io/serverless/sdks
Install and configure the Serverless SDK for Python, JavaScript, or Go to interact with Serverless endpoints programmatically.
## Python
The Python SDK requires Python 3.10 or higher.
### Install
Create a virtual environment and install the SDK:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
python3 -m venv venv
source venv/bin/activate
pip install runpod
```
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
python -m venv venv
venv\Scripts\activate
pip install runpod
```
To verify the installation:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
python -c "import runpod; print(runpod.__version__)"
```
### Configure your API key
Set your API key as an environment variable and reference it in your code:
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
import runpod
import os
runpod.api_key = os.getenv("RUNPOD_API_KEY")
```
Never hardcode your API key directly in your code. Always use environment variables or a secrets manager.
For more information, see the [Python SDK on GitHub](https://github.com/runpod/runpod-python).
## JavaScript
The JavaScript SDK supports Node.js and modern JavaScript environments using ES modules.
### Install
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
npm install --save runpod-sdk
# or
yarn add runpod-sdk
```
### Configure your API key
```javascript theme={"theme":{"light":"github-light","dark":"github-dark"}}
import runpodSdk from "runpod-sdk";
const { RUNPOD_API_KEY, ENDPOINT_ID } = process.env;
const runpod = runpodSdk(RUNPOD_API_KEY);
const endpoint = runpod.endpoint(ENDPOINT_ID);
```
For more information, see the [JavaScript SDK on GitHub](https://github.com/runpod/js-sdk) and the [npm package](https://www.npmjs.com/package/runpod-sdk).
## Go
### Install
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
go get github.com/runpod/go-sdk
go mod tidy
```
### Configure your API key
```go theme={"theme":{"light":"github-light","dark":"github-dark"}}
package main
import (
"os"
"github.com/runpod/go-sdk/pkg/sdk/config"
rpEndpoint "github.com/runpod/go-sdk/pkg/sdk/endpoint"
)
func main() {
apiKey := os.Getenv("RUNPOD_API_KEY")
endpointId := os.Getenv("ENDPOINT_ID")
endpoint, err := rpEndpoint.New(
&config.Config{ApiKey: &apiKey},
&rpEndpoint.Option{EndpointId: &endpointId},
)
if err != nil {
panic(err)
}
// Use the endpoint...
}
```
For more information, see the [Go SDK on GitHub](https://github.com/runpod/go-sdk) and the [Go package documentation](https://pkg.go.dev/github.com/runpod/go-sdk/pkg/sdk).
## Use the SDKs
For detailed examples of how to use the SDKs to interact with Serverless endpoints, see [Send API requests](/serverless/endpoints/send-requests).
## Next steps
Once you've installed and configured the SDK, you're ready to send requests to your Serverless endpoints:
* [Send API requests](/serverless/endpoints/send-requests): Learn how to submit jobs, check status, and stream results.
* [Create handler functions](/serverless/workers/handler-functions): Build custom workers to process your jobs.
* [API reference](/api-reference/overview): Explore the full Runpod API for managing Pods, endpoints, and other resources.
# Storage options
Source: https://docs.runpod.io/serverless/storage/overview
Storage options for Serverless workers: container disks, network volumes, and S3-compatible storage.
## Storage types
### Container disk
Temporary storage that exists only while a worker is running. Data is lost when the worker stops or scales down. Fast read/write speeds since storage is locally attached. Cost is included in the worker's running cost.
All data saved by a worker's [handler function](/serverless/workers/handler-functions) is stored in the container disk by default. To persist data beyond the current worker session, use a network volume or S3-compatible storage.
### Network volume
Persistent storage that can be attached to multiple workers. Ideal for sharing datasets, storing large models, and preserving data beyond individual worker sessions. Available in Standard and [High-Performance](/storage/high-performance-storage) tiers.
See [Network volumes for Serverless](/storage/network-volumes#network-volumes-for-serverless).
### S3-compatible storage
Connect to external object storage (AWS S3, MinIO, Backblaze B2, DigitalOcean Spaces, etc.) using your own credentials. Useful for large files exceeding API payload limits. Storage exists outside Runpod infrastructure with billing based on your provider.
See [S3-compatible storage](/serverless/endpoints/send-requests#s3-compatible-storage).
## Comparison
| Feature | Container Disk | Network Volume | S3-Compatible Storage |
| --------------- | ------------------------ | ---------------------------- | ---------------------------- |
| **Persistence** | Temporary (lost on stop) | Permanent | Permanent (external) |
| **Sharing** | Not shareable | Multi-worker | Via S3 credentials |
| **Speed** | Fastest (local) | Fast (networked NVMe) | Varies by provider |
| **Cost** | Included in worker cost | \$0.05-\$0.07/GB/month | Varies by provider |
| **Best for** | Temporary processing | Multi-worker sharing, models | Large files, external access |
## Behavior notes
* **Data isolation**: Workers don't share data unless a network volume is attached.
* **Caching**: Docker images cache locally on container disk, but loading large models into GPU memory still impacts cold start times. See [Reducing worker startup times](/serverless/endpoints/endpoint-configurations#reducing-worker-startup-times).
* **Location constraints**: Network volumes constrain deployments to the volume's data center, which may impact GPU availability.
# Troubleshooting
Source: https://docs.runpod.io/serverless/troubleshooting
Common issues and solutions for Serverless endpoints and workers. Review configuration and operations guidance for Runpod Serverless.
## Deployment issues
### Worker fails to start
If your worker fails to start or initialize:
1. **Check logs**: View endpoint logs in the [Runpod console](https://www.console.runpod.io/serverless) for error messages.
2. **Verify local testing**: Ensure your handler works in [local testing](/serverless/development/local-testing) before deploying.
3. **Check dependencies**: Verify all dependencies are installed in your [Docker image](/serverless/workers/create-dockerfile).
4. **GPU compatibility**: Ensure your Docker image is compatible with the selected GPU type.
5. **Input format**: Verify your [input format](/serverless/endpoints/send-requests) matches what your handler expects.
### Worker initializes but fails on requests
| Issue | Solution |
| ----------------------- | --------------------------------------------------------------------------- |
| Input validation errors | Add input validation in your handler and check logs for the expected format |
| Missing dependencies | Verify all required packages are in your Dockerfile |
| Model loading failures | Check GPU memory requirements and model path |
| Permission errors | Ensure files are readable and directories are writable |
## Job issues
### Jobs stuck in queue
If jobs remain `IN_QUEUE` for extended periods:
* **No workers available**: Check if `max_workers` is set appropriately.
* **Workers throttled**: Your endpoint may be hitting rate limits. Check the Workers tab for throttled workers.
* **Cold start delays**: First requests after idle periods require worker initialization. Consider increasing `min_workers` or enabling [FlashBoot](/serverless/endpoints/endpoint-configurations#flashboot).
### Jobs funneled to a single worker
If your endpoint has multiple workers but nearly all jobs run on one worker while the others sit idle, and jobs stay `IN_QUEUE` even though workers are available, you may be running an affected version of the Runpod Python SDK.
Versions 1.7.11 through 1.10.0 could corrupt per-worker job tracking on endpoints that use a network volume, causing most workers to stop pulling new jobs. This most often appeared on network volume endpoints such as ComfyUI workers.
To fix this, upgrade the Runpod Python SDK to version 1.10.1 or later:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
pip install --upgrade "runpod>=1.10.1"
```
Then rebuild and redeploy your worker image so the fix takes effect. No configuration or code changes are required.
### Jobs timing out
| Cause | Solution |
| ------------------------- | -------------------------------------------------------------------------------------------------------- |
| Processing takes too long | Increase `executionTimeout` in your [job policy](/serverless/endpoints/send-requests#execution-policies) |
| Model loading too slow | Use [model caching](/serverless/endpoints/model-caching) or bake models into your image |
| TTL too short | Set `ttl` to cover both queue time and execution time |
### Jobs failing
Check the job status response for error details. Common causes:
* **Handler exceptions**: Unhandled exceptions in your handler code. Add try/catch blocks and return structured errors.
* **OOM (Out of Memory)**: Model or batch size exceeds GPU memory. Reduce batch size or use a larger GPU.
* **Timeout**: Job exceeded execution timeout. Increase timeout or optimize processing.
## Endpoint scaling issues
### My endpoint was scaled down unexpectedly
If your endpoint's max workers dropped without any change on your end, Runpod scaled the endpoint down automatically. This happens in two situations:
* **Prolonged inactivity**: When an endpoint receives no requests for 3 days, its max workers is reduced to 2, and after 7 days its max workers is set to 0. Runpod emails you when the first reduction happens. For more details, see [idle endpoint scale-down](/serverless/endpoints/endpoint-configurations#idle-endpoint-scale-down).
* **Repeated unhealthy workers**: When an endpoint consistently produces unhealthy (crashing) workers, Runpod scales it down to stop billing and reduce thrashing, and sends you an email.
To bring the endpoint back, increase its max workers in the [Runpod console](https://www.console.runpod.io/serverless). If the scale-down was caused by unhealthy workers, fix the underlying problem first, or the endpoint may be scaled down again. Check the [logs](/serverless/development/logs) for crash errors, and verify your worker using [local testing](/serverless/development/local-testing).
## Cold start issues
### Slow cold starts
Cold start time includes container startup, model loading, and initialization. To reduce cold starts:
1. **Use model caching**: Store models on [network volumes](/serverless/endpoints/model-caching) instead of downloading on each start.
2. **Enable FlashBoot**: Use [FlashBoot](/serverless/endpoints/endpoint-configurations#flashboot) for faster container initialization.
3. **Optimize image size**: Use smaller base images and remove unnecessary dependencies.
4. **Initialize outside handler**: Load models at module level, not inside the handler function.
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
# Good: Load model once at startup
model = load_model()
def handler(job):
return model.predict(job["input"])
# Bad: Load model on every request
def handler(job):
model = load_model() # Slow!
return model.predict(job["input"])
```
### Too many cold starts
If you're seeing frequent cold starts:
* **Increase idle timeout**: Set a longer `idle_timeout` to keep workers warm between requests.
* **Set minimum workers**: Configure `min_workers` > 0 to maintain warm workers.
* **Check traffic patterns**: Sporadic traffic causes more cold starts than steady traffic.
## Logging issues
### Missing logs
If logs aren't appearing in the console:
1. **Check throttling**: Excessive logging triggers throttling. Reduce log verbosity.
2. **Verify output streams**: Ensure you're writing to stdout/stderr, not just files.
3. **Check worker status**: Logs only appear for successfully initialized workers.
4. **Retention period**: Logs older than 90 days are automatically removed.
### Log throttling
To avoid log throttling:
* Reduce log verbosity in production.
* Use structured logging for efficiency.
* Store detailed logs on [network volumes](/serverless/storage/overview) instead of console output.
## vLLM-specific issues
### OOM errors
If your vLLM worker runs out of memory:
* Lower `GPU_MEMORY_UTILIZATION` from 0.90 to 0.85.
* Reduce `MAX_MODEL_LEN` to limit context window.
* Use a GPU with more VRAM.
### Model not loading
| Issue | Solution |
| ------------------------- | ------------------------------------------------------------------------------------------ |
| Model not found | Verify `MODEL_NAME` matches the Hugging Face model ID exactly |
| Gated model access denied | Set `HF_TOKEN` with a token that has access to the model |
| Incompatible model | Check [vLLM supported models](https://docs.vllm.ai/en/latest/models/supported_models.html) |
### OpenAI API errors
| Error | Cause | Solution |
| ------------------ | ------------------ | --------------------------------------------------------------- |
| 401 Unauthorized | Invalid API key | Verify `RUNPOD_API_KEY` is correct |
| 404 Not Found | Wrong endpoint URL | Use the format `https://api.runpod.ai/v2/ENDPOINT_ID/openai/v1` |
| Connection refused | Endpoint not ready | Wait for workers to initialize |
## Load balancing endpoint issues
### "No workers available" error
This means workers didn't initialize in time. Common causes:
* **First request**: Workers need time to start. Retry the request. (See [Handling cold starts](/serverless/load-balancing/overview#handling-cold-starts) for more information.)
* **All workers busy**: Increase `max_workers` to handle more concurrent requests.
* **Workers crashing**: Check logs for initialization errors.
### Requests not reaching workers
Verify your HTTP server is:
* Listening on port 8000 (or the port specified in your configuration).
* Binding to `0.0.0.0`, not `127.0.0.1`.
* Returning proper HTTP responses.
## Getting help
If you're still experiencing issues:
1. **Check endpoint logs** for detailed error messages.
2. **SSH into workers** using [SSH access](/serverless/development/ssh-into-workers) to debug in real-time.
3. **Review metrics** in the Metrics tab to identify patterns.
4. **Contact support** at [help@runpod.io](mailto:help@runpod.io) with your endpoint ID and error details.
# Configure vLLM to work with your model
Source: https://docs.runpod.io/serverless/vllm/configuration
Learn how to set up vLLM endpoints to work with your chosen model. Review configuration and operations guidance for Runpod Serverless.
Most LLMs need specific configuration to run properly on vLLM. Default settings work for some models, but many require custom tokenization, attention mechanisms, or feature flags. Without the right settings, workers may fail to load or produce incorrect outputs.
When deploying a model, check its Hugging Face README and the [vLLM documentation](https://docs.vllm.ai/en/latest/usage/) for required settings.
## Environment variables
vLLM is configured using [command-line flags](https://docs.vllm.ai/en/latest/configuration/engine_args/). On Runpod, set these as [environment variables](/serverless/vllm/environment-variables) instead.
Convert flag names to uppercase with underscores.
For example:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
--tokenizer_mode mistral
```
Becomes:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
TOKENIZER_MODE=mistral
```
### Example: Deploying Mistral
CLI command:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
vllm serve mistralai/Ministral-8B-Instruct-2410 \
--tokenizer_mode mistral \
--config_format mistral \
--load_format mistral \
--enable-auto-tool-choice \
--tool-call-parser mistral
```
Equivalent Runpod environment variables:
| Variable | Value |
| ------------------------- | -------------------------------------- |
| `MODEL_NAME` | `mistralai/Ministral-8B-Instruct-2410` |
| `TOKENIZER_MODE` | `mistral` |
| `CONFIG_FORMAT` | `mistral` |
| `LOAD_FORMAT` | `mistral` |
| `ENABLE_AUTO_TOOL_CHOICE` | `true` |
| `TOOL_CALL_PARSER` | `mistral` |
## Model-specific configurations
Recommended environment variables for popular model families. Check your model's documentation for exact requirements.
| Model family | Example model | Key environment variables | Notes |
| ------------ | ----------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------ |
| Qwen3 | `Qwen/Qwen3-8B` | `ENABLE_AUTO_TOOL_CHOICE=true` `TOOL_CALL_PARSER=hermes` | For AWQ/GPTQ versions, set `QUANTIZATION` accordingly. |
| OpenChat | `openchat/openchat-3.5-0106` | None required | Use `CUSTOM_CHAT_TEMPLATE` if default templates produce poor results. |
| Gemma | `google/gemma-3-1b-it` | None required | Requires `HF_TOKEN`. Set `DTYPE=bfloat16` for best results. |
| DeepSeek-R1 | `deepseek-ai/DeepSeek-R1-Distill-Qwen-7B` | `REASONING_PARSER=deepseek_r1` | Enables reasoning mode for chain-of-thought outputs. |
| Phi-4 | `microsoft/Phi-4-mini-instruct` | None required | `ENFORCE_EAGER=true` can resolve initialization issues on older CUDA versions. |
| Llama 3 | `meta-llama/Llama-3.2-3B-Instruct` | `TOOL_CALL_PARSER=llama3_json` `ENABLE_AUTO_TOOL_CHOICE=true` | Use `MAX_MODEL_LEN` to prevent KV cache from exceeding GPU VRAM. |
| Mistral | `mistralai/Ministral-8B-Instruct-2410` | `TOKENIZER_MODE=mistral` `CONFIG_FORMAT=mistral` `LOAD_FORMAT=mistral` `TOOL_CALL_PARSER=mistral` `ENABLE_AUTO_TOOL_CHOICE=true` | Mistral models require specialized tokenizers. |
## GPU selection
vLLM pre-allocates memory for its KV cache, so you need more VRAM than the minimum to load the model.
### VRAM estimation
* **FP16/BF16**: 2 bytes per parameter.
* **INT8**: 1 byte per parameter.
* **INT4 (AWQ/GPTQ)**: 0.5 bytes per parameter.
* **KV cache**: vLLM reserves 10-30% of remaining VRAM for concurrent requests.
| Model size | Recommended GPUs | VRAM |
| -------------------- | ------------------- | --------- |
| **Small (\<10B)** | RTX 4090, A6000, L4 | 16-24 GB |
| **Medium (10B-30B)** | A6000, L40S | 32-48 GB |
| **Large (30B-70B)** | A100, H100, B200 | 80-180 GB |
### Troubleshooting memory issues
* **OOM errors**: Lower `GPU_MEMORY_UTILIZATION` from 0.90 to 0.85, or reduce `MAX_MODEL_LEN`.
* **Context window limits**: More context means more KV cache. A 7B model that OOMs at 32k context often runs fine at 16k.
* **Limited VRAM**: Use quantized models (AWQ/GPTQ) to reduce memory by 50-75%.
For production workloads, select multiple GPU types in your [endpoint configuration](/serverless/endpoints/endpoint-configurations) for hardware fallback.
## Additional resources
* [vLLM recipes](https://docs.vllm.ai/projects/recipes/en/latest/index.html): Step-by-step deployment guides.
* [Mistral + vLLM guide](https://docs.mistral.ai/models/deployment/local-deployment/vllm).
* [Qwen + vLLM guide](https://qwen.readthedocs.io/en/latest/deployment/vllm.html).
# vLLM environment variables
Source: https://docs.runpod.io/serverless/vllm/environment-variables
Configure your vLLM workers using environment variables. Review setup, configuration, deployment, and operations guidance for Runpod Serverless.
Environment variables let you configure your vLLM workers without rebuilding your Docker image. You can customize model behavior, performance settings, and other deployment options to suit your needs.
To set environment variables, go to your [endpoint settings](/serverless/endpoints/overview#edit-an-endpoint) and add them under **Environment Variables**.
## LLM settings
These variables control the core language model configuration.
| Variable | Default | Type(s) | Description |
| ------------------------------ | ------------------- | --------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `MODEL_NAME` | `facebook/opt-125m` | `str` | Hugging Face repo ID or local filesystem path for the model weights. Change this to deploy a different model. |
| `MODEL_REVISION` | `main` | `str` | The model revision to load. |
| `TOKENIZER` | None | `str` | The name or path of the Hugging Face tokenizer to use. |
| `SKIP_TOKENIZER_INIT` | `False` | `bool` | If `True`, skips the initialization of the tokenizer and detokenizer. |
| `TOKENIZER_MODE` | `auto` | `auto`, `slow` | The tokenizer mode. |
| `TRUST_REMOTE_CODE` | `False` | `bool` | If `True`, trusts remote code from Hugging Face. |
| `DOWNLOAD_DIR` | None | `str` | The directory to download and load the model weights from. |
| `LOAD_FORMAT` | `auto` | `str` | The format of the model weights to load. |
| `HF_TOKEN` | - | `str` | Hugging Face token required to download gated/private models. Not needed for public models. Provide it via secrets. |
| `DTYPE` | `auto` | `auto`, `half`, `float16`, `bfloat16`, `float`, `float32` | The data type for model weights and activations. |
| `KV_CACHE_DTYPE` | `auto` | `auto`, `fp8` | The data type for KV cache storage. |
| `QUANTIZATION_PARAM_PATH` | None | `str` | The path to the JSON file containing the KV cache scaling factors. |
| `MAX_MODEL_LEN` | None | `int` | Maximum context length (tokens) the engine will allocate KV cache for. Lower it to reduce VRAM usage; raise it for long-context models if supported and you have VRAM. |
| `GUIDED_DECODING_BACKEND` | `outlines` | `outlines`, `lm-format-enforcer` | The default engine for guided decoding. |
| `DISTRIBUTED_EXECUTOR_BACKEND` | None | `ray`, `mp` | The backend to use for distributed serving. |
| `WORKER_USE_RAY` | `False` | `bool` | Deprecated. Use `DISTRIBUTED_EXECUTOR_BACKEND=ray` instead. |
| `PIPELINE_PARALLEL_SIZE` | `1` | `int` | The number of pipeline stages. |
| `TENSOR_PARALLEL_SIZE` | `1` | `int` | Tensor parallel degree (number of GPUs to shard across). On multi-GPU machines, this worker auto-sets it to the number of visible GPUs. |
| `MAX_PARALLEL_LOADING_WORKERS` | None | `int` | Load model sequentially in multiple batches to avoid RAM OOM when using tensor parallelism and large models. |
| `RAY_WORKERS_USE_NSIGHT` | `False` | `bool` | If `True`, uses nsight to profile Ray workers. |
| `ENABLE_PREFIX_CACHING` | `False` | `bool` | If `True`, enables automatic prefix caching. |
| `DISABLE_SLIDING_WINDOW` | `False` | `bool` | If `True`, disables the sliding window, capping to the sliding window size. |
| `USE_V2_BLOCK_MANAGER` | `False` | `bool` | If `True`, uses the BlockSpaceMangerV2. |
| `NUM_LOOKAHEAD_SLOTS` | `0` | `int` | The number of lookahead slots, an experimental scheduling configuration for speculative decoding. |
| `SEED` | `0` | `int` | The random seed for operations. |
| `NUM_GPU_BLOCKS_OVERRIDE` | None | `int` | If specified, this value overrides the GPU profiling result for the number of GPU blocks. |
| `MAX_NUM_BATCHED_TOKENS` | None | `int` | The maximum number of batched tokens per iteration. |
| `MAX_NUM_SEQS` | `256` | `int` | Upper bound on sequences batched per iteration (affects throughput, VRAM, and tail latency). Higher can improve throughput for many concurrent short requests; lower reduces VRAM usage. |
| `MAX_LOGPROBS` | `20` | `int` | The maximum number of log probabilities to return when `logprobs` is specified in `SamplingParams`. |
| `DISABLE_LOG_STATS` | `False` | `bool` | If `True`, disables logging statistics. |
| `QUANTIZATION` | None | `awq`, `squeezellm`, `gptq`, `bitsandbytes` | Quantization backend for loading quantized checkpoints (AWQ/GPTQ/...) or BitsAndBytes. Must match the checkpoint format. |
| `ROPE_SCALING` | None | `dict` | The RoPE scaling configuration in JSON format. |
| `ROPE_THETA` | None | `float` | The RoPE theta value. Use with `ROPE_SCALING`. |
| `TOKENIZER_POOL_SIZE` | `0` | `int` | The size of the tokenizer pool for asynchronous tokenization. |
| `TOKENIZER_POOL_TYPE` | `ray` | `str` | The type of the tokenizer pool for asynchronous tokenization. |
| `TOKENIZER_POOL_EXTRA_CONFIG` | None | `dict` | Extra configuration for the tokenizer pool. |
## LoRA settings
Configure LoRA (Low-Rank Adaptation) adapters for your model.
| Variable | Default | Type | Description |
| --------------------------- | ------- | ---------------------------------------- | ------------------------------------------------------------------------------------------------------------ |
| `ENABLE_LORA` | `False` | `bool` | If `True`, enables the handling of LoRA adapters. |
| `MAX_LORAS` | `1` | `int` | The maximum number of LoRAs in a single batch. |
| `MAX_LORA_RANK` | `16` | `int` | The maximum LoRA rank. |
| `LORA_EXTRA_VOCAB_SIZE` | `256` | `int` | The maximum size of the extra vocabulary for LoRA adapters. |
| `LORA_DTYPE` | `auto` | `auto`, `float16`, `bfloat16`, `float32` | The data type for LoRA. |
| `LONG_LORA_SCALING_FACTORS` | None | `tuple` | Specifies multiple scaling factors for LoRA adapters. |
| `MAX_CPU_LORAS` | None | `int` | The maximum number of LoRAs to store in CPU memory. |
| `FULLY_SHARDED_LORAS` | `False` | `bool` | If `True`, enables fully sharded LoRA layers. |
| `LORA_MODULES` | `[]` | `list[dict]` | A list of LoRA adapters to add from Hugging Face. Example: `[{"name": "adapter1", "path": "user/adapter1"}]` |
## Speculative decoding settings
Configure speculative decoding to improve performance.
| Variable | Default | Type(s) | Description |
| ------------------------------------------------ | ------------------- | ------------------------------------------------- | ------------------------------------------------------------------------------------------- |
| `SCHEDULER_DELAY_FACTOR` | `0.0` | `float` | Applies a delay before scheduling the next prompt. |
| `ENABLE_CHUNKED_PREFILL` | `False` | `bool` | If `True`, enables chunked prefill requests. |
| `SPECULATIVE_MODEL` | None | `str` | The name of the draft model for speculative decoding. |
| `NUM_SPECULATIVE_TOKENS` | None | `int` | The number of speculative tokens to sample from the draft model. |
| `SPECULATIVE_DRAFT_TENSOR_PARALLEL_SIZE` | None | `int` | The number of tensor parallel replicas for the draft model. |
| `SPECULATIVE_MAX_MODEL_LEN` | None | `int` | The maximum sequence length supported by the draft model. |
| `SPECULATIVE_DISABLE_BY_BATCH_SIZE` | None | `int` | Disables speculative decoding if the number of enqueued requests is larger than this value. |
| `NGRAM_PROMPT_LOOKUP_MAX` | None | `int` | The maximum window size for ngram prompt lookup in speculative decoding. |
| `NGRAM_PROMPT_LOOKUP_MIN` | None | `int` | The minimum window size for ngram prompt lookup in speculative decoding. |
| `SPEC_DECODING_ACCEPTANCE_METHOD` | `rejection_sampler` | `rejection_sampler`, `typical_acceptance_sampler` | The acceptance method for draft token verification in speculative decoding. |
| `TYPICAL_ACCEPTANCE_SAMPLER_POSTERIOR_THRESHOLD` | None | `float` | Sets the lower bound threshold for the posterior probability of a token to be accepted. |
| `TYPICAL_ACCEPTANCE_SAMPLER_POSTERIOR_ALPHA` | None | `float` | A scaling factor for the entropy-based threshold for token acceptance. |
## System performance settings
Configure GPU memory and system resource utilization.
| Variable | Default | Type(s) | Description |
| ------------------------------ | ------- | --------------- | --------------------------------------------------------------------------------------------------------------------------------------------- |
| `GPU_MEMORY_UTILIZATION` | `0.95` | `float` | Fraction of GPU VRAM vLLM is allowed to use for KV cache and runtime allocations. Lower if you hit CUDA OOM; raise if you have VRAM headroom. |
| `MAX_PARALLEL_LOADING_WORKERS` | None | `int` | Loads the model sequentially in multiple batches to avoid RAM OOM when using tensor parallelism and large models. |
| `BLOCK_SIZE` | `16` | `8`, `16`, `32` | The token block size for contiguous chunks of tokens. |
| `SWAP_SPACE` | `4` | `int` | The CPU swap space size (in GiB) per GPU. |
| `ENFORCE_EAGER` | `False` | `bool` | If `True`, always uses eager-mode PyTorch. If `False`, uses a hybrid of eager mode and CUDA graphs for maximal performance and flexibility. |
| `MAX_SEQ_LEN_TO_CAPTURE` | `8192` | `int` | Maximum context length covered by CUDA graphs. When a sequence has context length larger than this, we fall back to eager mode. |
| `DISABLE_CUSTOM_ALL_REDUCE` | `0` | `int` | If `0`, enables custom all-reduce. If `1`, disables it. |
| `ENABLE_EXPERT_PARALLEL` | `False` | `bool` | Enable expert parallelism for Mixture of Experts (MoE) models. |
## Tokenizer settings
Customize tokenizer behavior and chat templates.
| Variable | Default | Type(s) | Description |
| ---------------------- | ------- | ----------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `TOKENIZER_NAME` | None | `str` | The tokenizer repository to use a different tokenizer than the model's default. |
| `TOKENIZER_REVISION` | None | `str` | The tokenizer revision to load. |
| `CUSTOM_CHAT_TEMPLATE` | None | `str` of single-line jinja template | Override the model chat template (single-line Jinja2). Useful when sending `messages` to a base model without a built-in chat template. See the [Hugging Face documentation](https://huggingface.co/docs/transformers/chat_templating) for more information. |
## Streaming and batch settings
Control how tokens are streamed back in HTTP responses.
These settings control how tokens are batched in HTTP responses when streaming. The batch size starts at `DEFAULT_MIN_BATCH_SIZE` and increases by a factor of `DEFAULT_BATCH_SIZE_GROWTH_FACTOR` with each request until it reaches `DEFAULT_BATCH_SIZE`.
For example, with default values, the batch sizes would be 1, 3, 9, 27, and then 50 for all subsequent requests. These settings do not affect vLLM's internal batching.
| Variable | Default | Type(s) | Description |
| ---------------------------------- | ------- | ------- | ------------------------------------------------------- |
| `DEFAULT_BATCH_SIZE` | `50` | `int` | The default and maximum batch size for token streaming. |
| `DEFAULT_MIN_BATCH_SIZE` | `1` | `int` | The initial batch size for the first request. |
| `DEFAULT_BATCH_SIZE_GROWTH_FACTOR` | `3` | `float` | The growth factor for the dynamic batch size. |
## OpenAI compatibility settings
Configure OpenAI API compatibility features.
| Variable | Default | Type(s) | Description |
| ----------------------------------- | ----------- | ---------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `RAW_OPENAI_OUTPUT` | `1` | boolean as `int` | If `1`, enables raw OpenAI SSE format string output when streaming. This is required for OpenAI compatibility. |
| `OPENAI_SERVED_MODEL_NAME_OVERRIDE` | None | `str` | Exposes a custom model ID via `/v1/models` and accepts it as the `model` field in OpenAI requests (alias for the served model). |
| `OPENAI_RESPONSE_ROLE` | `assistant` | `str` | The role of the LLM's response in OpenAI chat completions. |
| `ENABLE_AUTO_TOOL_CHOICE` | `false` | `bool` | Enables vLLM automatic tool selection for OpenAI Chat Completions. Only enable for tool-capable models. |
| `TOOL_CALL_PARSER` | None | `str` | Tool-call parser that matches your model's tool-call format (required for most tool-calling models). Supported values: `mistral`, `hermes`, `llama3_json`, `llama4_json`, `llama4_pythonic`, `granite`, `granite-20b-fc`, `deepseek_v3`, `internlm`, `jamba`, `phi4_mini_json`, `pythonic`. |
| `REASONING_PARSER` | None | `str` | Parser for reasoning-capable models (enables reasoning mode). Examples: `deepseek_r1`, `qwen3`, `granite`, `hunyuan_a13b`. Leave unset to disable. |
`TOOL_CALL_PARSER` tells vLLM how to interpret a model's tool-call output. If the parser doesn't match the model's format, tool calls may not be detected (or may error during parsing).
## Serverless and concurrency settings
Configure concurrency and logging for Serverless deployments.
| Variable | Default | Type(s) | Description |
| ---------------------- | ------- | ------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `MAX_CONCURRENCY` | `30` | `int` | Max concurrent requests per worker instance (Runpod-side). Not a vLLM engine arg; it controls Runpod worker concurrency and affects how requests are fed into vLLM (queueing/throughput/latency). |
| `DISABLE_LOG_STATS` | `False` | `bool` | If `False`, enables vLLM stats logging. |
| `DISABLE_LOG_REQUESTS` | `False` | `bool` | If `False`, enables vLLM request logging. |
## Advanced settings
Additional configuration options for specialized use cases.
| Variable | Default | Type | Description |
| --------------------------- | ------- | ------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `MODEL_LOADER_EXTRA_CONFIG` | None | `dict` | Extra configuration for the model loader. |
| `PREEMPTION_MODE` | None | `str` | The preemption mode. If `recompute`, the engine performs preemption-aware recomputation. If `save`, the engine saves activations to CPU memory during preemption. |
| `PREEMPTION_CHECK_PERIOD` | `1.0` | `float` | The frequency (in seconds) at which the engine checks for preemption. |
| `PREEMPTION_CPU_CAPACITY` | `2` | `float` | The percentage of CPU memory to use for saved activations. |
| `DISABLE_LOGGING_REQUEST` | `False` | `bool` | If `True`, disables logging requests. |
| `MAX_LOG_LEN` | None | `int` | The maximum number of prompt characters or prompt ID numbers to print in the log. |
## Docker build arguments
These variables are used when building custom Docker images with models baked in.
| Variable | Default | Type | Description |
| --------------------- | ---------------- | ----- | ----------------------------------------------------------- |
| `BASE_PATH` | `/runpod-volume` | `str` | The storage directory for the Hugging Face cache and model. |
| `WORKER_CUDA_VERSION` | `12.1.0` | `str` | The CUDA version for the worker image. |
## Deprecated variables
The following variables are deprecated and will be removed in future versions.
| Old Variable | New Variable | Note |
| ---------------------------- | ------------------------ | -------------------------- |
| `MAX_CONTEXT_LEN_TO_CAPTURE` | `MAX_SEQ_LEN_TO_CAPTURE` | Use the new variable name. |
| `kv_cache_dtype=fp8_e5m2` | `kv_cache_dtype=fp8` | Simplified fp8 format. |
# Deploy vLLM on Runpod Serverless
Source: https://docs.runpod.io/serverless/vllm/get-started
Create a Serverless endpoint to serve LLM inference via API request. Review configuration and operations guidance for Runpod Serverless.
## Requirements
* [Runpod account](/accounts-billing/manage-accounts).
* [Runpod API key](/get-started/api-keys).
* (For gated models) [Hugging Face access token](https://huggingface.co/docs/hub/en/security-tokens).
## Step 1: Choose a model
First, decide which LLM you want to deploy. The vLLM worker supports most models available on Hugging Face, including:
* Llama 3 (e.g., `meta-llama/Llama-3.2-3B-Instruct`).
* Mistral (e.g., `mistralai/Ministral-8B-Instruct-2410`).
* Qwen3 (e.g., `Qwen/Qwen3-8B`).
* OpenChat (e.g., `openchat/openchat-3.5-0106`).
* Gemma (e.g., `google/gemma-3-1b-it`).
* DeepSeek-R1 (e.g., `deepseek-ai/DeepSeek-R1-Distill-Qwen-7B`).
* Phi-4 (e.g., `microsoft/Phi-4-mini-instruct`).
For this tutorial, we'll use `openchat/openchat-3.5-0106`, but you can substitute this with [any compatible model](https://docs.vllm.ai/en/latest/models/supported_models.html).
Depending on the model you choose, you may need to [configure your endpoint](/serverless/vllm/configuration) with additional environment variables.
## Step 2: Deploy using the Runpod UI
The easiest way to deploy a vLLM worker is through Runpod's ready-to-deploy repos:
1. Find the [vLLM repo](https://console.runpod.io/hub/runpod-workers/worker-vllm) in the Runpod Hub.
2. Click **Deploy**, using the latest vLLM worker version.
3. In the **Model** field, end the model name: `openchat/openchat-3.5-0106`.
4. Click **Advanced** to expand the vLLM settings.
5. Set **Max Model Length** to `8192` (or an appropriate context length for your model).
6. Leave other settings at their defaults unless you have specific requirements, then click **Next**.
7. Click **Create Endpoint**
Your endpoint will now begin initializing. This may take several minutes while Runpod provisions resources and downloads the selected model.
For more details on how to optimize your endpoint, see [Endpoint configurations](/serverless/endpoints/endpoint-configurations).
## Step 3: Find your endpoint ID
Once deployment is complete, make a note of your **Endpoint ID**, as you'll need this to make API requests.
## Step 4: Send a test request using the UI
To test your worker, click the **Requests** tab in the endpoint detail page:
On the left you should see the default test request:
```json theme={"theme":{"light":"github-light","dark":"github-dark"}}
{
"input": {
"prompt": "Hello World"
}
}
```
Leave the default input as is and click **Run**. The system will take a few minutes to initialize your workers.
When the workers finish processing your request, you should see output on the right side of the page similar to this:
```json theme={"theme":{"light":"github-light","dark":"github-dark"}}
{
"delayTime": 638,
"executionTime": 3344,
"id": "f0706ead-c5ec-4689-937c-e21d5fbbca47-u1",
"output": [
{
"choices": [
{
"tokens": ["CHAT_RESPONSE"]
}
],
"usage": {
"input": 3,
"output": 100
}
}
],
"status": "COMPLETED",
"workerId": "0e7o8fgmm9xgty"
}
```
## Step 5: Send a test request using the API
To send a test request using the API, use the following command, replacing `YOUR_ENDPOINT_ID` and `YOUR_API_KEY` with your actual endpoint ID and API key:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
curl -X POST "https://api.runpod.ai/v2/YOUR_ENDPOINT_ID/runsync" \
-H "Authorization: Bearer YOUR_API_KEY" \
-H "Content-Type: application/json" \
-d '{"input": {"prompt": "Hello World"}}'
```
Congratulations! You've successfully deployed a vLLM worker on Runpod Serverless. You now have a powerful, scalable LLM inference API that's compatible with both the OpenAI client and Runpod's native API.
## Customize your deployment with environment variables (optional)
If you need to customize your model deployment, you can edit your endpoint settings to add environment variables. Here are some useful environment variables you might want to set:
* `MAX_MODEL_LEN`: Maximum context length (e.g., `16384`).
* `DTYPE`: Data type for model weights (`float16`, `bfloat16`, or `float32`).
* `GPU_MEMORY_UTILIZATION`: Controls VRAM usage (e.g., `0.95` for 95%).
* `CUSTOM_CHAT_TEMPLATE`: For models that need a custom chat template.
* `OPENAI_SERVED_MODEL_NAME_OVERRIDE`: Change the model name to use in OpenAI requests.
To add or modify environment variables:
1. Go to your endpoint details page.
2. Select **Manage**, then select **Edit Endpoint**.
3. Expand the **Public Environment Variables** section.
4. Add or edit your desired variables.
5. Click **Save Endpoint**.
For a complete list of available environment variables, see the [vLLM environment variables reference](/serverless/vllm/environment-variables).
You may also wish to adjust the input parameters for your request. For example, use the `max_tokens` parameter to increase the maximum number of tokens generated per response. To learn more, see [Send vLLM requests](/serverless/vllm/vllm-requests).
## Troubleshooting
If you encounter issues with your deployment:
* **Worker fails to initialize**: Check that your model is compatible with vLLM and your GPU has enough VRAM.
* **Slow response times**: Consider using a more powerful GPU or optimizing your request parameters.
* **Out of memory errors**: Try increasing GPU size or reducing `MAX_MODEL_LEN`.
* **API errors**: Verify your endpoint ID and API key are correct.
## Next steps
* [Send requests using the Runpod API](/serverless/vllm/vllm-requests).
* [Learn about vLLM's OpenAI API compatibility](/serverless/vllm/openai-compatibility).
* [Customize your vLLM worker's handler function](/serverless/workers/handler-functions).
* [Build a custom worker for more specialized workloads](/serverless/quickstart).
# OpenAI API compatibility
Source: https://docs.runpod.io/serverless/vllm/openai-compatibility
Use OpenAI client libraries with your vLLM workers. Review setup, configuration, deployment, and operations guidance for Runpod Serverless.
vLLM workers implement OpenAI API compatibility, so you can use [OpenAI client libraries](https://developers.openai.com/api/docs/libraries) with your deployed models.
To integrate with OpenAI-compatible tools, just configure the base URL and API key using your Runpod API key and Serverless endpoint ID.
## Setup
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
from openai import OpenAI
client = OpenAI(
api_key="RUNPOD_API_KEY",
base_url="https://api.runpod.ai/v2/ENDPOINT_ID/openai/v1"
)
```
```javascript theme={"theme":{"light":"github-light","dark":"github-dark"}}
import { OpenAI } from "openai";
const client = new OpenAI({
apiKey: "RUNPOD_API_KEY",
baseURL: "https://api.runpod.ai/v2/ENDPOINT_ID/openai/v1"
});
```
Replace `ENDPOINT_ID` and `RUNPOD_API_KEY` with your actual values.
## Supported endpoints
| Endpoint | Description |
| ------------------- | ------------------------------------------------- |
| `/chat/completions` | Chat model completions (instruction-tuned models) |
| `/completions` | Text completions (base models) |
| `/models` | List available models |
## Chat completions
For instruction-tuned models that follow a chat format.
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
response = client.chat.completions.create(
model="mistralai/Mistral-7B-Instruct-v0.2",
messages=[
{"role": "system", "content": "You are a helpful assistant."},
{"role": "user", "content": "Hello, who are you?"}
],
temperature=0.7,
max_tokens=500
)
print(response.choices[0].message.content)
```
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
stream = client.chat.completions.create(
model="mistralai/Mistral-7B-Instruct-v0.2",
messages=[
{"role": "system", "content": "You are a helpful assistant."},
{"role": "user", "content": "Write a short poem about stars."}
],
temperature=0.7,
max_tokens=200,
stream=True
)
for chunk in stream:
if chunk.choices[0].delta.content:
print(chunk.choices[0].delta.content, end="", flush=True)
```
## Text completions
For base models and raw text completion.
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
response = client.completions.create(
model="mistralai/Mistral-7B-Instruct-v0.2",
prompt="Write a poem about artificial intelligence:",
temperature=0.7,
max_tokens=150
)
print(response.choices[0].text)
```
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
stream = client.completions.create(
model="mistralai/Mistral-7B-Instruct-v0.2",
prompt="The future of AI is",
temperature=0.7,
max_tokens=100,
stream=True
)
for chunk in stream:
print(chunk.choices[0].text or "", end="", flush=True)
```
## Model name
The `model` parameter must match either:
* The Hugging Face model you deployed (e.g., `mistralai/Mistral-7B-Instruct-v0.2`)
* A custom name set via the `OPENAI_SERVED_MODEL_NAME_OVERRIDE` environment variable
List available models:
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
models = client.models.list()
print([model.id for model in models])
```
## Parameters
Standard OpenAI parameters are supported. Include them directly in your request.
| Parameter | Type | Default | Description |
| ------------------- | ------------------ | -------- | ------------------------------------------------ |
| `model` | `string` | Required | Your deployed model name. |
| `messages` | `list` | Required | Chat messages with `role` and `content`. |
| `prompt` | `string` | Required | Text completion prompt. |
| `temperature` | `float` | `0.7` | Sampling randomness. Lower = more deterministic. |
| `max_tokens` | `int` | `16` | Maximum tokens to generate. |
| `top_p` | `float` | `1.0` | Nucleus sampling threshold. |
| `n` | `int` | `1` | Number of completions to generate. |
| `stop` | `string` or `list` | None | Stop sequences. |
| `stream` | `bool` | `false` | Enable streaming. |
| `presence_penalty` | `float` | `0.0` | Penalize tokens already present. |
| `frequency_penalty` | `float` | `0.0` | Penalize frequent tokens. |
| Parameter | Type | Default | Description |
| --------------------- | ------- | ------- | ------------------------------------ |
| `best_of` | `int` | None | Generate this many, return top `n`. |
| `top_k` | `int` | `-1` | Top-k sampling. -1 = all tokens. |
| `repetition_penalty` | `float` | `1.0` | Penalize repeated tokens. |
| `min_p` | `float` | `0.0` | Minimum probability threshold. |
| `use_beam_search` | `bool` | `false` | Use beam search instead of sampling. |
| `length_penalty` | `float` | `1.0` | Length penalty for beam search. |
| `ignore_eos` | `bool` | `false` | Continue after EOS token. |
| `skip_special_tokens` | `bool` | `true` | Omit special tokens from output. |
| `echo` | `bool` | `false` | Include prompt in output. |
## Environment variables
Use these environment variables to customize the OpenAI compatibility:
| Variable | Default | Description |
| ----------------------------------- | ----------- | ------------------------------------------- |
| `RAW_OPENAI_OUTPUT` | `1` | Enable raw OpenAI SSE format for streaming. |
| `OPENAI_SERVED_MODEL_NAME_OVERRIDE` | None | Override model name in responses. |
| `OPENAI_RESPONSE_ROLE` | `assistant` | Role for chat completion responses. |
See [environment variables reference](/serverless/vllm/environment-variables) for all options.
## Differences from OpenAI
* **Token counting** may differ due to different tokenizers.
* **Rate limits** follow Runpod's policies, not OpenAI's.
* **Function/tool calling** depends on model and vLLM support.
* **Vision/multimodal** depends on underlying model support.
## Troubleshooting
| Issue | Solution |
| -------------------------- | ------------------------------------------- |
| "Invalid model" error | Verify model name matches your deployment. |
| Authentication error | Use your Runpod API key, not an OpenAI key. |
| Timeout errors | Increase client timeout for large models. |
| Unexpected response format | Set `RAW_OPENAI_OUTPUT=1`. |
## Integrate with workflow tools
Follow the [n8n integration guide](/integrations/n8n-integration) to connect an OpenAI-compatible Runpod endpoint to an automated workflow.
# Overview
Source: https://docs.runpod.io/serverless/vllm/overview
Deploy scalable LLM inference endpoints using vLLM workers. Review setup, configuration, deployment, and operations guidance for Runpod Serverless.
vLLM workers deploy and serve large language models on Runpod Serverless with fast inference and automatic scaling. Deploy directly from the [Runpod Hub](https://console.runpod.io/hub/runpod-workers/worker-vllm) or customize using the [runpod-workers/worker-vllm](https://github.com/runpod-workers/worker-vllm) repository as a base.
Deploy your first vLLM worker in minutes.
Configure your vLLM endpoint with environment variables.
Send requests using Runpod's native API.
Integrate vLLM with OpenAI-compatible tools.
## What is vLLM?
vLLM is an open-source inference engine optimized for serving large language models. It maximizes throughput and minimizes latency through techniques like PagedAttention and continuous batching.
* **[PagedAttention](https://docs.vllm.ai/en/latest/design/paged_attention.html)**: Breaks KV cache into pages for efficient memory use, enabling higher concurrency and larger models on smaller GPUs.
* **Continuous batching**: Processes requests as they arrive rather than waiting for batches, keeping GPUs busy and reducing latency.
* **OpenAI compatibility**: Drop-in replacement for OpenAI's API. Switch by changing the endpoint URL and API key.
* **Hugging Face integration**: Supports most models including Llama, Mistral, Qwen, Gemma, DeepSeek, and [many more](https://docs.vllm.ai/en/latest/models/supported_models.html).
* **Auto-scaling**: Scales from zero to many workers based on demand, with per-second billing.
## Deployment options
* **[Cached models](/serverless/endpoints/model-caching)** (recommended): Fastest setup with lower storage costs. Best for most deployments.
* **[Baked-in models](/serverless/workers/create-dockerfile#including-models-and-files)**: Eliminates download time and reduces cold starts to seconds. Requires building a custom Docker image.
## Configuration
Default settings work for many models, but some require additional [environment variables](/serverless/vllm/environment-variables) (which map to `vllm serve` flags). Consult your model's Hugging Face README and the [vLLM documentation](https://docs.vllm.ai/en/latest/usage/) for model-specific requirements.
# Send requests to vLLM workers
Source: https://docs.runpod.io/serverless/vllm/vllm-requests
Use Runpod's native API to send requests to vLLM workers. Review setup, configuration, deployment, and operations guidance for Runpod Serverless.
vLLM workers use the same `/run` and `/runsync` operations as other Runpod Serverless endpoints. The difference is the input format: vLLM expects prompts, messages, and sampling parameters for text generation.
## Input formats
### Messages (chat models)
Use for instruction-tuned models. The worker automatically applies the model's chat template.
```json theme={"theme":{"light":"github-light","dark":"github-dark"}}
{
"input": {
"messages": [
{"role": "system", "content": "You are a helpful assistant."},
{"role": "user", "content": "What is the capital of France?"}
],
"sampling_params": {
"temperature": 0.7,
"max_tokens": 100
}
}
}
```
### Prompt (text completion)
Use for base models or when providing raw text without a chat template.
```json theme={"theme":{"light":"github-light","dark":"github-dark"}}
{
"input": {
"prompt": "The capital of France is",
"sampling_params": {
"temperature": 0.7,
"max_tokens": 50
}
}
}
```
To apply the model's chat template to a prompt, add `"apply_chat_template": true`.
## Send requests
Submit a job that processes in the background. Poll `/status/{job_id}` for results.
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
import requests
response = requests.post(
"https://api.runpod.ai/v2/ENDPOINT_ID/run",
headers={
"Authorization": "Bearer RUNPOD_API_KEY",
"Content-Type": "application/json"
},
json={
"input": {
"messages": [{"role": "user", "content": "Explain quantum computing."}],
"sampling_params": {"temperature": 0.7, "max_tokens": 200}
}
}
)
job_id = response.json()["id"]
print(f"Job ID: {job_id}")
# Poll for results
status = requests.get(
f"https://api.runpod.ai/v2/ENDPOINT_ID/status/{job_id}",
headers={"Authorization": "Bearer RUNPOD_API_KEY"}
)
print(status.json())
```
Wait for the complete response in a single request.
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
import requests
response = requests.post(
"https://api.runpod.ai/v2/ENDPOINT_ID/runsync",
headers={
"Authorization": "Bearer RUNPOD_API_KEY",
"Content-Type": "application/json"
},
json={
"input": {
"messages": [{"role": "user", "content": "Explain quantum computing."}],
"sampling_params": {"temperature": 0.7, "max_tokens": 200}
}
}
)
print(response.json())
```
For more on request operations, see [Send requests to Serverless endpoints](/serverless/endpoints/send-requests).
## Streaming
Receive tokens as they're generated instead of waiting for the complete response.
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
import requests
import json
# Submit with streaming enabled
response = requests.post(
"https://api.runpod.ai/v2/ENDPOINT_ID/run",
headers={
"Authorization": "Bearer RUNPOD_API_KEY",
"Content-Type": "application/json"
},
json={
"input": {
"prompt": "Write a short story about a robot.",
"sampling_params": {"temperature": 0.8, "max_tokens": 500},
"stream": True
}
}
)
job_id = response.json()["id"]
# Stream results
stream_url = f"https://api.runpod.ai/v2/ENDPOINT_ID/stream/{job_id}"
with requests.get(stream_url, headers={"Authorization": "Bearer RUNPOD_API_KEY"}, stream=True) as r:
for line in r.iter_lines():
if line:
print(json.loads(line))
```
See [streaming documentation](/serverless/endpoints/send-requests#stream) for more details.
## Sampling parameters
Add parameters to control how the model generates text. Include these in the `sampling_params` object in your request.
| Parameter | Type | Default | Description |
| ------------------- | ------------------ | ------- | --------------------------------------------------- |
| `max_tokens` | `int` | `16` | Maximum tokens to generate. |
| `temperature` | `float` | `1.0` | Randomness of sampling. Lower = more deterministic. |
| `top_p` | `float` | `1.0` | Cumulative probability of top tokens to consider. |
| `top_k` | `int` | `-1` | Number of top tokens to consider. -1 = all. |
| `stop` | `string` or `list` | `None` | Stop generation when these strings are produced. |
| `presence_penalty` | `float` | `0.0` | Penalize tokens based on presence in output. |
| `frequency_penalty` | `float` | `0.0` | Penalize tokens based on frequency in output. |
| Parameter | Type | Default | Description |
| ------------------------------- | ----------- | ------- | ----------------------------------------------------------- |
| `n` | `int` | `1` | Number of output sequences to generate. |
| `best_of` | `int` | `n` | Generate this many sequences, return top `n`. |
| `repetition_penalty` | `float` | `1.0` | Penalize repeated tokens. Values > 1 discourage repetition. |
| `min_p` | `float` | `0.0` | Minimum probability threshold relative to top token. |
| `min_tokens` | `int` | `0` | Minimum tokens before allowing EOS. |
| `use_beam_search` | `bool` | `false` | Use beam search instead of sampling. |
| `length_penalty` | `float` | `1.0` | Length penalty for beam search. |
| `early_stopping` | `bool` | `false` | Stop beam search early. |
| `stop_token_ids` | `list[int]` | `None` | Token IDs that stop generation. |
| `ignore_eos` | `bool` | `false` | Continue generating after EOS token. |
| `skip_special_tokens` | `bool` | `true` | Omit special tokens from output. |
| `spaces_between_special_tokens` | `bool` | `true` | Add spaces between special tokens. |
| `truncate_prompt_tokens` | `int` | `None` | Truncate prompt to this many tokens. |
| Parameter | Type | Default | Description |
| -------------------------- | ------ | ----------- | ------------------------------- |
| `stream` | `bool` | `false` | Enable streaming output. |
| `max_batch_size` | `int` | env default | Max tokens per streaming chunk. |
| `min_batch_size` | `int` | env default | Min tokens per streaming chunk. |
| `batch_size_growth_factor` | `int` | env default | Growth factor for batch size. |
## Error handling
Implement retry logic with exponential backoff to handle network issues, rate limits, and cold starts.
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
import requests
import time
def send_request(url, headers, payload, max_retries=3):
for attempt in range(max_retries):
try:
response = requests.post(url, headers=headers, json=payload, timeout=300)
response.raise_for_status()
return response.json()
except requests.exceptions.HTTPError as e:
if e.response.status_code == 429: # Rate limit
time.sleep(5)
elif e.response.status_code >= 500:
time.sleep(2 ** attempt)
else:
raise
except requests.exceptions.RequestException:
time.sleep(2 ** attempt)
raise Exception("Max retries exceeded")
```
# Build a concurrent handler
Source: https://docs.runpod.io/serverless/workers/concurrent-handler
Build a concurrent handler function to process multiple requests simultaneously on a single worker. See Runpod Serverless setup and usage details.
## Requirements
* You've [created a Runpod account](/accounts-billing/manage-accounts).
* You've installed the Runpod SDK (`pip install runpod`).
* You know how to build a [basic handler function](/serverless/workers/handler-functions).
## Step 1: Set up your environment
First, set up a virtual environment and install the necessary packages:
```sh theme={"theme":{"light":"github-light","dark":"github-dark"}}
# Create a Python virtual environment
python3 -m venv venv
source venv/bin/activate # On Windows: venv\Scripts\activate
pip install runpod asyncio
```
## Step 2: Create a concurrent handler file
Create a file named `concurrent_handler.py` and add the following code:
```python concurrent_handler.py theme={"theme":{"light":"github-light","dark":"github-dark"}}
import runpod
import asyncio
import random
# Global variable to simulate a varying request rate
request_rate = 0
async def process_request(job):
# This function processes incoming requests concurrently.
#
# Args:
# job (dict): Contains the input data and request metadata
#
# Returns:
# str: The processed result
# Extract input data
job_input = job["input"]
delay = job_input.get("delay", 1)
# Simulate an asynchronous task (like a database query or API call)
await asyncio.sleep(delay)
return f"Processed: {job_input}"
# Placeholder code for a dynamic concurrency adjustment function
def adjust_concurrency(current_concurrency):
return 50
def update_request_rate():
"""Simulates changes in the request rate to mimic real-world scenarios."""
global request_rate
request_rate = random.randint(20, 100)
# Start the Serverless function when the script is run
if __name__ == "__main__":
runpod.serverless.start({
"handler": process_request,
"concurrency_modifier": adjust_concurrency
})
```
The `process_request` function uses the `async` keyword, enabling it to use non-blocking I/O operations with `await`. This allows the function to pause during I/O operations (simulated with `asyncio.sleep()`) and handle other requests while waiting.
The `update_request_rate` function simulates monitoring request patterns for adaptive scaling. This example uses a simple random number generator to simulate changing request patterns. In a production environment, you would:
* Track actual request counts and response times.
* Monitor system resource usage, such as CPU and memory.
* Adjust concurrency based on real performance metrics.
## Step 3: Implement dynamic concurrency adjustment
Let's enhance our handler with dynamic concurrency adjustment. This will allow your worker to handle more requests during high traffic periods and conserve resources during low traffic periods.
Replace the placeholder `adjust_concurrency` function with this improved version:
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
def adjust_concurrency(current_concurrency):
# Dynamically adjust the worker's concurrency level based on request load.
#
# Args:
# current_concurrency (int): The current concurrency level
#
# Returns:
# int: The new concurrency level
global request_rate
# In production, this would use real metrics
update_request_rate()
max_concurrency = 10 # Maximum allowable concurrency
min_concurrency = 1 # Minimum concurrency to maintain
high_request_rate_threshold = 50 # Threshold for high request volume
# Increase concurrency if under max limit and request rate is high
if (request_rate > high_request_rate_threshold and
current_concurrency < max_concurrency):
return current_concurrency + 1
# Decrease concurrency if above min limit and request rate is low
elif (request_rate <= high_request_rate_threshold and
current_concurrency > min_concurrency):
return current_concurrency - 1
return current_concurrency
```
Let's break down how this function works:
1. **Control parameters**:
* `max_concurrency = 10`: Sets an upper limit on concurrency to prevent resource exhaustion.
* `min_concurrency = 1`: Ensures at least one request can be processed at a time.
* `high_request_rate_threshold = 50`: Defines when to consider traffic "high".
You can adjust these parameters based on your specific workload.
2. **Scaling up logic**:
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
if (request_rate > high_request_rate_threshold and
current_concurrency < max_concurrency):
return current_concurrency + 1
```
This increases concurrency by 1 when:
* The request rate exceeds our threshold (50 requests).
* We haven't reached our maximum concurrency limit.
3. **Scaling down logic**:
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
elif (request_rate <= high_request_rate_threshold and
current_concurrency > min_concurrency):
return current_concurrency - 1
```
This decreases concurrency by 1 when:
* The request rate is at or below our threshold.
* We're above our minimum concurrency level.
4. **Default behavior**:
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
return current_concurrency
```
If neither condition is met, maintain the current concurrency level.
With these enhancements, your concurrent handler will now dynamically adjust its concurrency level based on the observed request rate, optimizing resource usage and responsiveness.
## Step 4: Create a test input file
Now we're ready to test our handler. Create a file named `test_input.json` to test your handler locally:
```json test_input.json theme={"theme":{"light":"github-light","dark":"github-dark"}}
{
"input": {
"message": "Test concurrent processing",
"delay": 0.5
}
}
```
## Step 5: Test your handler locally
Run your handler to verify that it works correctly:
```sh theme={"theme":{"light":"github-light","dark":"github-dark"}}
python concurrent_handler.py
```
You should see output similar to this:
```sh theme={"theme":{"light":"github-light","dark":"github-dark"}}
--- Starting Serverless Worker | Version 1.7.9 ---
INFO | Using test_input.json as job input.
DEBUG | Retrieved local job: {'input': {'message': 'Test concurrent processing', 'delay': 0.5}, 'id': 'local_test'}
INFO | local_test | Started.
DEBUG | local_test | Handler output: Processed: {'message': 'Test concurrent processing', 'delay': 0.5}
DEBUG | local_test | run_job return: {'output': "Processed: {'message': 'Test concurrent processing', 'delay': 0.5}"}
INFO | Job local_test completed successfully.
INFO | Job result: {'output': "Processed: {'message': 'Test concurrent processing', 'delay': 0.5}"}
INFO | Local testing complete, exiting.
```
## (Optional) Step 6: Implement real metrics collection
In a production environment, you should to replace the `update_request_rate` function with real metrics collection. Here is an example how you could build this functionality:
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
def update_request_rate(request_history):
# Collects real metrics about request patterns.
#
# Args:
# request_history (list): A list of request timestamps
#
# Returns:
# int: The new request rate
global request_rate
# Option 1: Track request count over a time window
current_time = time.time()
# Count requests in the last minute
recent_requests = [r for r in request_history if r > current_time - 60]
request_rate = len(recent_requests)
# Option 2: Use an exponential moving average
# request_rate = 0.9 * request_rate + 0.1 * new_requests
# Option 3: Read from a shared metrics service like Redis
# request_rate = redis_client.get('recent_request_rate')
```
## Automatic job stopping
When a worker processes multiple jobs concurrently, individual jobs can be stopped without affecting sibling jobs. If a request expires, times out, or is cancelled, Runpod signals the worker to stop only that job while other in-progress jobs continue running.
This behavior is automatic and requires no handler changes. If your async handler needs to release resources when stopped, catch `asyncio.CancelledError`:
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
async def process_request(job):
resource = acquire_resource()
try:
await do_work(job)
return result
except asyncio.CancelledError:
release_resource(resource)
raise
```
Re-raise `CancelledError` after cleanup so the worker correctly marks the job as stopped.
## Next steps
Now that you've created a concurrent handler, you're ready to:
* [Package and deploy your handler as a Serverless worker.](/serverless/workers/deploy)
* [Add error handling for more robust processing.](/serverless/workers/handler-functions#error-handling)
* [Implement streaming responses with generator functions.](/serverless/workers/handler-functions#generator-handlers)
* [Configure your endpoint for optimal performance.](/serverless/endpoints/endpoint-configurations)
# Create a Dockerfile
Source: https://docs.runpod.io/serverless/workers/create-dockerfile
Package your handler function for deployment. Review setup, configuration, deployment, and operations guidance for Runpod Serverless.
A Dockerfile defines the build process for a Docker image containing your and all its dependencies. This page explains how to organize your project files and create a Dockerfile for your Serverless worker.
New to Docker? Learn the fundamentals with the [introduction to containers](/tutorials/introduction/containers) tutorial series, which covers [creating Dockerfiles](/tutorials/introduction/containers/create-dockerfiles), [essential Docker commands](/tutorials/introduction/containers/docker-commands), and [data persistence](/tutorials/introduction/containers/persist-data).
## Project organization
Organize your project files in a clear directory structure:
`/Dockerfile/` contains the instructions for building your worker image.
`/src/handler.py/` is your .
`/requirements.txt/` lists the Python dependencies required by your handler. For example:
```txt title="requirements.txt" theme={"theme":{"light":"github-light","dark":"github-dark"}}
# Example requirements.txt
runpod~=1.7.6
torch==2.0.1
pillow==9.5.0
transformers==4.30.2
```
## Basic Dockerfile structure
A basic Dockerfile for a Runpod Serverless worker follows this structure:
```dockerfile title="Dockerfile" theme={"theme":{"light":"github-light","dark":"github-dark"}}
FROM python:3.11.1-slim
WORKDIR /
# Copy and install requirements
COPY builder/requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
# Copy your handler code
COPY src/handler.py .
# Command to run when the container starts
CMD ["python", "-u", "/handler.py"]
```
This Dockerfile:
1. Starts with a Python base image.
2. Sets the working directory to the root.
3. Copies and installs Python dependencies.
4. Copies your handler code.
5. Specifies the command to run when the container starts.
## Choosing a base image
The base image you choose affects your image size, startup time, and available system dependencies. Common options include:
### Python slim images
Recommended for most use cases. These images are smaller and faster to download:
```dockerfile theme={"theme":{"light":"github-light","dark":"github-dark"}}
FROM python:3.11.1-slim
```
### Python full images
Include more system tools and libraries but are larger:
```dockerfile theme={"theme":{"light":"github-light","dark":"github-dark"}}
FROM python:3.11.1
```
### images
Required if you need libraries for GPU-accelerated workloads:
```dockerfile theme={"theme":{"light":"github-light","dark":"github-dark"}}
FROM nvidia/cuda:12.1.0-runtime-ubuntu22.04
# Install Python
RUN apt-get update && apt-get install -y python3.11 python3-pip
```
### Custom base images
You can build on top of specialized images for specific frameworks:
```dockerfile theme={"theme":{"light":"github-light","dark":"github-dark"}}
FROM pytorch/pytorch:2.0.1-cuda11.7-cudnn8-runtime
```
## Including models and files
If your model is available on Hugging Face, we strongly recommend enabling [cached models](/serverless/endpoints/model-caching) instead of baking/downloading the model into your Docker image. Cached models provide faster startup times, lower costs, and uses less storage.
### Baking models into the image
If you need to include model files or other assets in your image, use the `COPY` instruction:
```dockerfile title="Dockerfile" theme={"theme":{"light":"github-light","dark":"github-dark"}}
FROM python:3.11.1-slim
WORKDIR /
# Copy and install requirements
COPY builder/requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
# Copy your code and model files
COPY src/handler.py .
COPY models/ /models/
# Set environment variables if needed
ENV MODEL_PATH=/models/my_model.pt
# Command to run when the container starts
CMD ["python", "-u", "/handler.py"]
```
### Downloading models during build
You can download models during the Docker build process:
```dockerfile title="Dockerfile" theme={"theme":{"light":"github-light","dark":"github-dark"}}
# Download model files
RUN wget -q URL_TO_YOUR_MODEL -O /models/my_model.pt
# Or use a script to download from Hugging Face
RUN python -c "from transformers import AutoModel; AutoModel.from_pretrained('model-name')"
```
## Environment variables
Set environment variables to configure your application without hardcoding values:
```dockerfile title="Dockerfile" theme={"theme":{"light":"github-light","dark":"github-dark"}}
ENV MODEL_PATH=/models/my_model.pt
ENV LOG_LEVEL=INFO
ENV MAX_BATCH_SIZE=4
```
You can override these at runtime through the Runpod console when configuring your endpoint.
For details on how to access environment variables in your handler functions, see [Environment variables](/serverless/development/environment-variables).
## Optimizing image size
Smaller images download and start faster, reducing cold start times. Use these techniques to minimize image size:
### Use multi-stage builds
Multi-stage builds let you compile dependencies in one stage and copy only the necessary files to the final image:
```dockerfile title="Dockerfile" theme={"theme":{"light":"github-light","dark":"github-dark"}}
# Build stage
FROM python:3.11.1 AS builder
WORKDIR /build
COPY builder/requirements.txt .
RUN pip install --no-cache-dir --target=/build/packages -r requirements.txt
# Runtime stage
FROM python:3.11.1-slim
WORKDIR /
COPY --from=builder /build/packages /usr/local/lib/python3.11/site-packages
COPY src/handler.py .
CMD ["python", "-u", "/handler.py"]
```
### Clean up build artifacts
Remove unnecessary files after installation:
```dockerfile title="Dockerfile" theme={"theme":{"light":"github-light","dark":"github-dark"}}
RUN apt-get update && apt-get install -y build-essential \
&& pip install --no-cache-dir -r requirements.txt \
&& apt-get remove -y build-essential \
&& apt-get autoremove -y \
&& rm -rf /var/lib/apt/lists/*
```
### Use .dockerignore
Create a `.dockerignore` file to exclude unnecessary files from the build context:
```txt title=".dockerignore" theme={"theme":{"light":"github-light","dark":"github-dark"}}
.git
.gitignore
README.md
tests/
*.pyc
__pycache__/
.venv/
venv/
```
## Next steps
After creating your Dockerfile, you can:
* [Build and deploy your image from Docker Hub](/serverless/workers/deploy).
* [Deploy directly from GitHub](/serverless/workers/github-integration).
* [Test your handler locally](/serverless/development/local-testing) before building the image.
# Deploy workers from Docker Hub
Source: https://docs.runpod.io/serverless/workers/deploy
Build, test, and deploy your worker image from Docker Hub. Review setup, configuration, deployment, and operations guidance for Runpod Serverless.
After [creating a Dockerfile](/serverless/workers/create-dockerfile) for your worker, you can build the image, test it locally, and deploy it to a Serverless endpoint.
## Requirements
* A [Dockerfile](/serverless/workers/create-dockerfile) that packages your handler function.
* [Docker](https://docs.docker.com/get-started/get-docker/) installed on your development machine.
* A [Docker Hub](https://hub.docker.com/) account.
## Build the Docker image
From your terminal, navigate to your project directory and build the Docker image:
```sh theme={"theme":{"light":"github-light","dark":"github-dark"}}
docker build --platform linux/amd64 \
-t DOCKER_USERNAME/WORKER_NAME:VERSION .
```
Replace `DOCKER_USERNAME` with your Docker Hub username, `WORKER_NAME` with a descriptive name for your worker, and `VERSION` with an appropriate version tag.
The `--platform linux/amd64` flag is required to ensure compatibility with Runpod's infrastructure. This is especially important if you're building on an ARM-based system (like Apple Silicon Macs), as the default platform would be incompatible with Runpod's infrastructure.
## Test the image locally
Before pushing it to the registry, you should test your Docker image locally:
```sh theme={"theme":{"light":"github-light","dark":"github-dark"}}
docker run -it DOCKER_USERNAME/WORKER_NAME:VERSION
```
If your handler is properly configured with a [test input](/serverless/workers/handler-functions#local-testing), you should see it process the test input and provide output.
## Push the image to Docker Hub
Make your image available to Runpod by pushing it to Docker Hub:
```sh theme={"theme":{"light":"github-light","dark":"github-dark"}}
# Log in to Docker Hub
docker login
# Push the image
docker push DOCKER_USERNAME/WORKER_NAME:VERSION
```
Once your image is in the Docker container registry, you can [create a Serverless endpoint](/serverless/endpoints/overview#create-an-endpoint) through the Runpod console.
## Image versioning
For production workloads, use SHA tags for absolute reproducibility:
```sh theme={"theme":{"light":"github-light","dark":"github-dark"}}
# Get the SHA after pushing
docker inspect --format='{{index .RepoDigests 0}}' DOCKER_USERNAME/WORKER_NAME:VERSION
# Use the SHA when deploying
# DOCKER_USERNAME/WORKER_NAME:VERSION@sha256:4d3d4b3c5a5c2b3a5a5c3b2a5a4d2b3a2b3c5a3b2a5d2b3a3b4c3d3b5c3d4a3
```
Versioning best practices:
* **Avoid the `:latest` tag for production.** The `:latest` tag is mutable—it changes every time you push a new image without specifying a version. This creates several problems:
* **Unpredictable deployments**: You can't guarantee which version of your code is running, making it difficult to reproduce issues or roll back to a known-good state.
* **Debugging difficulties**: When problems occur, you won't know which exact image version caused them.
* **Caching conflicts**: Runpod caches images for faster worker startup. If you push a new `:latest` image, workers may continue using the cached version, leading to confusion about which code is actually running.
* Use semantic versioning (e.g., `v1.0.0`, `v1.0.1`) for clarity and easy rollbacks.
* Use SHA tags for absolute reproducibility in critical deployments.
* Document the specific image version or SHA in your deployment documentation.
* Keep images as small as possible for faster startup times.
Runpod validates your image name when you create an endpoint, and a reference that doesn't resolve to a published image is rejected at creation time. For example, `runpod/pytorch:latest` is rejected because `:latest` is not a published tag for Runpod's base images. When you deploy a Runpod base image, specify a tag that actually exists. You can browse the available tags for each image on [Docker Hub](https://hub.docker.com/u/runpod). For your own worker images, use a specific version or SHA tag as described above rather than `:latest`.
## Deploy an endpoint
If your files are hosted on GitHub, you can [deploy your worker directly from a GitHub repository](/serverless/workers/github-integration) through the Runpod console.
You can deploy your worker image directly from a Docker registry through the Runpod console:
1. Navigate to the [Serverless section](https://www.console.runpod.io/serverless) of the Runpod console.
2. Click **New Endpoint**.
3. Click **Import from Docker Registry**.
4. In the **Container Image** field, enter your Docker image URL (e.g., `docker.io/yourusername/worker-name:v1.0.0`), then click **Next**.
5. Configure your endpoint settings:
* Enter an **Endpoint Name**.
* Choose your **Endpoint Type**: select **Queue** for traditional queue-based processing or **Load Balancer** for direct HTTP access (see [Load balancing endpoints](/serverless/load-balancing/overview) for details).
* Under **GPU Configuration**, select the appropriate GPU types for your workload.
* Configure [other settings](/serverless/endpoints/endpoint-configurations) as needed (active/max workers, timeouts, environment variables).
6. Click **Deploy Endpoint** to deploy your worker.
## Troubleshoot deployment issues
If your worker fails to start or process requests:
1. Check the [logs](/serverless/development/logs) in the Runpod console for error messages.
2. Verify your handler function works correctly in [local testing](/serverless/development/local-testing).
3. Ensure all dependencies are properly installed in the [Docker image](/serverless/workers/create-dockerfile).
4. Check that your Docker image is compatible with the selected GPU type.
5. Verify your [input format](/serverless/endpoints/send-requests) matches what your handler expects.
# Deploy workers from GitHub
Source: https://docs.runpod.io/serverless/workers/github-integration
Speed up development by deploying workers directly from GitHub. Review configuration and operations guidance for Runpod Serverless.
Runpod's GitHub integration simplifies your workflow by pulling your code and Dockerfile from GitHub, building the container image, storing it in Runpod's secure container registry, and deploying it to your endpoint.
## Requirements
To deploy a worker from GitHub, you need:
* A working [handler function](/serverless/workers/handler-functions) in a GitHub repository.
* A Dockerfile in your repository. See [Creating a Dockerfile](/serverless/workers/deploy#creating-a-dockerfile) for details.
* A GitHub account.
For an example repository containing the minimal files necessary for deployment, see [runpod-workers/worker-basic](https://github.com/runpod-workers/worker-basic) on GitHub.
## Authorize Runpod with GitHub
Before deploying from GitHub, you need to authorize Runpod to access your repositories:
1. Open the [settings page](https://console.runpod.io/user/settings) in the Runpod console.
2. Find the **GitHub** card under **Connections** and click **Connect**.
3. Sign in using the GitHub authorization flow. This will open your GitHub account settings page.
4. Choose which repositories Runpod can access:
* **All repositories:** Access to all current and future repositories.
* **Only select repositories:** Choose specific repositories.
5. Click **Save**.
You can manage this connection using Runpod settings or GitHub account settings, in the **Applications** tab.
## Deploy from GitHub
To deploy a worker from a GitHub repository:
1. Go to the [Serverless section](https://www.console.runpod.io/serverless) of the Runpod console
2. Click **New Endpoint**
3. Under **Import Git Repository**, use the search bar or menu to select the repository containing your code. This menu is populated with all repos connected to your account (repos you've forked/created, or owned by your GitHub organizations).
4. Configure your deployment options:
* **Branch:** Select which branch to deploy from.
* **Dockerfile Path:** Specify the path to your Dockerfile (if not in root).
Then click **Next**.
5. Configure your endpoint settings:
* Enter an **Endpoint Name**.
* Choose your **Endpoint Type**: select **Queue** for traditional queue-based processing or **Load Balancer** for direct HTTP access (see [Load balancing endpoints](/serverless/load-balancing/overview) for details).
* Under **GPU Configuration**, select the appropriate GPU types for your workload.
* Configure [other settings](/serverless/endpoints/endpoint-configurations) as needed (active/max workers, timeouts, environment variables).
6. Click **Deploy Endpoint** to deploy your worker.
Runpod will build your Docker image and deploy it to your endpoint automatically. You'll be redirected to the endpoint details page when complete.
## Monitor build status
You can monitor your build status in the **Builds** tab of your endpoint detail page. Builds progress through these statuses:
| Status | Description |
| --------- | --------------------------------------------------- |
| Pending | Runpod is scheduling the build. |
| Building | Runpod is building your container. |
| Uploading | Runpod is uploading your container to the registry. |
| Testing | Runpod is testing your Serverless worker. |
| Completed | Runpod completed the build and upload. |
| Failed | Something went wrong (check build logs). |
## Update your endpoint
When you make changes to your GitHub repository, they won't automatically be pushed to your endpoint. To trigger an update for the workers on your endpoint, create a new release for the GitHub repository.
For detailed instructions on creating releases, see the [GitHub documentation](https://docs.github.com/en/repositories/releasing-projects-on-github/managing-releases-in-a-repository).
## Roll back to a previous build
Roll back your endpoint to any previous build directly from the Runpod console. This restores your endpoint to an earlier version without waiting for a new GitHub release.
### Requirements
To roll back an endpoint, you need:
* An existing endpoint deployed with GitHub integration.
* At least one previous build available in your deployment history.
### Roll back a deployment
To roll back your endpoint to a previous build:
Open the endpoint details page in the [Serverless section](https://www.console.runpod.io/serverless) of the Runpod console.
Click the **Builds** tab to view your deployment history.
Find the build you want to roll back to, then click the three dots menu button next to that build.
Select **Rollback** from the menu.
Review the confirmation modal and click **Confirm** to proceed with the rollback.
After confirming, your endpoint rolls back to the selected build. A banner appears at the top of the endpoint page indicating the endpoint is on a rolled-back version.
### Rollback behavior
When you roll back an endpoint:
* Your endpoint immediately switches to the Docker image from the selected previous build.
* The rollback banner displays at the top of your endpoint page to indicate the current state.
* Your endpoint remains on the rolled-back version until you deploy a new release from GitHub.
* When you push a new commit and create a release, the new build automatically becomes the active version and supersedes the rollback.
## Manage multiple environments
GitHub integration enables streamlined development workflows by supporting multiple environments:
* Production endpoint tracking the `main` branch.
* Staging endpoint tracking the `dev` branch.
To set up multiple environments:
1. Create a new branch for your staging endpoint.
2. [Create an endpoint](#deploying-from-github) for your production branch.
3. On the Serverless page of the Runpod console, click the three dots to the top right of your production endpoint. Click **Clone Endpoint**.
4. Expand the **Repository Configuration** section and select your staging branch.
5. Click **Deploy Endpoint**.
Each environment maintains independent GPU and worker configurations.
## Continuous integration with GitHub Actions
You can enhance your workflow with GitHub Actions for testing before deployment:
1. Create a workflow file at `.github/workflows/test-and-deploy.yml`:
```yml theme={"theme":{"light":"github-light","dark":"github-dark"}}
name: Test and Deploy
on:
push:
branches: [ main ]
pull_request:
branches: [ main ]
jobs:
test-and-deploy:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- name: Build and push Docker image
uses: docker/build-push-action@v4
with:
context: .
push: true
tags: [DOCKER_USERNAME]/[WORKER_NAME]:${{ github.sha }}
- name: Run Tests
uses: runpod/runpod-test-runner@v1
with:
image-tag: [DOCKER_USERNAME]/[WORKER_NAME]:${{ github.sha }}
runpod-api-key: ${{ secrets.RUNPOD_API_KEY }} # Add your API key to a GitHub secret
test-filename: .github/tests.json
request-timeout: 300
```
To add your Runpod API key to a GitHub secret, see [Using secrets in GitHub Actions](https://docs.github.com/en/actions/how-tos/write-workflows/choose-what-workflows-do/use-secrets).
2. Create test cases for your repository at `.github/tests.json`:
```json theme={"theme":{"light":"github-light","dark":"github-dark"}}
[
{
"input": {
"prompt": "Test input 1"
},
"expected_output": {
"status": "COMPLETED"
}
},
{
"input": {
"prompt": "Test input 2",
"parameter": "value"
},
"expected_output": {
"status": "COMPLETED"
}
}
]
```
## Troubleshoot deployment issues
If your worker fails to deploy or process requests:
* Check the build logs in the Runpod console for error messages.
* Verify your Dockerfile is properly configured.
* Ensure your handler function works correctly in local testing.
* Check that your repository structure matches what's expected in your Dockerfile.
* Verify you have the necessary permissions on the GitHub repository.
## Disconnect from GitHub
To disconnect your GitHub account from Runpod:
1. Go to [Runpod Settings](https://www.console.runpod.io/user/settings) → **Connections** → **Edit Connection**
2. Select your GitHub account.
3. Click **Configure**.
4. Scroll down to the Danger Zone.
5. Uninstall "Runpod Inc."
## Limitations
Runpod has the following limitations when using the GitHub integration to deploy your worker:
* **Build time limit**: The GitHub integration has two separate time limits:
* **Total build window**: The entire build process (cloning, building, uploading, and testing) must complete within 160 minutes (2.5 hours).
* **Docker build timeout**: The `docker build` step must complete within 30 minutes. If your dependencies or model weights take longer to download and install, pre-build your image locally and push to a container registry instead.
If you see an error like "Build exceeded maximum time limit of 1800 seconds (30.0 minutes)", this refers to the Docker build timeout, not the overall 160-minute limit.
* **Image size restriction**: Docker images cannot exceed 80 GB. Plan your image requirements accordingly, particularly when including large model weights or dependencies.
* **Base image limitations**: The integration doesn't support privately hosted images as base images. Consider incorporating essential components directly into your Dockerfile instead.
* **Hardware-specific builds**: Builds requiring GPU access during construction (such as those using GPU-compiled versions of libraries like `bitsandbytes`) are not supported.
* **Platform exclusivity**: Images built through Runpod's image builder service are designed exclusively for Runpod's infrastructure and cannot be pulled or executed on other platforms.
* **Single GitHub connection**: Each Runpod account can link to only one GitHub account. This connection cannot be shared among team members, requiring separate Runpod accounts for collaborative projects.
# Overview
Source: https://docs.runpod.io/serverless/workers/handler-functions
Write custom handler functions to process incoming requests to your queue-based endpoints. See Runpod Serverless setup and usage details.
Handler functions form the core of your Runpod Serverless applications. They define how your process and return results. This section covers everything you need to know about creating effective handler functions.
Handler functions are only required for . If you're building a , you can define your own custom API endpoints using any HTTP framework of your choice (like FastAPI or Flask).
## Understanding job input
Before writing a handler function, make sure you understand the structure of the input. When your endpoint receives a request, it sends a JSON object to your handler function in this general format:
```json theme={"theme":{"light":"github-light","dark":"github-dark"}}
{
"id": "eaebd6e7-6a92-4bb8-a911-f996ac5ea99d",
"input": {
"key": "value"
}
}
```
`id` is a unique identifier for the randomly generated by Runpod, while `input` contains data sent by the client for your handler function to process.
To learn how to structure requests to your endpoint, see [Send API requests](/serverless/endpoints/send-requests).
## Basic handler implementation
Here's a simple handler function that processes an endpoint request:
```python handler.py theme={"theme":{"light":"github-light","dark":"github-dark"}}
import runpod
def handler(job):
job_input = job["input"] # Access the input from the request
# Add your custom code here to process the input
return "Your job results"
runpod.serverless.start({"handler": handler}) # Required
```
The handler takes extracts the input from the job request, processes it, and returns a result. The `runpod.serverless.start()` function launches your serverless application with the specified handler.
## Local testing
To test your handler locally, you can create a `test_input.json` file with the input data you want to test:
```json test_input.json theme={"theme":{"light":"github-light","dark":"github-dark"}}
{
"input": {
"prompt": "Hey there!"
}
}
```
Then run your handler function using your local terminal:
```sh theme={"theme":{"light":"github-light","dark":"github-dark"}}
python handler.py
```
Instead of creating a `test_input.json` file, you can also provide test input directly in the command line prompt:
```sh theme={"theme":{"light":"github-light","dark":"github-dark"}}
python handler.py --test_input '{"input": {"prompt": "Test prompt"}}'
```
For more information on local testing, including command-line flags and starting a local API server, see [Local testing](/serverless/development/local-testing).
## Handler types
You can create several types of handler functions depending on the needs of your application.
### Standard handlers
The simplest handler type, standard handlers process inputs synchronously and return them when the job is complete.
```python handler.py theme={"theme":{"light":"github-light","dark":"github-dark"}}
import runpod
import time
def handler(job):
job_input = job["input"]
prompt = job_input.get("prompt")
seconds = job_input.get("seconds", 0)
# Simulate processing time
time.sleep(seconds)
return prompt
runpod.serverless.start({"handler": handler})
```
### Streaming handlers
Streaming handlers stream results incrementally as they become available. Use these when your application requires real-time updates, for example when streaming results from a language model.
```python handler.py theme={"theme":{"light":"github-light","dark":"github-dark"}}
import runpod
def streaming_handler(job):
for count in range(3):
result = f"This is the {count} generated output."
yield result
runpod.serverless.start({
"handler": streaming_handler,
"return_aggregate_stream": True # Optional, makes results available via /run
})
```
By default, outputs from streaming handlers are only available using the `/stream` operation. Set `return_aggregate_stream` to `True` to make outputs available from the `/run` and `/runsync` operations as well.
To learn more about aggregating streaming outputs, including best practices for batch processing and handling local testing, see [Aggregate streaming outputs](/serverless/development/aggregate-outputs).
### Asynchronous handlers
Asynchronous handlers process operations concurrently for improved efficiency. Use these for tasks involving I/O operations, API calls, or processing large datasets.
```python handler.py theme={"theme":{"light":"github-light","dark":"github-dark"}}
import runpod
import asyncio
async def async_handler(job):
for i in range(5):
# Generate an asynchronous output token
output = f"Generated async token output {i}"
yield output
# Simulate an asynchronous task
await asyncio.sleep(1)
runpod.serverless.start({
"handler": async_handler,
"return_aggregate_stream": True
})
```
Async handlers allow your code to handle multiple tasks concurrently without waiting for each operation to complete. This approach offers excellent scalability for applications that deal with high-frequency requests, allowing your workers to remain responsive even under heavy load. Async handlers are also useful for streaming data scenarios and long-running tasks that produce incremental outputs.
When implementing async handlers, ensure proper use of `async` and `await` keywords throughout your code to maintain truly non-blocking operations and prevent performance bottlenecks, and consider leveraging the `yield` statement to generate outputs progressively over time.
Always test your async code thoroughly to properly handle asynchronous exceptions and edge cases, as async error patterns can be more complex than in synchronous code.
When a job is stopped due to timeout, expiration, or cancellation, async handlers receive an `asyncio.CancelledError`. Catch this exception to release resources, then re-raise it:
```python handler.py theme={"theme":{"light":"github-light","dark":"github-dark"}}
async def async_handler(job):
resource = acquire_resource()
try:
await process(job)
return result
except asyncio.CancelledError:
release_resource(resource)
raise
```
### Concurrent handlers
Concurrent handlers process multiple requests simultaneously with a single worker. Use these for small, rapid operations that don't fully utlize the worker's GPU.
When increasing concurrency, it's crucial to monitor memory usage carefully and test thoroughly to determine the optimal concurrency levels for your specific workload. Implement proper error handling to prevent one failing request from affecting others, and continuously monitor and adjust concurrency parameters based on real-world performance.
Learn how to build a concurrent handler by [following this guide](/serverless/workers/concurrent-handler).
## Error handling
When an exception occurs in your handler function, the Runpod SDK automatically captures it, marks the [job status](/serverless/endpoints/job-states) as `FAILED` and returns the exception details in the job results.
For custom error responses:
```python handler.py theme={"theme":{"light":"github-light","dark":"github-dark"}}
import runpod
def handler(job):
job_input = job["input"]
# Validate the presence of required inputs
if not job_input.get("seed", False):
return {
"error": "Input is missing the 'seed' key. Please include a seed."
}
# Proceed if the input is valid
return "Input validation successful."
runpod.serverless.start({"handler": handler})
```
Exercise caution when using `try/except` blocks to avoid unintentionally suppressing errors. Either return the error for a graceful failure or raise it to flag the job as `FAILED`.
## Advanced handler controls
Use these features to fine-tune your Serverless applications for specific use cases.
### Progress updates
Send progress updates during job execution to inform clients about the current state of processing:
```python handler.py theme={"theme":{"light":"github-light","dark":"github-dark"}}
import runpod
def handler(job):
for update_number in range(0, 3):
runpod.serverless.progress_update(job, f"Update {update_number}/3")
return "done"
runpod.serverless.start({"handler": handler})
```
Progress updates will be available when the job status is polled.
### Worker refresh
For long-running or complex jobs, you may want to refresh the worker after completion to start with a clean state for the next job. Enabling worker refresh clears all logs and wipes the worker state after a job is completed.
For example:
```python handler.py theme={"theme":{"light":"github-light","dark":"github-dark"}}
# Requires runpod python version 0.9.0+
import runpod
import time
def handler(job):
job_input = job["input"] # Access the input from the request
results = []
# Compute results
...
# Return the results and indicate the worker should be refreshed
return {"refresh_worker": True, "job_results": results}
# Configure and start the Runpod serverless function
runpod.serverless.start(
{
"handler": handler, # Required: Specify the sync handler
"return_aggregate_stream": True, # Optional: Aggregate results are accessible via /run operation
}
)
```
Your handler must return a dictionary that contains the `refresh_worker` flag. This flag will be removed before the remaining job output is returned.
## Handler function best practices
A short list of best practices to keep in mind as you build your handler function:
1. **Initialize outside the handler**: Load models and other heavy resources outside your handler function to avoid repeated initialization.
```python handler.py theme={"theme":{"light":"github-light","dark":"github-dark"}}
import runpod
import torch
from transformers import AutoModelForSequenceClassification, AutoTokenizer
# Load model and tokenizer outside the handler
model_name = "distilbert-base-uncased-finetuned-sst-2-english"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForSequenceClassification.from_pretrained(model_name)
# Move model to GPU if available
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
model.to(device)
def handler(job):
# ...
runpod.serverless.start({"handler": handler})
```
2. **Input validation**: [Validate inputs](#error-handling) before processing to avoid errors during execution.
3. **Local testing**: [Test your handlers locally](/serverless/development/local-testing) before deployment.
## Payload limits
Be aware of payload size limits when designing your handler:
* `/run` operation: 10 MB
* `/runsync` operation: 20 MB
If your results exceed these limits, consider stashing them in cloud storage and returning links instead.
## Next steps
Once you've created your handler function, you can:
* [Learn how to aggregate streaming outputs.](/serverless/development/aggregate-outputs)
* [Explore flags for local testing.](/serverless/development/local-testing)
* [Create a Dockerfile for your worker.](/serverless/workers/create-dockerfile)
* [Deploy your worker image to a Serverless endpoint.](/serverless/workers/deploy)
# Overview
Source: https://docs.runpod.io/serverless/workers/overview
Package your handler function for deployment. Review setup, configuration, deployment, and operations guidance for Runpod Serverless.
Workers are containerized environments that run your code on Runpod Serverless.
## Deployment workflow
After creating your [handler function](/serverless/workers/handler-functions), package it into a Docker image and deploy it to an endpoint:
Package your handler function and all its dependencies [into a Docker image](/serverless/workers/create-dockerfile).
Push your image and create an endpoint using one of two methods:
* [Deploy from Docker Hub](/serverless/workers/deploy): Build locally and push to a container registry.
* [Deploy from GitHub](/serverless/workers/github-integration): Auto-build and deploy directly from your repository.
## Model deployment
To deploy workers with AI/ML models, follow this order of preference:
1. [**Use cached models**](/serverless/endpoints/model-caching): For models on Hugging Face (public or gated), this is the recommended approach. Cached models provide the fastest cold starts and persist across worker restarts.
2. [**Bake the model into your Docker image**](/serverless/workers/create-dockerfile#including-models-and-files): For private models not on Hugging Face, embed them directly in your container image. This increases image size and initialization time, but ensures the model is stored directly on machine disk before the worker starts, so it's ready to serve requests quickly after billing begins.
3. [**Use network volumes**](/storage/network-volumes): For development workflows or very large models (500GB+), store models on a network volume. Reading from network volume will occur after a worker starts running and billing begins, and will be slower than reading cached or baked models directly from machine disk. However, worker initialization time will be faster, and changes to the underlying image will not require rebuilding and repushing the image with the model artifact, making it faster to iterate during development.
## Worker types
Workers can run in two modes depending on your latency and cost requirements:
* **Active workers** run continuously (24/7) and are always ready to process requests instantly. They eliminate cold starts entirely, making them ideal for latency-sensitive or high-traffic applications.
* **Flex workers** scale dynamically based on demand, spinning down to zero when idle. They incur cold starts when scaling up but cost nothing when not in use, making them ideal for variable or sporadic workloads.
The system may also spin up **extra workers** during traffic spikes when Docker images are cached on hosts (default: 2).
## Worker states
| State | Description | Billing |
| ---------------- | --------------------------------------------------------------------------------------- | ---------------------- |
| **Initializing** | Downloading image, loading code, and downloading cached models (if using model caching) | No |
| **Idle** | Scaled down, waiting for requests | No |
| **Running** | Processing requests | Yes |
| **Throttled** | Temporarily unable to run due to host resource constraints | No |
| **Outdated** | Marked for replacement after update | Yes (while processing) |
| **Unhealthy** | Crashed; auto-retries for up to 7 days | No |
If an endpoint repeatedly produces unhealthy workers, Runpod automatically scales it down; see [My endpoint was scaled down unexpectedly](/serverless/troubleshooting#my-endpoint-was-scaled-down-unexpectedly).
View worker states in the **Workers** tab of your endpoint in the [Runpod console](https://www.console.runpod.io/serverless).
## Max worker limits
Account balance determines your maximum workers (flex + active combined):
| Balance | Max workers |
| ------- | ----------- |
| Default | 5 |
| \$100+ | 10 |
| \$200+ | 20 |
| \$300+ | 30 |
| \$500+ | 40 |
| \$700+ | 50 |
| \$900+ | 60 |
Need more capacity? [Contact support](https://www.runpod.io/contact).
## Best practices
| Practice | Benefit |
| ------------------------------------------------------------ | ------------------------------------- |
| [Optimize image size](/serverless/workers/create-dockerfile) | Faster downloads, reduced cold starts |
| [Use model caching](/serverless/endpoints/model-caching) | Fastest cold starts |
| [Test locally first](/serverless/development/local-testing) | Catch issues before deployment |
| [Use logs and SSH](/serverless/development/logs) | Debug and optimize effectively |
# High-performance storage
Source: https://docs.runpod.io/storage/high-performance-storage
Premium storage tier for demanding AI workloads with up to 3x throughput and 4x IOPS. Review configuration and usage guidance for Runpod storage.
High-performance storage is a network volume tier designed for data-intensive AI workloads. Its parallel storage architecture delivers high throughput and low latency under heavy concurrent load, even when thousands of processes access it simultaneously.
## Performance comparison
In internal benchmarks, high-performance storage delivers up to 3x the throughput and 4x the IOPS of standard storage on parallel operations.
| | Standard storage | High-performance storage |
| ------------- | ------------------------------------------------------ | -------------------------------------------------- |
| Best for | General-purpose work, development, cost-sensitive jobs | Training, fine-tuning, latency-sensitive inference |
| Throughput | High | Up to 3x higher |
| IOPS | High | Up to 4x higher |
| Concurrency | Moderate | Optimized for massive parallelism |
| Relative cost | Lower | Premium |
Both tiers are fully supported. Choose based on whether storage performance sits on the critical path of your workload.
## When to choose High-performance storage
If your GPUs ever sit idle waiting on data (slow dataloaders, long checkpoint writes, sluggish model loads) you're a candidate for high-performance storage. If storage isn't your bottleneck, standard storage is the cost-effective choice.
| Use case | Why it helps |
| ----------------------------- | ---------------------------------------------------------------------------------------------------- |
| Training | Feeds GPUs at the rate they can consume data, preventing I/O bottlenecks and reducing training time. |
| Fine-tuning and checkpointing | Speeds up write-heavy operations like checkpoints, so jobs spend more time computing. |
| Inference | Cuts model load time from a central volume, lowering endpoint cold-start latency. |
## Create a high-performance volume
High-performance storage is available as a network volume type in select data centers.
1. Navigate to the [Storage page](https://console.runpod.io/user/storage) in the Runpod console.
2. Click **Create network volume**.
3. Select a data center that supports high-performance storage. You can either select the **High-performance** tab to filter data centers that support it, or look for the purple ⚡ icon on individual data centers.
4. Enter a name and set the volume size in GB.
5. The **High-performance storage** toggle appears automatically and is enabled by default. Leave it on to use high-performance storage, or turn it off to use standard storage instead.
6. Click **Create network volume**.
## Attach to your workloads
High-performance volumes work across the platform like standard network volumes.
* ****: Attach during Pod creation, or from the volume manager for existing setups.
* ****: Attach to an endpoint to speed up model loading and reduce cold-start latency. See [Network volumes for Serverless](/storage/network-volumes#network-volumes-for-serverless).
* **s**: New clusters can be provisioned with high-performance storage by default.
## Pricing
High-performance storage is priced per-GB at a premium to standard storage. The console displays per-GB and total monthly cost as you configure a volume.
Exact pricing varies by data center. Check the volume creation flow in the console for current rates.
## FAQ
**Is standard storage going away?**
No. Standard storage remains fully supported and is the recommended choice for general-purpose work. High-performance storage is an additional tier for performance-sensitive workloads.
**Which data centers support High-performance storage?**
Multiple US and European data centers support High-performance storage, with more rolling out over time. Supported data centers are marked with a purple ⚡ icon in the network volume creation flow.
**Can I migrate an existing standard volume to high-performance?**
Direct migration is not currently supported. To move data to a high-performance volume, create a new high-performance volume and transfer your data using the [S3-compatible API](/storage/s3-api) or [runpodctl](/storage/network-volumes#using-runpodctl).
## Next steps
Learn more about network volume management and features.
Manage files without launching compute resources.
# Network volumes
Source: https://docs.runpod.io/storage/network-volumes
Persistent, portable storage for your AI workloads. Review configuration, access methods, and usage guidance for managing storage on Runpod.
Network volumes provide persistent storage that exists independently of your compute resources. Data is retained when terminate or scale to zero. Use them to share data across multiple machines and Runpod products.
Network volumes are backed by high-performance NVMe SSDs with transfer speeds of 200-400 MB/s (up to 10 GB/s peak).
## Storage tiers
Network volumes are available in two tiers:
* **Standard storage**: Cost-effective storage for general-purpose work, development, and workloads where storage isn't the bottleneck.
* **[High-performance storage](/storage/high-performance-storage)**: Premium tier with up to 3x throughput and 4x IOPS, optimized for training, fine-tuning, and latency-sensitive inference.
Select the storage tier when creating a network volume. For details on choosing between tiers, see [High-performance storage](/storage/high-performance-storage).
## Pricing
**Standard storage:**
* **First 1 TB**: \$0.07/GB/month
* **Beyond 1 TB**: \$0.05/GB/month
**High-performance storage** is priced at a premium. See the [High-performance storage](/storage/high-performance-storage#pricing) page for details.
When your account balance reaches \$0, Pods with an attached network volume are stopped and your data is preserved on the volume. Storage charges continue to accrue while the Pod is stopped, so if your balance stays at \$0 and these charges can't be covered, the network volume may eventually be terminated and its data cannot be recovered. Enable [low balance notifications](/accounts-billing/billing#low-balance-notifications) to get alerted before this happens.
## Create a network volume
Volume size can be increased later but cannot be decreased. For volumes beyond 4 TB, [contact support](https://www.runpod.io/contact).
1. Navigate to the [Storage page](https://www.console.runpod.io/user/storage).
2. Click **New Network Volume**.
3. Select a data center, enter a name, and specify size in GB.
4. Select a storage tier: **Standard** or **High-Performance** (available in select data centers).
5. Click **Create Network Volume**.
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
curl --request POST \
--url https://rest.runpod.io/v1/networkvolumes \
--header 'Authorization: Bearer RUNPOD_API_KEY' \
--header 'Content-Type: application/json' \
--data '{
"name": "my-network-volume",
"size": 100,
"dataCenterId": "US-KS-2"
}'
```
See [network volumes API reference](/api-reference/network-volumes/POST/networkvolumes) for details.
## Network volumes for Serverless
Network volumes mount at `/runpod-volume` within Serverless workers. Benefits include reduced times (no re-downloading models), lower costs, and centralized data management.
**Attach to an endpoint:**
1. Go to [Serverless](https://www.console.runpod.io/serverless/user/endpoints) and select your endpoint.
2. Click **Manage** → **Edit Endpoint**.
3. Expand **Advanced**, click **Network Volumes**, and select volumes to attach.
4. Click **Save Endpoint**.
Writing to the same volume from multiple workers simultaneously may cause data corruption. Handle concurrent write access in your application logic.
### Attach multiple volumes
Attaching a single network volume constrains worker deployments to that volume's datacenter, which may limit GPU availability and reduce failover options.
To improve availability and reduce downtime during datacenter maintenance, attach multiple network volumes from different datacenters. Workers are distributed across these datacenters, with each worker receiving exactly one volume based on its assigned location.
You can only select one network volume per datacenter.
**Data does not sync automatically between volumes.** To make the same data available to all workers regardless of datacenter, manually copy data using the [S3-compatible API](/storage/s3-api) or [runpodctl](#using-runpodctl).
## Network volumes for Pods
Network volumes replace the Pod's default volume disk, typically mounted at `/workspace`.
Network volumes are only available for Pods in the [Secure Cloud](/pods/overview#pod-types).
**Attach to a Pod:**
1. Navigate to [Pods](https://www.console.runpod.io/pods) and click **Deploy**.
2. Select **Network Volume** and choose your volume.
3. Select a GPU type (available options depend on volume location).
4. Configure template and other settings, then click **Deploy On-Demand**.
Network volumes must be attached during Pod deployment. They cannot be attached or detached later without deleting the Pod.
## Network volumes for Instant Clusters
Network volumes for s work like Pods. Attach during cluster creation; mounts at `/workspace` on each node.
1. Go to [Instant Clusters](https://www.console.runpod.io/cluster) and click **Create Cluster**.
2. Click **Network Volume** and select the volume to attach.
3. Configure other settings and click **Deploy Cluster**.
## S3-compatible API
The [S3-compatible API](/storage/s3-api) lets you manage files on network volumes without launching compute resources. Upload datasets before launching Pods, automate workflows with standard S3 tools, or pre-populate volumes to improve cold start performance.
## Migrate files between volumes
### Using runpodctl
The simplest way to migrate files between network volumes is to use `runpodctl send` and `receive` on two running Pods:
Deploy Pods with the source and destination volumes attached. Open web terminals on both.
On the source Pod:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
cd /workspace
runpodctl send *
```
Copy the receive command from the output.
On the destination Pod:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
cd /workspace
runpodctl receive 8338-galileo-collect-fidel # Use your code
```
### Using rsync over SSH
For faster migration speed and more reliability for large transfers, you can use `rsync` over SSH on two running Pods:
Deploy Pods with source and destination volumes attached.
On the source Pod, install required packages and generate an SSH key pair:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
apt update && apt install -y rsync
ssh-keygen -t ed25519 -f ~/.ssh/id_ed25519 -N "" -q
cat ~/.ssh/id_ed25519.pub
```
Copy the public key.
On the destination Pod, install required packages and add the source Pod's public key to `authorized_keys`:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
apt update && apt install -y vim rsync && \
ip=$(printenv RUNPOD_PUBLIC_IP) && \
port=$(printenv RUNPOD_TCP_PORT_22) && \
echo "rsync -avzP --inplace -e \"ssh -p $port\" /workspace/ root@$ip:/workspace" && \
vi ~/.ssh/authorized_keys
```
In the editor that opens, paste the public key you copied from the source Pod, then save and exit (press `Esc`, type `:wq`, and press `Enter`).
The command above also displays the `rsync` command you'll need to run on the source Pod. Copy this command for the next step.
On the source Pod, run the `rsync` command from the previous step.
If you didn't copy it, you can construct it manually using the destination Pod's IP address and port number:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
# Replace DESTINATION_PORT and DESTINATION_IP with values from the destination Pod
rsync -avzP --inplace -e "ssh -p DESTINATION_PORT" /workspace/ root@DESTINATION_IP:/workspace
# Example:
rsync -avzP --inplace -e "ssh -p 18598" /workspace/ root@157.66.254.13:/workspace
```
You can run the `rsync` command multiple times if the transfer is interrupted. The `--inplace` flag ensures that `rsync` resumes from where it left off rather than starting over.
# S3-compatible API
Source: https://docs.runpod.io/storage/s3-api
Use Runpod's S3-compatible API to access and manage your network volumes. Review configuration and usage guidance for Runpod storage.
Runpod provides an S3-protocol compatible API for direct access to your [network volumes](/storage/network-volumes). This allows you to manage files on your network volumes without launching a Pod, reducing cost and operational friction.
Using the S3-compatible API does not affect pricing. Network volumes are billed hourly at \$0.07/GB/month for the first 1TB, and \$0.05/GB/month for additional storage.
[Watch this video](https://www.youtube.com/watch?v=XA01UEE4TYc) for a quickstart example demonstrating how to automatically sync a network volume with a local directory. You can download the syncing script on GitHub: [/runpod/examples/s3-api-downloader](https://github.com/runpod/examples/tree/main/s3-api-downloader).
## Datacenter availability
The S3-compatible API is available for network volumes in select datacenters. Each datacenter has a unique endpoint URL that you'll use when calling the API:
| Datacenter | Endpoint URL |
| ---------- | ----------------------------------- |
| `EU-CZ-1` | `https://s3api-eu-cz-1.runpod.io/` |
| `EU-RO-1` | `https://s3api-eu-ro-1.runpod.io/` |
| `EUR-IS-1` | `https://s3api-eur-is-1.runpod.io/` |
| `EUR-NO-1` | `https://s3api-eur-no-1.runpod.io/` |
| `US-CA-2` | `https://s3api-us-ca-2.runpod.io/` |
| `US-GA-2` | `https://s3api-us-ga-2.runpod.io/` |
| `US-IL-1` | `https://s3api-us-il-1.runpod.io/` |
| `US-KS-2` | `https://s3api-us-ks-2.runpod.io/` |
| `US-MD-1` | `https://s3api-us-md-1.runpod.io/` |
| `US-MO-1` | `https://s3api-us-mo-1.runpod.io/` |
| `US-MO-2` | `https://s3api-us-mo-2.runpod.io/` |
| `US-NC-1` | `https://s3api-us-nc-1.runpod.io/` |
| `US-NC-2` | `https://s3api-us-nc-2.runpod.io/` |
| `US-NE-1` | `https://s3api-us-ne-1.runpod.io/` |
| `US-WA-1` | `https://s3api-us-wa-1.runpod.io/` |
Create your network volume in a supported datacenter to use the S3-compatible API.
## Setup and authentication
First, create a network volume in a [supported datacenter](#datacenter-availability). See [Network volumes -> Create a network volume](/storage/network-volumes#create-a-network-volume) for detailed instructions.
Next, you'll need to generate a new key called an "S3 API key" (this is separate from your Runpod API key).
1. Go to the [Settings page](https://www.console.runpod.io/user/settings) in the Runpod console.
2. Expand **S3 API Keys** and select **Create an S3 API key**.
3. Name your key and select **Create**.
4. Save the **access key** (e.g., `user_***...`) and **secret** (e.g., `rps_***...`) to use in the next step.
For security, Runpod will show your API key secret only once, so you may wish to save it elsewhere (e.g., in your password manager, or in a GitHub secret). Treat your API key secret like a password and don't share it with anyone.
To use the S3-compatible API with your Runpod network volumes, you must configure your AWS CLI with the Runpod S3 API key you created.
1. If you haven't already, [install the AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html) on your local machine.
2. Run the command `aws configure` in your terminal.
3. Provide the following when prompted:
* **AWS Access Key ID**: Enter your Runpod user ID. You can find this in the [Secrets section](https://www.console.runpod.io/user/secrets) of the Runpod console, in the description of your S3 API key. By default, the description will look similar to: `Shared Secret for user_2f21CfO73Mm2Uq2lEGFiEF24IPw 1749176107073`. `user_2f21CfO73Mm2Uq2lEGFiEF24IPw` is the user ID (yours will be different).
* **AWS Secret Access Key**: Enter your Runpod S3 API key's secret access key.
* **Default Region name**: You can leave this blank.
* **Default output format**: You can leave this blank or set it to `json`.
This will configure the AWS CLI to use your Runpod S3 API key by storing these details in your AWS credentials file (typically at `~/.aws/credentials`).
### Verifying your AWS configuration
If you're experiencing authentication issues, use the following command to check your current AWS configuration:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
aws configure list
```
This command displays which credentials are currently active, the source of each credential (such as the config file or environment variables), and whether all required credentials are properly set.
### Environment variables override config files
AWS CLI uses the following priority order for credentials (highest to lowest):
1. **Environment variables** (`AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`)
2. **AWS credentials file** (`~/.aws/credentials`)
3. **AWS config file** (`~/.aws/config`)
If you have environment variables set from a previous session, they will override your config file settings. To resolve this:
1. Check for existing environment variables:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
echo $AWS_ACCESS_KEY_ID
echo $AWS_SECRET_ACCESS_KEY
```
2. If outdated environment variables are set, unset them:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
unset AWS_ACCESS_KEY_ID
unset AWS_SECRET_ACCESS_KEY
```
3. Verify your configuration again:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
aws configure list
```
## Using the S3-compatible API
You can use the S3-compatible API to interact with your Runpod network volumes using standard S3 tools:
* [AWS s3 CLI](https://awscli.amazonaws.com/v2/documentation/api/latest/reference/s3/index.html).
* [AWS s3api CLI](https://docs.aws.amazon.com/cli/latest/reference/s3api/).
* [The Boto3 Python library](https://docs.aws.amazon.com/boto3/latest/reference/services/s3.html).
Standard AWS CLI operations such as `ls`, `cp`, `mv`, and `rm` work as expected for most file operations. The `sync` command works for basic use cases but may encounter issues with large numbers of files (10,000+) or complex directory structures.
Network volumes are mounted to Serverless workers at `/runpod-volume` and to Pods at `/workspace` by default. The S3-compatible API maps file paths as follows:
* **Pod filesystem path**: `/workspace/my-folder/file.txt`
* **Serverless worker path**: `/runpod-volume/my-folder/file.txt`
* **S3 API path**: `s3://NETWORK_VOLUME_ID/my-folder/file.txt`
## s3 CLI examples
When using `aws s3` commands, you must pass in the [endpoint URL](#datacenter-availability) for your network volume using the `--endpoint-url` flag and the datacenter ID using the `--region` flag.
Unlike traditional S3 key-value stores, object names in the Runpod S3-compatible API correspond to actual file paths on your network volume. Object names containing special characters (e.g., `#`) may need to be URL-encoded to ensure proper processing.
Use `ls` to list objects in a network volume directory:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
aws s3 ls --region DATACENTER \
--endpoint-url https://s3api-DATACENTER.runpod.io/ \
s3://NETWORK_VOLUME_ID/REMOTE_DIR
```
Unlike standard S3 buckets, `ls` and `ListObjects` operations will list empty directories.
`ls` operations may take a long time when used on a directory containing many files (over 10,000) or large amounts of data (over 10GB), or when used recursively on a network volume containing either.
Use `cp` to copy a file to a network volume:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
aws s3 cp --region DATACENTER \
--endpoint-url https://s3api-DATACENTER.runpod.io/ \
LOCAL_FILE \
s3://NETWORK_VOLUME_ID
```
Use `cp` to copy a file from a network volume to a local directory:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
aws s3 cp --region DATACENTER \
--endpoint-url https://s3api-DATACENTER.runpod.io/ \
s3://NETWORK_VOLUME_ID/remote-file.txt LOCAL_DIR
```
Use `rm` to remove a file from a network volume:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
aws s3 rm --region DATACENTER \
--endpoint-url https://s3api-DATACENTER.runpod.io/ \
s3://NETWORK_VOLUME_ID/remote-file.txt
```
If you encounter a 502 "bad gateway" error during file transfer, try increasing `AWS_MAX_ATTEMPTS` to 10 or more:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
export AWS_RETRY_MODE=standard
export AWS_MAX_ATTEMPTS=10
```
This command syncs a local directory (source) to a network volume directory (destination):
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
aws s3 sync --region DATACENTER \
--endpoint-url https://s3api-DATACENTER.runpod.io/ \
LOCAL_DIR \
s3://NETWORK_VOLUME_ID/REMOTE_DIR
```
## s3api CLI example
You can also use `aws s3api` commands (instead of `aws s3`) to interact with the S3-compatible API.
For example, here's how you could use `aws s3api get-object` to download an object from a network volume:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
aws s3api get-object --bucket NETWORK_VOLUME_ID \
--key REMOTE_FILE \
--region DATACENTER \
--endpoint-url https://s3api-DATACENTER.runpod.io/ \
LOCAL_FILE
```
Replace `LOCAL_FILE` with the desired path and name of the file after download—for example: `~/local-dir/my-file.txt`.
For a list of available `s3api` commands, see the [AWS s3api reference](https://docs.aws.amazon.com/cli/latest/reference/s3api/).
## Boto3 Python example
You can also use the Boto3 library to interact with the S3-compatible API, using it to transfer files to and from a Runpod network volume.
The script below demonstrates how to upload a file to a Runpod network volume using the Boto3 library. It takes command-line arguments for the network volume ID (as an S3 bucket), the datacenter-specific S3 endpoint URL, the local file path, the desired object (file path on the network volume), and the AWS Region (which corresponds to the Runpod datacenter ID).
To run this script, your Runpod S3 API key credentials must be set as environment variables using the values from the [Setup and authentication](#setup-and-authentication) step:
* `AWS_ACCESS_KEY_ID`: Should be set to your Runpod S3 API key **access key** (e.g., `user_***...`).
* `AWS_SECRET_ACCESS_KEY`: Should be set to your Runpod S3 API key's **secret** (e.g., `rps_***...`).
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
#!/usr/bin/env python3
import os
import argparse
import boto3 # AWS SDK for Python, used to interact with Runpod S3-compatible APIs
def create_s3_client(region: str, endpoint_url: str):
# Creates and returns an S3 client configured for Runpod network volume S3-compatible API.
#
# Args:
# region (str): The Runpod datacenter ID, used as the AWS region
# (e.g., 'ca-qc-1').
# endpoint_url (str): The S3 endpoint URL for the specific Runpod datacenter
# (e.g., 'https://ca-qc-1-s3api.runpod.io/').
# Returns:
# boto3.client: An S3 client object, configured for the Runpod S3 API.
# Retrieve Runpod S3 API key credentials from environment variables.
aws_access_key_id = os.environ.get("AWS_ACCESS_KEY_ID")
aws_secret_access_key = os.environ.get("AWS_SECRET_ACCESS_KEY")
# Ensure necessary S3 API key credentials are set in the environment
if not aws_access_key_id or not aws_secret_access_key:
raise EnvironmentError(
"Please set AWS_ACCESS_KEY_ID (with S3 API Key Access Key) and "
"AWS_SECRET_ACCESS_KEY (with S3 API Key Secret Access Key) environment variables. "
"These are obtained from 'S3 API Keys' in the Runpod console settings."
)
# Initialize and return the S3 client for Runpod's S3-compatible API
return boto3.client(
"s3",
aws_access_key_id=aws_access_key_id,
aws_secret_access_key=aws_secret_access_key,
region_name=region, # Corresponds to the Runpod datacenter ID
endpoint_url=endpoint_url, # Datacenter-specific S3 API endpoint
)
def put_object(s3_client, bucket_name: str, object_name: str, file_path: str):
# Uploads a local file to the specified Runpod network volume.
#
# Args:
# s3_client: The S3 client object (e.g., returned by create_s3_client).
# bucket_name (str): The ID of the target Runpod network volume.
# object_name (str): The desired file path for the object on the network volume.
# file_path (str): The local path to the file (including the filename) that will be uploaded.
try:
# Attempt to upload the file to the Runpod network volume.
s3_client.upload_file(file_path, bucket_name, object_name)
print(f"Successfully uploaded '{file_path}' to Network Volume '{bucket_name}' as '{object_name}'")
except Exception as e:
# Catch any exception during upload, print an error, and re-raise
print(f"Error uploading file '{file_path}' to Network Volume '{bucket_name}' as '{object_name}': {e}")
raise
def main():
# Parses command-line arguments and orchestrates the file upload process
# to a Runpod network volume.
# Set up command-line argument parsing
parser = argparse.ArgumentParser(
description="Upload a file to a Runpod Network Volume using its S3-compatible API. "
"Requires AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY env vars to be set "
"with your Runpod S3 API key credentials."
)
parser.add_argument(
"-b", "--bucket",
required=True,
help="The ID of your Runpod Network Volume (acts as the S3 bucket name)."
)
parser.add_argument(
"-e", "--endpoint",
required=True,
help="The S3 endpoint URL for your Runpod datacenter (e.g., 'https://s3api-DATACENTER.runpod.io/')."
)
parser.add_argument(
"-f", "--file",
required=True,
help="The local path to the file to be uploaded."
)
parser.add_argument(
"-o", "--object",
required=True,
help="The S3 object key (i.e., the desired file path on the Network Volume)."
)
parser.add_argument(
"-r", "--region",
required=True,
help="The Runpod datacenter ID, used as the AWS region (e.g., 'ca-qc-1'). Find this in the Runpod console's Storage section or endpoint URL."
)
args = parser.parse_args()
# Create the S3 client using the parsed arguments, configured for Runpod.
client = create_s3_client(args.region, args.endpoint)
# Upload the object to the specified network volume.
put_object(client, args.bucket, args.object, args.file)
if __name__ == "__main__":
main()
```
When uploading files with Boto3, you must specify the complete file path (including the filename) for both source and destination files.
For example, for the `put_objects` method above, you must specify these arguments:
* `file_path`: The local source file (e.g., `local_directory/file.txt`).
* `object_name`: The remote destination file to be created on the network volume (e.g., `remote_directory/file.txt`).
With that in mind, here's an example of how to run the script above using command-line arguments:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
./s3_example_put.py --endpoint https://s3api-eur-is-1.runpod.io/ \
--region 'EUR-IS-1' \
--bucket 'network_volume_id' \
--object 'remote_directory/file.txt' \
--file 'local_directory/file.txt'
```
## Uploading very large files
You can upload large files to network volumes using S3 multipart upload operations (see the [compatibility reference](#s3-api-compatibility-reference) below). You can also download [this helper script](https://github.com/runpod/runpod-s3-examples/blob/main/upload_large_file.py), which dramatically improves reliability when uploading very large files (10GB+) by handling timeouts and retries automatically.
[Click here to download the script on GitHub.](https://github.com/runpod/runpod-s3-examples/blob/main/upload_large_file.py)
Here's an example of how to run the script using command line arguments:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
./upload_large_file.py --file /path/to/large/file.mp4 \
--bucket NETWORK_VOLUME_ID \
--access_key YOUR_ACCESS_KEY_ID \
--secret_key YOUR_SECRET_ACCESS_KEY \
--endpoint https://s3api-eur-is-1.runpod.io/ \
--region EUR-IS-1
```
## S3 API compatibility reference
The tables below show which S3 API operations and AWS CLI commands are currently supported. Use the tables below to understand what functionality is available and plan your development workflows accordingly.
For detailed information on these operations, refer to the [AWS S3 API documentation](https://docs.aws.amazon.com/AmazonS3/latest/API/API_Operations_Amazon_Simple_Storage_Service.html).
If a function is not listed below, that means it's not currently implemented. We are continuously expanding the S3-compatible API based on user needs and usage patterns.
| Operation | Supported | CLI Command | Notes |
| --------------- | --------- | ---------------------------------------- | ----------------------------------------------------- |
| `CopyObject` | ✅ | `aws s3 cp`, `aws s3api copy-object` | Copy objects between locations |
| `DeleteObject` | ✅ | `aws s3 rm`, `aws s3api delete-object` | Remove individual objects |
| `GetObject` | ✅ | `aws s3 cp`, `aws s3api get-object` | Download objects |
| `HeadBucket` | ✅ | `aws s3 ls`, `aws s3api head-bucket` | Verify bucket exists and permissions |
| `HeadObject` | ✅ | `aws s3api head-object` | Retrieve object metadata |
| `ListBuckets` | ✅ | `aws s3 ls`, `aws s3api list-buckets` | List available network volumes |
| `ListObjects` | ✅ | `aws s3 ls`, `aws s3api list-objects` | List objects in a bucket (includes empty directories) |
| `ListObjectsV2` | ✅ | `aws s3 ls`, `aws s3api list-objects-v2` | Enhanced version of ListObjects |
| `PutObject` | ✅ | `aws s3 cp`, `aws s3api put-object` | Upload objects (\<500MB) |
| `DeleteObjects` | ❌ | `aws s3api delete-objects` | Planned |
| `RestoreObject` | ❌ | `aws s3api restore-object` | Not supported |
`ListObjects` operations may take a long time when used on a directory containing many files (over 10,000) or large amounts of data (over 10GB), or when used recursively on a network volume containing either.
Files larger than 500MB must be uploaded using multipart uploads. The AWS CLI performs multipart uploads automatically.
| Operation | Supported | CLI Command | Notes |
| ------------------------- | --------- | ------------------------------------- | -------------------------------------- |
| `CreateMultipartUpload` | ✅ | `aws s3api create-multipart-upload` | Start multipart upload for large files |
| `UploadPart` | ✅ | `aws s3api upload-part` | Upload individual parts |
| `CompleteMultipartUpload` | ✅ | `aws s3api complete-multipart-upload` | Finish multipart upload |
| `AbortMultipartUpload` | ✅ | `aws s3api abort-multipart-upload` | Cancel multipart upload |
| `ListMultipartUploads` | ✅ | `aws s3api list-multipart-uploads` | View in-progress uploads |
| `ListParts` | ✅ | `aws s3api list-parts` | List parts of a multipart upload |
| Operation | Supported | CLI Command | Notes |
| ---------------------- | --------- | --------------------------------- | ------------------------------------------------ |
| `CreateBucket` | ❌ | `aws s3api create-bucket` | Use the Runpod console to create network volumes |
| `DeleteBucket` | ❌ | `aws s3api delete-bucket` | Use the Runpod console to delete network volumes |
| `GetBucketLocation` | ❌ | `aws s3api get-bucket-location` | Datacenter info available in the Runpod console |
| `GetBucketVersioning` | ❌ | `aws s3api get-bucket-versioning` | Versioning is not supported |
| `PutBucketVersioning` | ❌ | `aws s3api put-bucket-versioning` | Versioning is not supported |
| `GeneratePresignedURL` | ❌ | `aws s3 presign` | Pre-signed URLs are not supported |
| Operation | Supported | CLI Command | Notes |
| ----------------- | --------- | ----------- | --------------------------------- |
| `GetBucketAcl` | ❌ | N/A | ACLs are not supported |
| `PutBucketAcl` | ❌ | N/A | ACLs are not supported |
| `GetObjectAcl` | ❌ | N/A | ACLs are not supported |
| `PutObjectAcl` | ❌ | N/A | ACLs are not supported |
| `GetBucketPolicy` | ❌ | N/A | Bucket policies are not supported |
| `PutBucketPolicy` | ❌ | N/A | Bucket policies are not supported |
| Operation | Supported | CLI Command | Notes |
| --------------------- | --------- | ----------- | ------------------------------- |
| `GetObjectTagging` | ❌ | N/A | Object tagging is not supported |
| `PutObjectTagging` | ❌ | N/A | Object tagging is not supported |
| `DeleteObjectTagging` | ❌ | N/A | Object tagging is not supported |
| Operation | Supported | CLI Command | Notes |
| ---------------------------- | --------- | ----------- | ------------------------------- |
| `GetBucketEncryption` | ❌ | N/A | Encryption is not supported |
| `PutBucketEncryption` | ❌ | N/A | Encryption is not supported |
| `GetObjectLockConfiguration` | ❌ | N/A | Object locking is not supported |
| `PutObjectLockConfiguration` | ❌ | N/A | Object locking is not supported |
## Known issues and limitations
When running `aws s3 ls` or `ListObjects` on a directory with many files or large amounts of data (typically >10,000 files or >10 GB of data) for the first time, it may run very slowly, or you may encounter the following error:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
"fatal error: Error during pagination: The same next token was received twice: ..."
```
This occurs because Runpod must compute and cache the MD5 checksum (i.e., ETag) for files created without the S3-compatible API. This computation can take several minutes for large directories or files, as the `ListObjects` request must wait until the checksum is ready.
Workarounds:
* The operation will typically complete successfully if you wait for the process to finish.
* If the client aborts with a pagination error, retry the operation after a brief pause.
* **Storage capacity**: Network volumes have a fixed storage capacity, unlike the virtually unlimited storage of standard S3 buckets. The `CopyObject` and `UploadPart` actions do not check for available free space beforehand and may fail if the volume runs out of space.
* **Maximum file size**: 4TB (the maximum size of a network volume).
* **Object names**: Unlike traditional S3 key-value stores, object names in the Runpod S3-compatible API correspond to actual file paths on your network volume. Object names containing special characters (e.g., `#`) may need to be URL encoded to ensure proper processing.
* **Time synchronization**: Requests that are out of time sync by 1 hour will be rejected. This is more lenient than the 15-minute window specified by the AWS SigV4 authentication specification.
* The maximum size for a single part of a multipart upload is 500MB.
* The AWS S3 minimum part size of 5MB is not enforced.
* Multipart upload parts and metadata are stored in a hidden `.s3compat_uploads/` folder. This folder and its contents are automatically cleaned up when you call `CompleteMultipartUpload` or `AbortMultipartUpload`.
The `aws s3 sync` command has limited support in Runpod's S3-compatible API. While it works for basic use cases (syncing small numbers of files and simple directory structures), you may encounter errors when syncing directories with very large numbers of files (over 10,000) or complex nested structures.
Common issues include:
* EOF errors with 200 OK responses.
* Duplicate ContinuationToken errors in ListObjectsV2.
* Intermittent AccessDenied errors.
To sync large numbers of files, consider breaking your sync operations into smaller batches or using individual `cp` commands for better reliability. Full `sync` support is in development.
When uploading large files (10GB+), you may encounter timeout errors during the `CompleteMultipartUpload` operation. To resolve this, we recommend using the [multipart upload helper script](#uploading-very-large-files).
Or you can try increasing the timeout settings in your AWS tools:
For `aws s3` and `aws s3api`, use the `--cli-read-timeout` parameter:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
# Sets the timeout to 7200 seconds (2 hours)
aws s3 cp large-file.zip s3://your-volume-name/ --cli-read-timeout 7200 --endpoint-url https://storage.datacenter.runpod.io
```
Or, configure timeout in `~/.aws/config`:
```ini theme={"theme":{"light":"github-light","dark":"github-dark"}}
[default]
cli_read_timeout = 7200
```
Use the `read_timeout` parameter to configure the timeout when creating the S3 client:
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
import boto3
from botocore.config import Config
# Sets the timeout to 7200 seconds (2 hours)
custom_config = Config(
read_timeout=7200,
)
# Create S3 client with custom timeout
s3_client = boto3.client('s3', config=custom_config)
```
## Reference documentation
For comprehensive documentation on AWS S3 commands and libraries, refer to:
* [AWS CLI S3 reference](https://awscli.amazonaws.com/v2/documentation/api/latest/reference/s3/index.html).
* [AWS S3 API reference](https://docs.aws.amazon.com/AmazonS3/latest/API/API_Operations_Amazon_Simple_Storage_Service.html).
* [Boto3 S3 reference](https://docs.aws.amazon.com/boto3/latest/reference/services/s3.html).
# Using TMUX for persistent sessions
Source: https://docs.runpod.io/tips-and-tricks/tmux
Use TMUX on a Runpod Pod to keep terminal sessions and long-running workloads active through disconnects or JupyterLab interruptions.
TMUX is a terminal multiplexer that creates persistent terminal sessions on your Pod.
## Why use TMUX
While you can run long-running tasks in JupyterLab notebooks, the kernel can crash unexpectedly, losing hours or days of computation. TMUX runs your commands in persistent terminal sessions that continue even if JupyterLab crashes or you disconnect.
## Requirements
To use TMUX, you need terminal access to your Runpod Pod (SSH, web terminal, or JupyterLab terminal).
## Video tutorial
For a comprehensive video guide on using TMUX, check out this excellent tutorial: [TMUX Tutorial on YouTube](https://www.youtube.com/watch?v=nTqu6w2wc68)
## Installing TMUX
TMUX is not installed by default on Runpod Pods. To install it, run:
```bash Install TMUX theme={"theme":{"light":"github-light","dark":"github-dark"}}
apt-get update && apt-get install -y tmux
```
Most Runpod templates are based on Ubuntu/Debian images that support `apt-get`. If you're using Alpine Linux, use `apk add tmux` instead. Some minimal images may lack required dependencies.
## Practical example: running model training
Here's a typical workflow for running long-duration model training:
```bash Training workflow theme={"theme":{"light":"github-light","dark":"github-dark"}}
# Start a new TMUX session
tmux new -s model_training
# Navigate to your project directory
cd /workspace/my_project
# Start your training script
python train.py --epochs 100 --batch-size 32
# Detach from the session with Ctrl+B, then D
# You can now safely disconnect from the Pod
# Later, reconnect to the Pod and reattach
tmux attach -t model_training
```
## Command reference
### Starting a new session
Create a new TMUX session with a descriptive name:
```bash Create named session theme={"theme":{"light":"github-light","dark":"github-dark"}}
tmux new -s training
```
### Detaching from a session
To detach from a session and leave it running in the background, press:
```
Ctrl+B, then D
```
Your processes will continue running even if you disconnect from the Pod.
### Listing active sessions
View all active TMUX sessions:
```bash List sessions theme={"theme":{"light":"github-light","dark":"github-dark"}}
tmux ls
```
### Reattaching to a session
Reconnect to a previously created session:
```bash Reattach to session theme={"theme":{"light":"github-light","dark":"github-dark"}}
tmux attach -t training
```
### Killing a session
End a session when you no longer need it:
```bash Kill session theme={"theme":{"light":"github-light","dark":"github-dark"}}
tmux kill-session -t training
```
### Killing all sessions
If you need to restart fresh and kill all TMUX sessions:
```bash Kill all sessions theme={"theme":{"light":"github-light","dark":"github-dark"}}
tmux kill-server
```
Alternatively, you can kill all sessions except the current one:
```bash Kill other sessions theme={"theme":{"light":"github-light","dark":"github-dark"}}
tmux kill-session -a
```
## Advanced TMUX features
### Window management
TMUX supports multiple windows within a session:
* `Ctrl+B, C` - Create new window
* `Ctrl+B, N` - Next window
* `Ctrl+B, P` - Previous window
* `Ctrl+B, 0-9` - Switch to window by number
### Pane splitting
Split your terminal into multiple panes:
* `Ctrl+B, %` - Split vertically
* `Ctrl+B, "` - Split horizontally
* `Ctrl+B, Arrow keys` - Navigate between panes
* `Ctrl+B, X` - Close current pane
### Scrolling and copy mode
To scroll through terminal output:
1. Enter copy mode: `Ctrl+B, [`
2. Use arrow keys or Page Up/Down to scroll
3. Press `q` to exit copy mode
## Best practices
When using TMUX on Runpod Pods, keep these tips in mind:
* Always name your sessions descriptively to easily identify them later.
* Regularly check on long-running processes by reattaching to sessions.
* Clean up finished sessions to avoid confusion.
* Use TMUX for any process that takes longer than a few minutes to complete.
* Consider creating a TMUX session immediately when connecting to a Pod for important work.
## Troubleshooting
If you can't reattach to a session after reconnecting to a Pod, the Pod may have been restarted. TMUX sessions don't persist across Pod restarts, so ensure you save your work regularly and use network volumes for persistent storage.
# Build a REST API with Flash
Source: https://docs.runpod.io/tutorials/flash/build-rest-api-with-load-balancer
Learn how to build a production-ready REST API using Flash load-balanced endpoints with custom HTTP routes.
This tutorial shows you how to build a REST API using Flash load-balanced endpoints. You'll create a multi-route API that handles text processing, demonstrates both CPU and GPU endpoints, and deploys to production.
## Requirements
* You've [created a Runpod account](/accounts-billing/manage-accounts)
* You've [created a Runpod API key](/get-started/api-keys)
* You've installed [Python 3.10, 3.11, 3.12, or 3.13](https://www.python.org/downloads/).
* You've completed the [Flash quickstart](/flash/quickstart) or are familiar with Flash basics
## What you'll build
By the end of this tutorial, you'll have a working REST API that:
* Accepts text input via `POST /analyze`
* Returns system health via `GET /health`
* Provides API information via `GET /info`
* Runs GPU-accelerated sentiment analysis via `POST /sentiment` (optional GPU route)
* Deploys to Runpod Serverless with proper authentication
## Step 1: Set up your project
Create a new directory for your project and set up a Python virtual environment:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
mkdir flash-api
cd flash-api
```
Install Flash using [uv](https://docs.astral.sh/uv/):
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
uv venv
source .venv/bin/activate
uv pip install runpod-flash
```
Set your API key in the environment:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
export RUNPOD_API_KEY=YOUR_API_KEY
# Or create a .env file
echo "RUNPOD_API_KEY=YOUR_API_KEY" > .env
```
Replace `YOUR_API_KEY` with your actual Runpod API key.
## Step 2: Create the API server file
Create a new file called `api.py`:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
touch api.py
```
## Step 3: Define the load-balanced endpoint
Add the following code to `api.py`:
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
from runpod_flash import Endpoint
# CPU load-balanced endpoint for general API routes
api = Endpoint(
name="text-api",
cpu="cpu5c-4-8", # 4 vCPU, 8GB RAM
workers=(0, 3), # Scale from 0 to 3 workers
idle_timeout=600 # Keep workers active for 10 minutes
)
```
This configuration creates a CPU load-balanced endpoint that can handle multiple HTTP routes.
**Worker Quota Considerations**: The `workers` setting determines the maximum number of concurrent workers. Standard Runpod accounts have a total quota of 30 workers across all endpoints. If you have other endpoints running, you may need to reduce `workers` to `(0, 1)`. Check your quota in the [Runpod console](https://console.runpod.io/serverless).
## Step 4: Add API routes
Add three routes to your API - health check, info, and text analysis:
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
@api.get("/health")
async def health_check() -> dict:
"""Health check endpoint for monitoring."""
return {
"status": "healthy",
"service": "text-api",
"version": "1.0.0"
}
@api.get("/info")
async def get_info() -> dict:
"""API information endpoint."""
return {
"name": "Text Analysis API",
"version": "1.0.0",
"endpoints": [
{"method": "GET", "path": "/health", "description": "Health check"},
{"method": "GET", "path": "/info", "description": "API information"},
{"method": "POST", "path": "/analyze", "description": "Analyze text"}
]
}
@api.post("/analyze")
async def analyze_text(text: str) -> dict:
"""Analyze text and return statistics."""
words = text.split()
word_count = len(words)
char_count = len(text)
avg_word_length = sum(len(word) for word in words) / word_count if word_count > 0 else 0
return {
"text": text,
"statistics": {
"word_count": word_count,
"character_count": char_count,
"average_word_length": round(avg_word_length, 2),
"sentence_count": text.count('.') + text.count('!') + text.count('?')
}
}
```
All three routes share the same `api` endpoint, meaning they deploy to a single Serverless endpoint.
## Step 5: Add a GPU-accelerated route (optional)
For GPU-accelerated sentiment analysis, add a separate endpoint:
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
from runpod_flash import Endpoint, GpuGroup
# GPU endpoint for ML inference
gpu_api = Endpoint(
name="gpu-sentiment",
gpu=GpuGroup.ANY, # Use any available GPU for better availability
workers=(0, 1), # Scale from 0 to 1 worker
idle_timeout=300, # 5 minutes
dependencies=["transformers", "torch"]
)
@gpu_api.post("/sentiment")
async def analyze_sentiment(text: str) -> dict:
"""Analyze sentiment using a pretrained model."""
from transformers import pipeline
import torch
# Load sentiment analysis pipeline
device = 0 if torch.cuda.is_available() else -1
sentiment_analyzer = pipeline(
"sentiment-analysis",
model="distilbert-base-uncased-finetuned-sst-2-english",
device=device
)
# Analyze sentiment
result = sentiment_analyzer(text)[0]
return {
"text": text,
"sentiment": {
"label": result["label"],
"score": round(result["score"], 4)
},
"device": "GPU" if torch.cuda.is_available() else "CPU"
}
```
This creates a second endpoint specifically for GPU-accelerated tasks.
The sentiment analysis route uses a separate GPU endpoint because it requires different hardware than the CPU routes. This is a common pattern: use CPU endpoints for lightweight API logic and GPU endpoints for ML inference.
**GPU Availability**: Using `GpuGroup.ANY` provides better availability than specific GPU types like `GpuGroup.ADA_24`. First requests to GPU endpoints may take 3-10 minutes due to:
* GPU provisioning (depends on current availability)
* Dependency installation (transformers, torch)
* Model downloads (distilbert is \~250MB)
During high demand periods, GPU provisioning may take longer. Check [GPU availability](https://console.runpod.io/serverless) in the console.
## Step 6: Add the main execution block
Add the following at the end of `api.py` to enable local testing:
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
import asyncio
async def main():
"""Test the API locally."""
print("Testing Text Analysis API\n")
# Test health check
print("1. Testing health check...")
health = await health_check()
print(f" Result: {health}\n")
# Test info endpoint
print("2. Testing info endpoint...")
info = await get_info()
print(f" Result: {info}\n")
# Test text analysis
print("3. Testing text analysis...")
sample_text = "Flash makes it easy to build REST APIs with GPU acceleration."
analysis = await analyze_text(sample_text)
print(f" Result: {analysis}\n")
# Test sentiment analysis (if GPU route is defined)
print("4. Testing sentiment analysis...")
try:
sentiment = await analyze_sentiment(sample_text)
print(f" Result: {sentiment}\n")
except ModuleNotFoundError as e:
print(f" Skipped (dependencies not installed locally): {e}")
print(f" Note: This will work when deployed to Flash with dependencies=['transformers', 'torch']\n")
if __name__ == "__main__":
asyncio.run(main())
```
## Step 7: Test locally
Run your script to test the API locally:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
python api.py
```
You should see output similar to:
```text theme={"theme":{"light":"github-light","dark":"github-dark"}}
Testing Text Analysis API
1. Testing health check...
Result: {'status': 'healthy', 'service': 'text-api', 'version': '1.0.0'}
2. Testing info endpoint...
Result: {'name': 'Text Analysis API', 'version': '1.0.0', 'endpoints': [...]}
3. Testing text analysis...
Result: {'text': '...', 'statistics': {'word_count': 11, ...}}
4. Testing sentiment analysis...
Skipped (dependencies not installed locally): No module named 'transformers'
Note: This will work when deployed to Flash with dependencies=['transformers', 'torch']
```
The first three endpoints will run locally. The sentiment endpoint will be skipped unless you install transformers and torch locally, but it will work when deployed to Flash.
**Local Testing Limitations**: The GPU sentiment endpoint requires `transformers` and `torch` to be installed locally for testing. For full testing of all endpoints including GPU routes, use `flash dev` (covered in Step 9) instead of direct Python execution.
## Step 8: Build a Flash app for production
To deploy your API to production, create a Flash app:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
flash init api-project
cd api-project
# If using uv:
uv run flash init api-project
```
This creates a project structure with separate worker files. Now, split your API code into the appropriate worker files:
### Create `lb_worker.py` (CPU routes):
Replace the contents of `lb_worker.py` with:
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
from runpod_flash import Endpoint
# CPU load-balanced endpoint for general API routes
api = Endpoint(
name="text-api",
cpu="cpu5c-4-8", # 4 vCPU, 8GB RAM
workers=(0, 3), # Scale from 0 to 3 workers
idle_timeout=600 # Keep workers active for 10 minutes
)
@api.get("/health")
async def health_check() -> dict:
"""Health check endpoint for monitoring."""
return {
"status": "healthy",
"service": "text-api",
"version": "1.0.0"
}
@api.get("/info")
async def get_info() -> dict:
"""API information endpoint."""
return {
"name": "Text Analysis API",
"version": "1.0.0",
"endpoints": [
{"method": "GET", "path": "/health", "description": "Health check"},
{"method": "GET", "path": "/info", "description": "API information"},
{"method": "POST", "path": "/analyze", "description": "Analyze text"}
]
}
@api.post("/analyze")
async def analyze_text(text: str) -> dict:
"""Analyze text and return statistics."""
words = text.split()
word_count = len(words)
char_count = len(text)
avg_word_length = sum(len(word) for word in words) / word_count if word_count > 0 else 0
return {
"text": text,
"statistics": {
"word_count": word_count,
"character_count": char_count,
"average_word_length": round(avg_word_length, 2),
"sentence_count": text.count('.') + text.count('!') + text.count('?')
}
}
```
### Create `gpu_worker.py` (GPU route):
If you added the GPU sentiment route, replace the contents of `gpu_worker.py` with:
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
from runpod_flash import Endpoint, GpuGroup
# GPU endpoint for ML inference
gpu_api = Endpoint(
name="gpu-sentiment",
gpu=GpuGroup.ANY, # Use any available GPU for better availability
workers=(0, 1), # Scale from 0 to 1 worker
idle_timeout=300, # 5 minutes
dependencies=["transformers", "torch"]
)
@gpu_api.post("/sentiment")
async def analyze_sentiment(text: str) -> dict:
"""Analyze sentiment using a pretrained model."""
from transformers import pipeline
import torch
# Load sentiment analysis pipeline
device = 0 if torch.cuda.is_available() else -1
sentiment_analyzer = pipeline(
"sentiment-analysis",
model="distilbert-base-uncased-finetuned-sst-2-english",
device=device
)
# Analyze sentiment
result = sentiment_analyzer(text)[0]
return {
"text": text,
"sentiment": {
"label": result["label"],
"score": round(result["score"], 4)
},
"device": "GPU" if torch.cuda.is_available() else "CPU"
}
```
### Configure environment:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
cp .env.example .env
echo "RUNPOD_API_KEY=YOUR_API_KEY" > .env
```
Replace `YOUR_API_KEY` with your actual Runpod API key.
## Step 9: Test with the development server
Start the Flash development server:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
flash dev
# If using uv:
uv run flash dev
```
You'll see output showing all available endpoints:
```text theme={"theme":{"light":"github-light","dark":"github-dark"}}
Flash Dev Server localhost:8888
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━┓
┃ Local path ┃ Description ┃ Type ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━┩
│ GET /lb_worker/health │ Health check endpoint for monitoring. │ LB │
│ GET /lb_worker/info │ API information endpoint. │ LB │
│ POST /lb_worker/analyze │ Analyze text and return statistics. │ LB │
│ POST /gpu_worker/sentiment │ Analyze sentiment using a pretrained │ LB │
│ │ model. │ │
└─────────────────────────────┴─────────────────────────────────────────┴──────┘
```
**Development Server Path Prefixes**: The `flash dev` server adds worker file prefixes to routes (e.g., `/lb_worker/health`, `/gpu_worker/sentiment`). When deployed to production, endpoints use the paths as defined in the route decorators (e.g., `/health`, `/sentiment`) without the prefixes.
Open [http://localhost:8888/docs](http://localhost:8888/docs) in your browser to see the interactive API documentation. You can test all your routes directly in the Swagger UI.
Test with curl:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
# Test health check
curl -X GET http://localhost:8888/lb_worker/health
# Test text analysis
curl -X POST http://localhost:8888/lb_worker/analyze \
-H "Content-Type: application/json" \
-d '{"text": "Flash makes building APIs easy"}'
# Test sentiment analysis (if you added the GPU route)
# Note: First request may take 1-3 minutes for GPU provisioning and model download
curl -X POST http://localhost:8888/gpu_worker/sentiment \
-H "Content-Type: application/json" \
-d '{"text": "I love using Flash for my APIs"}'
```
Expected responses:
```json theme={"theme":{"light":"github-light","dark":"github-dark"}}
// Health check
{
"status": "healthy",
"service": "text-api",
"version": "1.0.0"
}
// Text analysis
{
"text": "Flash makes building APIs easy",
"statistics": {
"word_count": 5,
"character_count": 30,
"average_word_length": 5.2,
"sentence_count": 0
}
}
// Sentiment analysis
{
"text": "I love using Flash for my APIs",
"sentiment": {
"label": "POSITIVE",
"score": 0.9998
},
"device": "GPU"
}
```
**GPU Cold Starts**: The first request to a GPU endpoint may take 3-10 minutes due to GPU provisioning, dependency installation, and model downloads. During high demand periods, provisioning may take longer. Subsequent requests will be much faster. The default timeout is 60 seconds, which may be too short for the first request. If you encounter timeout errors, wait and retry - the GPU may still be initializing.
## Step 10: Deploy to production
When you're ready to deploy, use `flash deploy`:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
flash deploy
# If using uv:
uv run flash deploy
```
After deployment, Flash displays your endpoint URLs:
```text theme={"theme":{"light":"github-light","dark":"github-dark"}}
✓ Deployment Complete
Load-balanced endpoints:
https://api-abc123.runpod.net (text-api)
GET /health
GET /info
POST /analyze
https://api-def456.runpod.net (gpu-sentiment)
POST /sentiment
```
## Step 11: Call your deployed API
Call your production endpoints with authentication:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
# Health check
curl -X GET https://api-abc123.runpod.net/health \
-H "Authorization: Bearer $RUNPOD_API_KEY"
# Text analysis
curl -X POST https://api-abc123.runpod.net/analyze \
-H "Authorization: Bearer $RUNPOD_API_KEY" \
-H "Content-Type: application/json" \
-d '{"text": "Flash makes building APIs easy and fast"}'
# GPU sentiment analysis
curl -X POST https://api-def456.runpod.net/sentiment \
-H "Authorization: Bearer $RUNPOD_API_KEY" \
-H "Content-Type: application/json" \
-d '{"text": "I love using Flash for my APIs"}'
```
Expected response:
```json theme={"theme":{"light":"github-light","dark":"github-dark"}}
{
"text": "I love using Flash for my APIs",
"sentiment": {
"label": "POSITIVE",
"score": 0.9998
},
"device": "GPU"
}
```
**Production Path Note**: In production, the endpoints use the exact paths defined in your route decorators (e.g., `/health`, `/sentiment`), without the worker file prefixes used in `flash dev`.
## Understanding the deployment architecture
Your deployed API creates two independent Serverless endpoints:
```mermaid theme={"theme":{"light":"github-light","dark":"github-dark"}}
%%{init: {'theme':'base', 'themeVariables': { 'primaryColor':'#9289FE','primaryTextColor':'#fff','primaryBorderColor':'#9289FE','lineColor':'#5F4CFE','secondaryColor':'#AE6DFF','tertiaryColor':'#FCB1FF','edgeLabelBackground':'#5F4CFE', 'fontSize':'14px','fontFamily':'font-inter'}}}%%
flowchart TB
Client([Client])
subgraph Runpod [RUNPOD SERVERLESS]
CPU[text-api endpoint CPU load balancer GET /health GET /info POST /analyze]
GPU[gpu-sentiment endpoint GPU load balancer POST /sentiment]
end
Client -->|HTTPS + Auth| CPU
Client -->|HTTPS + Auth| GPU
style Runpod fill:#1a1a2e,stroke:#5F4CFE,stroke-width:2px
style Client fill:#4D38F5,stroke:#4D38F5,color:#fff
style CPU fill:#5F4CFE,stroke:#5F4CFE,color:#fff
style GPU fill:#22C55E,stroke:#22C55E,color:#000
```
**Key points:**
* **CPU endpoint** (`text-api`) handles three routes on one Serverless endpoint
* **GPU endpoint** (`gpu-sentiment`) handles GPU inference on a separate endpoint
* Both endpoints scale independently based on load
* All requests require authentication with your API key
## Troubleshooting
### Worker quota exceeded
**Issue**: `Max workers across all endpoints must not exceed your workers quota (30)`
**Solution**:
1. Check your current worker usage in the [Runpod console](https://console.runpod.io/serverless)
2. Reduce `workers` in your configuration:
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
api = Endpoint(
name="text-api",
cpu="cpu5c-4-8",
workers=(0, 1) # Reduce this value
)
```
3. Clean up unused endpoints before deploying new ones
### GPU endpoint timeout
**Issue**: Request times out after 60 seconds on first GPU endpoint call
**Solutions**:
1. This is normal for the first request - GPU provisioning takes time
2. Wait 1-3 minutes and try again
3. Use `GpuGroup.ANY` instead of specific GPU types for better availability
4. Consider using CPU for development testing:
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
# For testing without GPU
api = Endpoint(name="sentiment-cpu", cpu="cpu5c-4-8")
```
### Port already in use
**Issue**: `ERROR: [Errno 48] Address already in use` when running `flash dev`
**Solutions**:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
# Use a different port
flash dev --port 8889
# Or kill the process using port 8888
lsof -ti:8888 | xargs kill -9
```
### Import errors in sentiment analysis
**Issue**: `ModuleNotFoundError: No module named 'transformers'`
**Solution**: Ensure dependencies are specified on the endpoint:
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
gpu_api = Endpoint(
name="gpu-sentiment",
gpu=GpuGroup.ANY,
dependencies=["transformers", "torch"] # Must include these
)
```
For local testing, install dependencies manually:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
pip install transformers torch
```
### Endpoint stays in queue
**Issue**: GPU sentiment route stays in `IN_QUEUE` status
**Solutions**:
1. Check [GPU availability](https://console.runpod.io/serverless) in console
2. Use flexible GPU selection:
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
gpu=GpuGroup.ANY # Use any available GPU
```
3. Increase worker quota if at limit
## Next steps
Now that you've built a REST API with Flash, you can:
### Add more routes
Expand your API with additional functionality:
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
@api.post("/summarize")
async def summarize_text(text: str, max_length: int = 100) -> dict:
"""Summarize long text."""
# Summarization logic
return {"summary": text[:max_length]}
@api.post("/translate")
async def translate_text(text: str, target_lang: str) -> dict:
"""Translate text to another language."""
# Translation logic
return {"translated": text, "target": target_lang}
```
### Add authentication middleware
Implement custom authentication for your API:
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
@api.post("/protected")
async def protected_route(text: str, api_key: str) -> dict:
"""Route with custom authentication."""
if api_key != "your-secret-key":
return {"error": "Unauthorized"}, 401
return {"data": "protected content"}
```
### Monitor your API
* Track endpoint health in the [Runpod console](https://console.runpod.io/serverless)
* Monitor request counts and error rates
* Adjust `workers` based on traffic patterns
### Use multiple environments
Deploy to different environments for testing:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
flash deploy --env dev # Development
flash deploy --env staging # Staging
flash deploy --env production # Production
# If using uv:
uv run flash deploy --env dev
uv run flash deploy --env staging
uv run flash deploy --env production
```
## Related resources
* [Endpoint class guide](/flash/create-endpoints)
* [Endpoint class parameter reference](/flash/configuration/parameters)
* [Deploy Flash apps](/flash/apps/deploy-apps)
# Generate images with Flash and SDXL
Source: https://docs.runpod.io/tutorials/flash/image-generation-with-sdxl
Learn how to use Flash with Stable Diffusion XL to generate high-quality images from text prompts. Follow the implementation steps in this Runpod tutorial.
This tutorial shows you how to build an image generation script using Flash and Stable Diffusion XL (SDXL). You'll learn how to load a pretrained diffusion model on a GPU worker and generate images from text prompts.
## Requirements
* You've [created a Runpod account](/accounts-billing/manage-accounts).
* You've [created a Runpod API key](/get-started/api-keys).
* You've installed [Python 3.10, 3.11, 3.12, or 3.13](https://www.python.org/downloads/).
* You've completed the [Flash quickstart](/flash/quickstart) or are familiar with Flash basics.
## What you'll build
By the end of this tutorial, you'll have a working image generation application that:
* Accepts text prompts as input.
* Generates photorealistic images using Stable Diffusion XL.
* Runs entirely on Runpod's GPU infrastructure.
* Saves generated images to your local machine.
## Step 1: Set up your project
Create a new directory for your project and set up a Python virtual environment:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
mkdir flash-image-generation
cd flash-image-generation
```
Install Flash using [uv](https://docs.astral.sh/uv/):
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
uv venv
source .venv/bin/activate
uv pip install runpod-flash python-dotenv
```
Create a `.env` file with your Runpod API key:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
touch .env && echo "RUNPOD_API_KEY=YOUR_API_KEY" > .env
```
Replace `YOUR_API_KEY` with your actual API key from the [Runpod console](https://console.runpod.io/user/settings).
## Step 2: Understand Stable Diffusion XL
[Stable Diffusion XL (SDXL)](https://huggingface.co/stabilityai/stable-diffusion-xl-base-1.0) is a state-of-the-art text-to-image model from Stability AI. It offers:
* **High-quality images**: Generates photorealistic 1024x1024 images
* **Better prompt understanding**: Improved text comprehension compared to SD 1.5
* **Fine details**: Enhanced rendering of hands, faces, and text
* **Open source**: Available for free on Hugging Face
SDXL requires significant GPU resources:
* **Model size**: \~7GB of weights
* **VRAM requirement**: Minimum 16GB (24GB recommended)
* **Generation time**: 20-40 seconds per image on RTX 4090
We'll use the [diffusers](https://huggingface.co/docs/diffusers/index) library from Hugging Face, which provides a clean Python API for Stable Diffusion models.
## Step 3: Create your project file
Create a new file called `image_generation.py`:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
touch image_generation.py
```
Open this file in your code editor. The following steps walk through building the image generation application.
## Step 4: Add imports and configuration
Add the necessary imports and Flash configuration:
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
import asyncio
import base64
from pathlib import Path
from dotenv import load_dotenv
from runpod_flash import Endpoint, GpuGroup
# Load environment variables from .env file
load_dotenv()
```
## Step 5: Define the image generation function
Add the endpoint function that will run on the GPU worker:
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
@Endpoint(
name="image-generation",
gpu=[GpuGroup.ADA_24, GpuGroup.AMPERE_24], # 24GB GPUs
workers=2,
idle_timeout=900, # Keep workers active for 15 minutes
dependencies=["diffusers", "torch", "transformers", "accelerate"]
)
def generate_image(prompt, negative_prompt="", num_steps=30, guidance_scale=7.5):
"""Generate an image using Stable Diffusion XL."""
import torch
from diffusers import StableDiffusionXLPipeline
import base64
from io import BytesIO
# Load the SDXL model
model_id = "stabilityai/stable-diffusion-xl-base-1.0"
pipe = StableDiffusionXLPipeline.from_pretrained(
model_id,
torch_dtype=torch.float16,
use_safetensors=True,
variant="fp16"
)
# Move model to GPU
device = "cuda" if torch.cuda.is_available() else "cpu"
pipe = pipe.to(device)
# Generate image
image = pipe(
prompt=prompt,
negative_prompt=negative_prompt,
num_inference_steps=num_steps,
guidance_scale=guidance_scale,
height=1024,
width=1024
).images[0]
# Convert image to base64 for transmission
buffered = BytesIO()
image.save(buffered, format="PNG")
img_str = base64.b64encode(buffered.getvalue()).decode()
return {
"image_base64": img_str,
"prompt": prompt,
"negative_prompt": negative_prompt,
"num_steps": num_steps,
"guidance_scale": guidance_scale,
"device": device,
"resolution": "1024x1024"
}
```
**Configuration breakdown**:
* **`name="image-generation"`**: Identifies your endpoint in the Runpod console.
* **`gpu=[GpuGroup.ADA_24, GpuGroup.AMPERE_24]`**: Uses RTX 4090 or L4/A5000 GPUs (both have 24GB VRAM, sufficient for SDXL).
* **`workers=2`**: Allows up to 2 parallel workers.
* **`idle_timeout=900`**: Keeps workers active for 15 minutes (SDXL models are large, so we want longer caching).
SDXL requires at least 16GB VRAM. Using 24GB GPUs provides comfortable headroom and faster generation.
This function:
* Loads the SDXL model from Hugging Face.
* Moves the model to the GPU.
* Generates an image from the prompt.
* Encodes the image as base64.
* Returns the image as a base64 string (and other metadata).
Expand this section for a full breakdown:
**Dependencies**: The function requires four packages:
* `diffusers`: Hugging Face library for diffusion models
* `torch`: PyTorch for GPU computation
* `transformers`: Text encoder dependencies
* `accelerate`: Efficient model loading
**Model loading**:
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
pipe = StableDiffusionXLPipeline.from_pretrained(
model_id,
torch_dtype=torch.float16,
use_safetensors=True,
variant="fp16"
)
```
This downloads SDXL from Hugging Face. Key parameters:
* `torch_dtype=torch.float16`: Use half-precision (saves VRAM, faster)
* `use_safetensors=True`: Use safe tensor format
* `variant="fp16"`: Download the fp16 version (\~7GB instead of \~14GB)
**GPU acceleration**:
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
pipe = pipe.to(device)
```
Moves the entire pipeline (text encoder, UNet, VAE) to GPU.
**Image generation**:
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
image = pipe(
prompt=prompt,
negative_prompt=negative_prompt,
num_inference_steps=num_steps,
guidance_scale=guidance_scale,
height=1024,
width=1024
).images[0]
```
Parameters:
* **`prompt`**: What you want to see in the image
* **`negative_prompt`**: What you don't want (e.g., "blurry, low quality")
* **`num_inference_steps`**: More steps = better quality but slower (20-50 typical)
* **`guidance_scale`**: How closely to follow the prompt (7-10 recommended)
* **`height/width`**: SDXL is trained for 1024x1024
**Image encoding**:
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
buffered = BytesIO()
image.save(buffered, format="PNG")
img_str = base64.b64encode(buffered.getvalue()).decode()
```
We encode the image as base64 to return it through Flash. This allows us to transmit the image data as a string.
## Step 6: Add the main function and image saving
Create functions to call the generator and save images:
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
def save_image(base64_string, filename):
"""Save a base64-encoded image to disk."""
import base64
from PIL import Image
from io import BytesIO
# Decode base64 string
img_data = base64.b64decode(base64_string)
# Open and save image
image = Image.open(BytesIO(img_data))
image.save(filename)
print(f"✓ Image saved to {filename}")
async def main():
print("Generating image with Stable Diffusion XL on Runpod GPU...")
print("This may take 1-2 minutes on first run (downloading model)...\n")
# Define your prompt
prompt = "A serene landscape with mountains, a lake, and sunset, highly detailed, photorealistic"
negative_prompt = "blurry, low quality, distorted, ugly"
# Generate image
result = await generate_image(
prompt=prompt,
negative_prompt=negative_prompt,
num_steps=30,
guidance_scale=7.5
)
# Save the generated image
output_dir = Path("generated_images")
output_dir.mkdir(exist_ok=True)
filename = output_dir / "sdxl_output.png"
save_image(result["image_base64"], filename)
# Display metadata
print(f"\n{'='*60}")
print("GENERATION DETAILS")
print('='*60)
print(f"Prompt: {result['prompt']}")
print(f"Negative prompt: {result['negative_prompt']}")
print(f"Steps: {result['num_steps']}")
print(f"Guidance scale: {result['guidance_scale']}")
print(f"Resolution: {result['resolution']}")
print(f"Device: {result['device']}")
print('='*60)
if __name__ == "__main__":
asyncio.run(main())
```
This main function:
* Calls the remote function with `await`.
* Creates a `generated_images` directory if it doesn't exist.
* Decodes and saves the base64 image to disk.
* Displays generation metadata.
## Step 7: Run your first generation
Run the application:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
python image_generation.py
```
**First run output** (takes 2-3 minutes):
```text theme={"theme":{"light":"github-light","dark":"github-dark"}}
Generating image with Stable Diffusion XL on Runpod GPU...
This may take 1-2 minutes on first run (downloading model)...
Creating endpoint: server_Endpoint_a1b2c3d4
Provisioning Serverless endpoint...
Endpoint ready
Executing function on Runpod endpoint ID: xvf32dan8rcilp
Initial job status: IN_QUEUE
Downloading model weights from Hugging Face...
Model loaded, generating image...
Job completed, output received
✓ Image saved to generated_images/sdxl_output.png
============================================================
GENERATION DETAILS
============================================================
Prompt: A serene landscape with mountains, a lake, and sunset, highly detailed, photorealistic
Negative prompt: blurry, low quality, distorted, ugly
Steps: 30
Guidance scale: 7.5
Resolution: 1024x1024
Device: cuda
============================================================
```
**Subsequent runs** (takes 30-40 seconds):
```text theme={"theme":{"light":"github-light","dark":"github-dark"}}
Generating image with Stable Diffusion XL on Runpod GPU...
Resource Endpoint_a1b2c3d4 already exists, reusing.
Executing function on Runpod endpoint ID: xvf32dan8rcilp
Initial job status: IN_QUEUE
Job completed, output received
✓ Image saved to generated_images/sdxl_output.png
[Results appear]
```
Open `generated_images/sdxl_output.png` to see your generated image!
The first run downloads \~7GB of model weights, which takes 1-2 minutes. Subsequent runs reuse the cached model and complete in 30-40 seconds.
## Step 8: Experiment with different prompts
Try various prompts to see SDXL's capabilities:
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
async def main():
# Create output directory
output_dir = Path("generated_images")
output_dir.mkdir(exist_ok=True)
# Try different prompts
prompts = [
{
"prompt": "A cyberpunk city at night with neon lights, flying cars, rain, cinematic",
"negative": "blurry, low quality",
"filename": "cyberpunk_city.png"
},
{
"prompt": "A cute corgi puppy wearing a space suit, floating in space, highly detailed",
"negative": "distorted, ugly, bad anatomy",
"filename": "space_corgi.png"
},
{
"prompt": "An ancient wizard's study filled with books, potions, magical artifacts, candlelight",
"negative": "blurry, modern, plastic",
"filename": "wizard_study.png"
}
]
for i, p in enumerate(prompts, 1):
print(f"\n{'='*60}")
print(f"Generating image {i}/{len(prompts)}")
print(f"Prompt: {p['prompt'][:50]}...")
print('='*60)
result = await generate_image(
prompt=p['prompt'],
negative_prompt=p['negative'],
num_steps=30,
guidance_scale=7.5
)
filename = output_dir / p['filename']
save_image(result["image_base64"], filename)
print(f"✓ Saved to {filename}\n")
if __name__ == "__main__":
asyncio.run(main())
```
Run it:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
python image_generation.py
```
You'll see three different images generated sequentially on the same GPU worker. Each generation takes about 30-40 seconds after the first one.
## Understanding generation parameters
Let's explore how different parameters affect image quality:
### Number of inference steps
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
# Fast but lower quality (15-20 steps)
result = await generate_image(prompt, num_steps=20)
# Balanced (30 steps) - recommended
result = await generate_image(prompt, num_steps=30)
# High quality but slower (50 steps)
result = await generate_image(prompt, num_steps=50)
```
**Effects**:
* **15-20 steps**: Faster (15-20 seconds) but less refined details.
* **30 steps**: Good balance of quality and speed (30-40 seconds) - **recommended**.
* **50+ steps**: Diminishing returns, minimal quality improvement.
### Guidance scale
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
# Low guidance - more creative, less faithful to prompt
result = await generate_image(prompt, guidance_scale=5.0)
# Medium guidance - balanced (recommended)
result = await generate_image(prompt, guidance_scale=7.5)
# High guidance - very faithful to prompt, may oversaturate
result = await generate_image(prompt, guidance_scale=12.0)
```
**Effects**:
* **3-5**: More artistic freedom, less literal interpretation.
* **7-10**: Balanced, follows prompt closely - **recommended**.
* **12+**: Very literal, may produce oversaturated or exaggerated images.
### Negative prompts
Negative prompts tell the model what to avoid:
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
# Good negative prompts for photorealistic images
negative_prompt = "blurry, low quality, distorted, ugly, bad anatomy, watermark"
# Good negative prompts for artistic images
negative_prompt = "realistic, photograph, blurry, low quality"
# Good negative prompts for portraits
negative_prompt = "distorted face, bad anatomy, extra limbs, low quality"
```
Use negative prompts to:
* Remove common artifacts ("distorted", "low quality").
* Avoid unwanted styles ("cartoon", "3D render").
* Fix common issues ("bad anatomy", "extra fingers").
## Troubleshooting
### Out of memory error
**Issue**: `RuntimeError: CUDA out of memory`.
**Cause**: SDXL requires significant VRAM (16GB minimum).
**Solutions**:
1. Verify you're using 24GB GPUs:
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
gpu=[GpuGroup.ADA_24, GpuGroup.AMPERE_24] # 24GB GPUs
```
2. Use half-precision (already in the example):
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
torch_dtype=torch.float16 # Half precision
```
3. If still failing, use 48GB GPUs:
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
gpu=GpuGroup.AMPERE_48 # A40/A6000 with 48GB
```
### Model download fails
**Issue**: `Error: Failed to download model from Hugging Face`.
**Solutions**:
1. Increase execution timeout for first run:
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
@Endpoint(
name="image-generation",
gpu=GpuGroup.ADA_24,
execution_timeout_ms=600000 # 10 minutes for first download
)
```
2. Check Hugging Face Hub status at [status.huggingface.co](https://status.huggingface.co).
3. Try a smaller model first to test connectivity:
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
model_id = "runwayml/stable-diffusion-v1-5" # Smaller SD 1.5
```
### Image quality is poor
**Issue**: Generated images look blurry or low quality.
**Solutions**:
1. Increase inference steps:
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
num_steps=40 # More steps = better quality
```
2. Adjust guidance scale:
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
guidance_scale=8.5 # Higher guidance
```
3. Improve your prompt:
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
prompt = "A detailed portrait, highly detailed, sharp focus, 8k, professional photography"
```
4. Add quality keywords to your prompt:
* "highly detailed"
* "sharp focus"
* "8k"
* "photorealistic"
* "professional"
### Slow generation
**Issue**: Image generation takes >60 seconds per image.
**Possible causes**:
1. Worker scaled down (cold start).
2. Model not cached.
3. Too many inference steps.
**Solutions**:
1. Increase `idle_timeout` to keep workers active:
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
idle_timeout=1800 # Keep active for 30 minutes
```
2. Reduce inference steps:
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
num_steps=20 # Faster but slightly lower quality
```
3. Set `workers=(1, 2)` to always have a warm worker ready.
### Images look distorted or have artifacts
**Issue**: Generated images have weird artifacts or distortions.
**Solutions**:
1. Use negative prompts:
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
negative_prompt="distorted, ugly, bad anatomy, extra limbs, disfigured"
```
2. Adjust guidance scale (try 7-9 range):
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
guidance_scale=8.0
```
3. Increase inference steps for better refinement:
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
num_steps=35
```
## Next steps
Now that you've built an image generation script with Flash, you can:
### Try other Stable Diffusion models
Explore different models from Hugging Face:
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
# SDXL Turbo - 4x faster, 1 step generation
model_id = "stabilityai/sdxl-turbo"
# Stable Diffusion 1.5 - smaller, faster
model_id = "runwayml/stable-diffusion-v1-5"
# Stable Diffusion 2.1 - better at artistic styles
model_id = "stabilityai/stable-diffusion-2-1"
```
### Add image-to-image generation
Use an existing image as a starting point:
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
from diffusers import StableDiffusionXLImg2ImgPipeline
# Load img2img pipeline
pipe = StableDiffusionXLImg2ImgPipeline.from_pretrained(...)
# Generate variations of an existing image
image = pipe(prompt, image=init_image, strength=0.75).images[0]
```
### Build a Flash app
Convert your script to a production [Flash app](/flash/apps/overview):
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
flash init image-generation-app
# Move your function to workers/gpu/endpoint.py
# Add FastAPI routes for HTTP API
flash deploy
# If using uv:
uv run flash init image-generation-app
uv run flash deploy
```
### Optimize with network volumes
Use [network volumes](/flash/configuration/storage) to cache models across workers:
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
from runpod_flash import Endpoint, GpuGroup, NetworkVolume
vol = NetworkVolume(name="model-cache") # Finds existing or creates new
@Endpoint(
name="image-generation",
gpu=GpuGroup.ADA_24,
volume=vol,
dependencies=["diffusers", "torch", "transformers", "accelerate"]
)
def generate_image(prompt, ...):
# Models at /runpod-volume/ persist across workers
...
```
### Explore advanced features
* **LoRA fine-tuning**: Customize SDXL for specific styles.
* **ControlNet**: Guide generation with edge maps, depth, or pose.
* **Inpainting**: Edit specific parts of images.
* **Upscaling**: Generate higher resolution images.
## Related resources
* [Endpoint functions guide](/flash/create-endpoints).
* [Configuration reference](/flash/configuration/parameters).
* [Stable Diffusion XL model card](https://huggingface.co/stabilityai/stable-diffusion-xl-base-1.0).
# Generate text with Flash and transformers
Source: https://docs.runpod.io/tutorials/flash/text-generation-with-transformers
Learn how to use Flash with Hugging Face transformers to build a GPU-accelerated text generation application.
This tutorial shows you how to build a text generation script using Flash and Hugging Face's transformers library. You'll learn how to load a pretrained language model on a GPU worker and generate text from prompts.
## Requirements
* You've [created a Runpod account](/accounts-billing/manage-accounts).
* You've [created a Runpod API key](/get-started/api-keys).
* You've installed [Python 3.10, 3.11, 3.12, or 3.13](https://www.python.org/downloads/).
* You've completed the [Flash quickstart](/flash/quickstart) or are familiar with Flash basics.
## What you'll build
By the end of this tutorial, you'll have a working text generation application that:
* Accepts text prompts as input.
* Generates natural language completions using GPT-2.
* Runs entirely on Runpod's GPU infrastructure.
* Returns generated text with execution metadata.
## Step 1: Set up your project
Create a new directory for your project and set up a Python virtual environment:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
mkdir flash-text-generation
cd flash-text-generation
```
Install Flash using [uv](https://docs.astral.sh/uv/):
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
uv venv
source .venv/bin/activate
uv pip install runpod-flash python-dotenv
```
Create a `.env` file with your Runpod API key:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
touch .env && echo "RUNPOD_API_KEY=YOUR_API_KEY" > .env
```
Replace `YOUR_API_KEY` with your actual API key from the [Runpod console](https://console.runpod.io/user/settings).
## Step 2: Understand the Hugging Face transformers library
[Hugging Face transformers](https://huggingface.co/docs/transformers/index) is a popular Python library for working with pretrained language models. It provides:
* **Thousands of pretrained models**: GPT-2, GPT-3, BERT, T5, LLaMA, and more
* **Unified API**: Same code works across different model architectures
* **Model hub integration**: Download models directly from [Hugging Face Hub](https://huggingface.co/models)
* **Production-ready**: Used by companies and researchers worldwide
For this tutorial, we'll use **GPT-2**, a 124M parameter language model from OpenAI. It's small enough to load quickly but powerful enough to generate coherent text.
## Step 3: Create your project file
Create a new file called `text_generation.py`:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
touch text_generation.py
```
Open this file in your code editor. The following steps walk through building the text generation application.
## Step 4: Add imports and configuration
Add the necessary imports and Flash configuration:
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
import asyncio
from dotenv import load_dotenv
from runpod_flash import Endpoint, GpuGroup
# Load environment variables from .env file
load_dotenv()
```
## Step 5: Define the text generation function
Add the endpoint function that will run on the GPU worker:
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
@Endpoint(
name="text-generation",
gpu=[GpuGroup.AMPERE_24, GpuGroup.ADA_24], # 24GB GPUs
workers=3,
idle_timeout=600, # 10 minutes
dependencies=["transformers", "torch", "accelerate"]
)
def generate_text(prompt, max_length=50):
"""Generate text using a pretrained language model."""
import torch
from transformers import AutoTokenizer, AutoModelForCausalLM
# Load the GPT-2 model and tokenizer
model_name = "gpt2"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(model_name)
# Move model to GPU if available
device = "cuda" if torch.cuda.is_available() else "cpu"
device_name = torch.cuda.get_device_name(0) if torch.cuda.is_available() else "CPU"
model = model.to(device)
# Tokenize the input prompt
inputs = tokenizer(prompt, return_tensors="pt").to(device)
# Generate text
with torch.no_grad():
outputs = model.generate(
**inputs,
max_length=max_length,
num_return_sequences=1,
temperature=0.7,
do_sample=True,
pad_token_id=tokenizer.eos_token_id
)
# Decode the generated tokens back to text
generated_text = tokenizer.decode(outputs[0], skip_special_tokens=True)
return {
"prompt": prompt,
"generated_text": generated_text,
"model_name": model_name,
"device": device,
"device_name": device_name,
"max_length": max_length
}
```
**Configuration breakdown**:
* **`name="text-generation"`**: Identifies your endpoint in the Runpod console
* **`gpu=[GpuGroup.AMPERE_24, GpuGroup.ADA_24]`**: Allows workers to use L4, A5000, RTX 3090, or RTX 4090 GPUs (all have 24GB VRAM)
* **`workers=3`**: Allows up to 3 parallel workers for concurrent requests
* **`idle_timeout=600`**: Keeps workers active for 10 minutes after last use (reduces cold starts)
GPT-2 only requires about 2GB of VRAM, so 24GB GPUs are more than sufficient. For larger models like LLaMA or GPT-J, you might need 48GB or 80GB GPUs.
This function:
* Loads the GPT-2 model from Hugging Face.
* Moves the model to the GPU.
* Tokenizes the input prompt.
* Generates text from the prompt.
* Decodes the generated tokens back to text.
* Returns the generated text and other metadata.
Expand this section for a full breakdown:
**Dependencies**: The function requires three packages:
* `transformers`: Hugging Face library for language models
* `torch`: PyTorch for GPU computation
* `accelerate`: Helper library for loading large models efficiently
**Model loading**:
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(model_name)
```
These lines download and load the GPT-2 model from Hugging Face Hub. The first time this runs, it downloads \~500MB of model weights. Subsequent runs use the cached version.
**GPU acceleration**:
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
device = "cuda" if torch.cuda.is_available() else "cpu"
model = model.to(device)
```
This moves the model to GPU for faster inference. On Runpod workers, `torch.cuda.is_available()` returns `True`.
**Tokenization**:
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
inputs = tokenizer(prompt, return_tensors="pt").to(device)
```
Converts your text prompt into token IDs that the model understands. The `.to(device)` moves these tokens to GPU memory.
**Generation parameters**:
* `max_length=50`: Maximum number of tokens to generate
* `temperature=0.7`: Controls randomness (0.0 = deterministic, 1.0+ = very random)
* `do_sample=True`: Use sampling instead of greedy decoding for more diverse outputs
* `num_return_sequences=1`: Generate one completion per prompt
**No gradient tracking**:
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
with torch.no_grad():
```
Disables gradient computation, reducing memory usage and speeding up inference.
## Step 6: Add the main function
Create the main function to test your text generator:
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
async def main():
print("Starting text generation on Runpod GPU...")
# Define a prompt
prompt = "The future of artificial intelligence is"
# Generate text
result = await generate_text(prompt, max_length=100)
# Display results
print("\n" + "="*60)
print("TEXT GENERATION RESULTS")
print("="*60)
print(f"\nPrompt: {result['prompt']}")
print(f"\nGenerated text:\n{result['generated_text']}")
print("\n" + "-"*60)
print(f"Model: {result['model_name']}")
print(f"Device: {result['device']}")
print(f"GPU: {result['device_name']}")
print(f"Max length: {result['max_length']} tokens")
print("="*60)
if __name__ == "__main__":
asyncio.run(main())
```
This main function:
* Calls the remote function with `await` (runs asynchronously).
* Waits for the GPU worker to complete text generation.
* Displays the results in a formatted output.
## Step 7: Run your first generation
Run the application:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
python text_generation.py
```
**First run output** (takes 60-90 seconds):
```text theme={"theme":{"light":"github-light","dark":"github-dark"}}
Starting text generation on Runpod GPU...
Creating endpoint: server_Endpoint_a1b2c3d4
Provisioning Serverless endpoint...
Endpoint ready
Registering Runpod endpoint at https://api.runpod.ai/xvf32dan8rcilp
Executing function on Runpod endpoint ID: xvf32dan8rcilp
Initial job status: IN_QUEUE
Installing dependencies: transformers torch accelerate
Downloading model weights...
Job completed, output received
============================================================
TEXT GENERATION RESULTS
============================================================
Prompt: The future of artificial intelligence is
Generated text:
The future of artificial intelligence is bright and full of possibilities. With advancements in machine learning and deep learning, we're seeing AI systems that can understand natural language, recognize images, and even create art. The potential applications are endless, from healthcare to transportation to education.
------------------------------------------------------------
Model: gpt2
Device: cuda
GPU: NVIDIA GeForce RTX 4090
Max length: 100 tokens
============================================================
```
**Subsequent runs** (takes 2-5 seconds):
```text theme={"theme":{"light":"github-light","dark":"github-dark"}}
Starting text generation on Runpod GPU...
Resource Endpoint_a1b2c3d4 already exists, reusing.
Registering Runpod endpoint at https://api.runpod.ai/xvf32dan8rcilp
Executing function on Runpod endpoint ID: xvf32dan8rcilp
Initial job status: IN_QUEUE
Job completed, output received
[Results appear immediately]
```
Notice the dramatic speed improvement on subsequent runs—the endpoint is already provisioned, dependencies are installed, and the model is cached.
## Step 8: Experiment with different prompts
Modify the main function to try different prompts:
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
async def main():
print("Starting text generation on Runpod GPU...")
# Try multiple prompts
prompts = [
"Once upon a time in a distant galaxy",
"The secret to happiness is",
"In the year 2050, technology will"
]
for prompt in prompts:
print(f"\n{'='*60}")
print(f"Generating for: {prompt}")
print('='*60)
result = await generate_text(prompt, max_length=80)
print(f"\n{result['generated_text']}\n")
if __name__ == "__main__":
asyncio.run(main())
```
Run it again:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
python text_generation.py
```
You'll see three different completions generated sequentially on the same GPU worker.
## Troubleshooting
### Model download fails
**Issue**: `Error: Failed to download model from Hugging Face`.
**Solutions**:
1. Check internet connectivity from workers (rare issue on Runpod).
2. Try a different model that might be available faster.
3. Increase execution timeout in configuration:
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
@Endpoint(
name="text-generation",
gpu=GpuGroup.ADA_24,
execution_timeout_ms=300000 # 5 minutes
)
```
### Out of memory error
**Issue**: `RuntimeError: CUDA out of memory`.
**Solutions**:
1. Use smaller models (GPT-2 instead of GPT-2 Large).
2. Reduce `max_length` parameter.
3. Use larger GPUs:
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
gpu=GpuGroup.AMPERE_48 # 48GB GPUs
```
### Slow generation
**Issue**: Text generation takes >30 seconds per request.
**Possible causes**:
1. Worker scaled down (cold start).
2. Model not cached.
3. Large `max_length` value.
**Solutions**:
1. Increase `idle_timeout` to keep workers active:
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
idle_timeout=1800 # Keep active for 30 minutes
```
2. Set `workers=(1, 3)` to always have a warm worker ready.
3. Reduce `max_length` to generate fewer tokens.
### Generation quality is poor
**Issue**: Generated text is incoherent or repetitive.
**Solutions**:
1. Adjust `temperature` (try 0.7-0.9)
2. Add `top_p` and `top_k` sampling:
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
outputs = model.generate(
**inputs,
max_length=max_length,
temperature=0.8,
top_p=0.9,
top_k=50,
do_sample=True
)
```
3. Try a larger model (GPT-2 Medium or Large).
## Next steps
Now that you've built a text generation script with Flash, you can:
### Explore other models
Try different models from Hugging Face:
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
# Instruction-following model
model_name = "facebook/opt-1.3b"
# Code generation model
model_name = "Salesforce/codegen-350M-mono"
# Dialogue model
model_name = "microsoft/DialoGPT-medium"
```
### Build a chat interface
Extend your app to handle multi-turn conversations:
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
@Endpoint(
name="chat",
gpu=GpuGroup.ADA_24,
dependencies=["transformers", "torch"]
)
def chat(conversation_history):
"""Multi-turn chat with context."""
# Concatenate conversation history
prompt = "\n".join(conversation_history)
# Generate response
# Return new message
```
### Deploy as a Flash app
Convert your script to a production [Flash app](/flash/apps/overview):
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
flash init text-generation-app
# Move your function to workers/gpu/endpoint.py
# Add FastAPI routes
flash deploy
# If using uv:
uv run flash init text-generation-app
uv run flash deploy
```
When deploying queue-based functions with `flash deploy`, each function must have its own unique endpoint configuration. If your script has multiple functions sharing the same config (like `generate_text` and `chat` in this tutorial), create separate endpoints for each function when converting to a Flash app. See [understanding endpoint architecture](/flash/apps/deploy-apps#understanding-endpoint-architecture) for details.
### Optimize performance
* Use [network volumes](/flash/configuration/storage) to cache models across workers.
* Implement [request batching](/flash/create-endpoints#parallel-execution) for higher throughput.
* Try [quantized models](https://huggingface.co/docs/transformers/main_classes/quantization) for faster inference.
## Related resources
* [Endpoint functions guide](/flash/create-endpoints).
* [Configuration reference](/flash/configuration/parameters).
* [Hugging Face transformers documentation](https://huggingface.co/docs/transformers/index).
* [Hugging Face model hub](https://huggingface.co/models).
# Introduction to containers
Source: https://docs.runpod.io/tutorials/introduction/containers
Learn about Docker containers, images, and how they enable portable application deployment on Runpod.
Containers are the foundation of modern cloud computing, enabling you to package applications with all their dependencies and run them consistently across different environments. This tutorial series teaches you the container fundamentals you need to work effectively with Runpod's [Serverless](/serverless/overview) and [Pods](/pods/overview) platforms.
## Requirements
To follow this tutorial series, you need:
* Docker Desktop installed on your system.
* Basic command-line familiarity.
* A text editor for creating Dockerfiles and scripts.
If you haven't installed Docker yet, follow Docker's [official installation guide](https://docs.docker.com/get-started/get-docker/) for your operating system.
## What are containers?
A container is an isolated environment for your code. Containers package applications with everything they need to run, including the code, runtime, system tools, libraries, and settings. Unlike virtual machines, containers share the host operating system's kernel, making them lightweight and fast to start.
When you run a container, it has no knowledge of your operating system or your files. It runs in a self-contained environment with only the resources and files you explicitly provide. This isolation ensures that applications behave consistently regardless of where they run—whether on your laptop, a teammate's computer, or in the cloud.
### Why use containers?
Containers solve the classic "it works on my machine" problem by ensuring applications run identically across all environments. This consistency makes containers essential for modern software development and deployment.
Key benefits include:
* **Portability**: Run containers anywhere Docker runs—on-premises, in the cloud, or in hybrid environments.
* **Consistency**: Applications behave the same way in development, testing, and production.
* **Fast cold starts**: Containers start in seconds, making them ideal for serverless computing.
* **Dependency isolation**: Each container includes its own dependencies, preventing conflicts between applications.
* **Resource efficiency**: Containers share the host OS kernel, using fewer resources than virtual machines.
### Containers and Runpod
Runpod uses containers extensively across its platform:
* **Serverless workers**: When you [deploy a Serverless endpoint](/serverless/quickstart), you provide a container image that defines how your [worker processes requests](/serverless/workers/overview). Your handler code runs inside the container, and Runpod automatically scales workers up and down based on demand.
* **Pods**: With [Pods](/pods/overview), you can bring your own container (BYOC) to [run long-running GPU workloads](/pods/manage-pods) like training, inference servers, or development environments. [Choose from pre-built templates](/pods/choose-a-pod) or deploy custom containers.
* **Templates**: Runpod's [templates](/pods/templates/overview) are pre-configured container images optimized for specific tasks. You can [create custom templates](/pods/templates/create-custom-template) to standardize your container configurations across deployments.
Understanding containers is essential for leveraging Runpod's full capabilities.
## What are images?
Docker images are read-only templates used to create containers. Think of an image as a snapshot that includes your application code, runtime environment, libraries, and all dependencies needed to run your application.
Images are built using a Dockerfile, which contains a series of instructions for assembling the image. Once built, images can be stored in registries like Docker Hub or private registries, making them easy to share and deploy.
### How images and containers relate
The relationship between images and containers is similar to the relationship between a class and an instance in programming:
* **Image**: A template or blueprint (like a class definition).
* **Container**: A running instance created from an image (like an object).
You can create multiple containers from a single image, and each container runs independently with its own state and data.
### Why use images?
Docker images provide several advantages for development and deployment:
* **Reproducibility**: Images ensure your application runs the same way every time.
* **Version control**: You can tag images with version numbers and roll back to previous versions if needed.
* **Easy distribution**: Store images in registries and pull them onto any system with Docker installed.
* **Efficient storage**: Images use layers that can be shared between different images, reducing storage requirements.
### When to use custom images
While you can use pre-built images from Docker Hub, you'll often need custom images that:
* Include your specific application code.
* Install custom dependencies or libraries.
* Configure [environment variables](/serverless/development/environment-variables) or settings.
* Optimize for your specific use case.
For Runpod Serverless deployments, you'll create custom images that include your [handler functions](/serverless/workers/handler-functions) and model dependencies. For Pods, you might use existing [templates](/pods/templates/manage-templates) or create custom ones with your specific tools and frameworks installed.
## What is Docker Hub?
[Docker Hub](https://hub.docker.com/) is the default public registry for Docker images. It hosts millions of pre-built images for popular software, frameworks, and operating systems. You can pull images from Docker Hub, use them as base images for your own containers, or push your custom images to share with others.
When you run a command like `docker run nginx`, Docker automatically pulls the `nginx` image from Docker Hub if it's not already available locally.
For production deployments on Runpod, you'll typically push your custom images to Docker Hub or a private registry, then configure your endpoint or Pod to use that image.
## Tutorial series
This tutorial series guides you through container fundamentals in a hands-on way:
Learn how to write Dockerfiles, build custom images, and run your first container. Start with Docker installation and work up to creating executable containers.
Master essential Docker CLI commands for building, running, managing, and debugging containers. Includes Runpod-specific guidance for deployment.
Understand how to use Docker volumes to persist data outside of containers, essential for machine learning training and data processing workflows.
## Next steps
Ready to get hands-on with Docker? Start with [creating your first Dockerfile](/tutorials/introduction/containers/create-dockerfiles).
For more in-depth container concepts, see Docker's [container concepts documentation](https://docs.docker.com/get-started/docker-concepts/the-basics/what-is-a-container/).
When you're ready to deploy containers on Runpod:
**For Serverless:**
* Learn about [Serverless workers](/serverless/workers/overview) for scalable, GPU-powered inference.
* Review [creating Dockerfiles for Serverless](/serverless/workers/create-dockerfile) with Runpod-specific best practices.
* Understand [endpoint configurations](/serverless/endpoints/endpoint-configurations) for optimizing performance and cost.
* Explore [deploying your first endpoint](/serverless/quickstart) to get started quickly.
**For Pods:**
* Explore [Pods](/pods/overview) for long-running GPU workloads and development environments.
* Learn how to [choose the right Pod](/pods/choose-a-pod) for your workload.
* Understand [connecting to Pods](/pods/connect-to-a-pod) via SSH, web terminal, or IDE.
* Review [Pod storage options](/pods/storage/types) for persisting data.
# Create Dockerfiles
Source: https://docs.runpod.io/tutorials/introduction/containers/create-dockerfiles
Learn how to write Dockerfiles, build custom images, and run your first containers. Follow the implementation steps in this Runpod tutorial.
A Dockerfile is a text file containing instructions for building a Docker image. By creating a Dockerfile, you can package your application with its dependencies and configuration, making it easy to deploy anywhere Docker runs. This guide walks you through creating your first Dockerfile, building an image, and running a container.
## Requirements
Before starting, you need:
* Docker Desktop installed and running (see the [overview](/tutorials/introduction/containers) for installation instructions).
* Basic command-line familiarity.
* A text editor for creating files.
## Step 1: Verify Docker installation
First, verify that Docker is installed correctly by checking the version:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
docker version
```
You should see output showing both the client and server versions. If you see an error about the Docker daemon not running, make sure Docker Desktop is started.
To test that Docker can pull and run images, run a simple command using the `busybox` image:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
docker run busybox echo "Hello from Docker!"
```
This command downloads the lightweight `busybox` image (if not already present), starts a container from it, runs the `echo` command inside the container, and then exits. You should see "Hello from Docker!" printed to your terminal.
Breaking down what happened:
* `docker run`: Creates and starts a new container.
* `busybox`: The image to use (automatically pulled from Docker Hub if needed).
* `echo "Hello from Docker!"`: The command to run inside the container.
## Step 2: Create a project directory
Create a new directory for this tutorial and navigate into it:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
mkdir my-first-container
cd my-first-container
```
This keeps your Dockerfile and related files organized in one place.
## Step 3: Write a Dockerfile
Create a file named `Dockerfile` (no file extension) with the following content:
```dockerfile theme={"theme":{"light":"github-light","dark":"github-dark"}}
FROM busybox
COPY entrypoint.sh /
RUN chmod +x /entrypoint.sh
ENTRYPOINT ["/entrypoint.sh"]
```
Let's understand each instruction:
**FROM busybox**: This specifies the base image for your container. `busybox` is a minimal Linux image with basic utilities. Every Dockerfile must start with a `FROM` instruction. For real applications, you might use images like `python:3.11`, `node:20`, or `nvidia/cuda:12.0.0-runtime-ubuntu22.04`.
**COPY entrypoint.sh /**: This copies the `entrypoint.sh` file from your local directory into the root directory of the container's filesystem. The `COPY` instruction is how you add your application code and files to the image.
**RUN chmod +x /entrypoint.sh**: This executes a command during the image build process to make the script executable. `RUN` instructions execute commands and save the results as a new layer in the image.
**ENTRYPOINT \["/entrypoint.sh"]**: This specifies the command that runs when a container starts from this image. Using the JSON array syntax (with brackets and quotes) is recommended as it prevents shell processing quirks.
## Step 4: Create the entrypoint script
Create a file named `entrypoint.sh` in the same directory:
```sh theme={"theme":{"light":"github-light","dark":"github-dark"}}
#!/bin/sh
echo "Container started at: $(date)"
echo "Running on: $(uname -a)"
```
This simple script prints the current date/time and system information when the container starts.
### Understanding entrypoint scripts
An entrypoint script is the command that runs when your container starts. Think of it as the "main" function of your container. Common uses include:
* **Starting applications**: Launch a web server, API, or background process.
* **Setup tasks**: Initialize databases, check configurations, or [set environment variables](/pods/templates/environment-variables).
* **Processing workflows**: Run data processing pipelines or batch jobs.
For Runpod Serverless workers, your entrypoint typically starts a Python script that imports the `runpod` library and defines your [handler function](/serverless/workers/handler-functions). For example, you might run `python handler.py` which calls `runpod.serverless.start()`. For Pods, the entrypoint might start [JupyterLab](/tutorials/pods/run-your-first), a training script, or a development environment like VS Code.
While we named this script `entrypoint.sh`, you'll see various naming conventions in Docker projects:
* `start.sh`
* `docker-entrypoint.sh`
* `run.sh`
* `cmd.sh`
These scripts are often placed in a `/scripts` or `/app` directory, depending on the project structure.
## Step 5: Build the image
Now build a Docker image from your Dockerfile:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
docker build -t my-time-image .
```
Breaking down this command:
* `docker build`: Initiates the image build process.
* `-t my-time-image`: Tags the image with a name for easy reference. Without a tag, you'd have to use the image ID.
* `.`: Specifies the build context (current directory). Docker looks for a Dockerfile here and can access files in this directory.
You'll see output showing each Dockerfile instruction being executed. Docker builds images in layers, and each instruction creates a new layer. These layers are cached, so rebuilding after small changes is fast.
### Why build custom images?
Custom images let you:
* **Package dependencies**: Install specific libraries, frameworks, or tools your application needs.
* **Configure environments**: Set [environment variables](/serverless/development/environment-variables), create directories, or configure settings.
* **Include application code**: Bundle your code so it's ready to run anywhere.
* **Version applications**: Tag images with version numbers to track changes over time.
* **Ensure consistency**: Eliminate "works on my machine" problems by standardizing the environment.
For Runpod deployments, custom images are essential:
* **Serverless workers** need images with your [handler code](/serverless/workers/handler-functions), inference libraries, and optionally [cached models](/serverless/endpoints/model-caching).
* **Pods** might need images with specific ML frameworks, CUDA versions, development tools, or [custom configurations saved as templates](/pods/templates/create-custom-template).
## Step 6: Run the container
Run a container from your newly built image:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
docker run my-time-image
```
You should see output showing the container start time and system information, confirming that your entrypoint script ran successfully.
The container executes the script and then exits. This is normal behavior for containers that complete their task. In production, containers often run continuously (like web servers) or process tasks and exit (like batch jobs or Serverless functions).
## Step 7: Experiment with your container
Try a few variations to understand how containers work:
**Run the container multiple times** to see different timestamps:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
docker run my-time-image
docker run my-time-image
```
Each invocation creates a new container instance with a fresh environment.
**View running containers** (in another terminal, run a long-running command):
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
docker run busybox sleep 30
```
Then in another terminal:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
docker ps
```
This shows currently running containers with their IDs, names, and status.
**See all containers**, including stopped ones:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
docker ps -a
```
You'll see all the containers you've created, even those that have exited.
## Understanding Dockerfile best practices
As you create more complex Dockerfiles, keep these practices in mind:
**Use specific base image tags**: Instead of `FROM python:3`, use `FROM python:3.11-slim` to ensure consistent builds.
**Minimize layers**: Combine related `RUN` commands with `&&` to reduce image size:
```dockerfile theme={"theme":{"light":"github-light","dark":"github-dark"}}
RUN apt-get update && \
apt-get install -y package1 package2 && \
apt-get clean
```
**Order instructions by change frequency**: Put instructions that change rarely (like installing system packages) before instructions that change often (like copying application code). This maximizes layer caching.
**Clean up in the same layer**: Remove temporary files in the same `RUN` command that creates them:
```dockerfile theme={"theme":{"light":"github-light","dark":"github-dark"}}
RUN wget https://example.com/file.tar.gz && \
tar -xzf file.tar.gz && \
rm file.tar.gz
```
For more best practices, see Docker's [Dockerfile reference documentation](https://docs.docker.com/reference/dockerfile/).
## Building for Runpod
When building images for Runpod, keep these platform-specific considerations in mind:
**Use the correct architecture**: Runpod's infrastructure uses `linux/amd64` architecture. If you're building on an Apple Silicon Mac (ARM64), specify the platform:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
docker build --platform=linux/amd64 -t my-image .
```
**Optimize for cold starts**: Smaller images start faster, reducing [cold start times](/serverless/overview#cold-starts) for Serverless workers. Use minimal base images like `-slim` or `-alpine` variants when possible.
**Include model caching**: For ML models, consider using Runpod's [model caching feature](/serverless/endpoints/model-caching) instead of baking large models into your image. This dramatically reduces cold starts and deployment costs.
**Configure GPU access**: For GPU workloads, ensure your base image includes the correct [CUDA version for your framework](/pods/choose-a-pod#gpu-compatibility).
For detailed guidance on creating Dockerfiles for Serverless workers, see [creating Dockerfiles for Serverless](/serverless/workers/create-dockerfile).
## Next steps
Now that you can create Dockerfiles and build images, continue learning:
**Continue the tutorial series:**
* [Master Docker commands](/tutorials/introduction/containers/docker-commands) for building, running, and managing containers.
* [Learn about data persistence](/tutorials/introduction/containers/persist-data) with Docker volumes.
**Deploy on Runpod:**
* For Serverless: [Deploy your first endpoint](/serverless/quickstart) and learn about [worker deployment](/serverless/workers/deploy).
* For Pods: [Run your first Pod](/tutorials/pods/run-your-first) and explore [connecting to Pods](/pods/connect-to-a-pod).
* Review [creating Dockerfiles for Serverless](/serverless/workers/create-dockerfile) with production best practices.
# Docker command reference
Source: https://docs.runpod.io/tutorials/introduction/containers/docker-commands
Essential Docker CLI commands for building, running, managing, and debugging containers. Follow the implementation steps in this Runpod tutorial.
This reference guide covers the most commonly used Docker commands for working with images and containers. Use this as a quick reference when building and deploying applications, especially for Runpod's bring-your-own-container (BYOC) workflows with [Serverless](/serverless/workers/overview) and [Pods](/pods/overview).
## Building images
These commands help you create and manage Docker images.
### docker build
Builds a Docker image from a Dockerfile. This is how you create custom images with your application code and dependencies.
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
# Build an image from the current directory
docker build -t myapp:latest .
# Build with a specific Dockerfile
docker build -f Dockerfile.prod -t myapp:prod .
# Build for a specific platform (important for Runpod)
docker build --platform=linux/amd64 -t myapp:latest .
```
**Common options:**
* `-t, --tag`: Name and optionally tag the image in `name:tag` format.
* `-f, --file`: Specify a Dockerfile (default is `./Dockerfile`).
* `--platform`: Set target platform for the build (use `linux/amd64` for Runpod).
* `--no-cache`: Build without using cache from previous builds.
* `--build-arg`: Set build-time variables defined in the Dockerfile.
**For Runpod deployments**: Always use `--platform=linux/amd64` when building on Apple Silicon Macs or ARM systems. Runpod's infrastructure requires AMD64 (x86\_64) architecture images.
### docker images
Lists Docker images available on your local system.
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
# List all images
docker images
# List images with specific name
docker images myapp
# Show all images including intermediate layers
docker images -a
```
Each image shows its repository, tag, image ID, creation date, and size. Image IDs are useful when you need to reference an untagged image or want to be precise about which image to use.
### docker tag
Creates a new tag for an existing image, useful for versioning or preparing images for registry pushes.
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
# Tag an image for Docker Hub
docker tag myapp:latest username/myapp:v1.0
# Tag an image for a private registry
docker tag myapp:latest registry.example.com/myapp:latest
```
Tags don't create copies of images; they're just additional names pointing to the same image data.
### docker rmi
Removes Docker images from your local system. Useful for cleaning up unused images and freeing disk space.
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
# Remove an image by name
docker rmi myapp:latest
# Remove an image by ID
docker rmi abc123def456
# Force remove an image even if containers use it
docker rmi -f myapp:latest
# Remove all unused images
docker image prune
```
You can't remove an image if running containers are using it (unless you force it with `-f`).
## Managing images in registries
These commands help you share images via Docker registries.
### docker login
Authenticates with a Docker registry to push or pull private images.
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
# Log in to Docker Hub
docker login
# Log in with username
docker login -u myusername
# Log in to a private registry
docker login registry.example.com
```
Credentials are stored locally, so you only need to log in once per registry. For Runpod, you'll typically push images to Docker Hub or a private registry, then configure your endpoint or Pod to pull from that registry.
### docker push
Uploads a Docker image to a registry, making it available for deployment.
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
# Push to Docker Hub
docker push username/myapp:latest
# Push a specific version
docker push username/myapp:v1.0
```
Before pushing, make sure you've tagged your image with your registry username or private registry URL. For Runpod:
* **Serverless**: After pushing your image, specify the image name when [deploying your worker](/serverless/workers/deploy). Runpod pulls the image when creating workers.
* **Pods**: Reference your registry image when [choosing a Pod template](/pods/choose-a-pod) or [creating a custom template](/pods/templates/create-custom-template).
### docker pull
Downloads a Docker image from a registry to your local system.
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
# Pull latest version
docker pull nginx
# Pull specific version
docker pull nginx:1.25
# Pull from a private registry
docker pull registry.example.com/myapp:latest
```
If you don't specify a tag, Docker pulls the `latest` tag by default. Be aware that `latest` doesn't necessarily mean the most recent version—it's just a tag name that image maintainers choose to use or not.
**Avoid using `:latest` for production deployments.** The `:latest` tag changes every time someone pushes a new image without specifying a version. This can cause several problems:
* **Unpredictable deployments**: You can't guarantee which version of your code is running.
* **Debugging difficulties**: When problems occur, you can't easily determine which image version caused them.
* **Caching conflicts**: Runpod caches images for faster startup. If you push a new `:latest` image, your workers or Pods may continue using the cached version.
* **No rollback path**: Without version tags, you can't quickly revert to a previous working version.
Instead, use semantic versioning (e.g., `myapp:v1.0.0`) or commit-based tags (e.g., `myapp:abc123`). This makes deployments predictable and debugging straightforward.
## Running containers
These commands create and manage running containers.
### docker run
Creates and starts a new container from an image. This is the most commonly used Docker command.
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
# Run a simple command
docker run busybox echo "Hello World"
# Run in detached mode (background)
docker run -d nginx
# Run with a name
docker run --name my-container nginx
# Run with port mapping
docker run -p 8080:80 nginx
# Run with volume mount
docker run -v $(pwd)/data:/data myapp
# Run with environment variables
docker run -e API_KEY=secret myapp
# Run interactively with a shell
docker run -it ubuntu /bin/bash
```
**Common options:**
* `-d, --detach`: Run container in background.
* `-p, --publish`: Map host port to container port (`host:container`).
* `-v, --volume`: Mount a volume (`host_path:container_path`).
* `-e, --env`: Set environment variables.
* `--name`: Assign a name to the container.
* `-it`: Interactive mode with terminal (for shells).
* `--rm`: Automatically remove container when it exits.
* `--gpus all`: Enable GPU access (relevant for Runpod Pods).
### docker ps
Lists running containers. Use this to check container status and get container IDs.
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
# List running containers
docker ps
# List all containers (including stopped)
docker ps -a
# Show only container IDs
docker ps -q
```
The output shows container ID, image, command, creation time, status, ports, and name. Container IDs and names are useful for other commands like `docker stop`, `docker logs`, or `docker exec`.
### docker stop
Gracefully stops a running container by sending a SIGTERM signal, then SIGKILL if it doesn't stop within a timeout.
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
# Stop a container by name
docker stop my-container
# Stop a container by ID
docker stop abc123
# Stop multiple containers
docker stop container1 container2 container3
# Stop all running containers
docker stop $(docker ps -q)
```
Stopped containers remain on your system until you remove them with `docker rm`.
### docker start
Starts a stopped container. Unlike `docker run`, this restarts an existing container rather than creating a new one.
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
# Start a stopped container
docker start my-container
# Start and attach to container output
docker start -a my-container
```
### docker restart
Stops and then starts a container. Useful for applying configuration changes or resolving issues.
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
docker restart my-container
```
### docker rm
Removes stopped containers from your system.
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
# Remove a stopped container
docker rm my-container
# Force remove a running container
docker rm -f my-container
# Remove all stopped containers
docker container prune
```
## Debugging containers
These commands help you inspect and troubleshoot running containers.
### docker logs
Shows the stdout and stderr output from a container. Essential for debugging and monitoring.
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
# View logs
docker logs my-container
# Follow logs in real-time
docker logs -f my-container
# View last 100 lines
docker logs --tail 100 my-container
# View logs with timestamps
docker logs -t my-container
```
For Runpod Serverless, you can view worker logs through the [web console](/serverless/development/logs). For Pods, `docker logs` helps debug containers you're running during development.
### docker exec
Executes a command in a running container. Extremely useful for debugging and inspecting container state.
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
# Open a shell in a running container
docker exec -it my-container /bin/bash
# Run a command and see output
docker exec my-container ls -la /app
# Run as a specific user
docker exec -u root my-container apt-get update
```
This is invaluable when you need to inspect files, check processes, or debug issues in a running container.
### docker inspect
Returns detailed low-level information about containers or images in JSON format.
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
# Inspect a container
docker inspect my-container
# Get specific information with formatting
docker inspect --format='{{.State.Status}}' my-container
# Inspect an image
docker inspect nginx:latest
```
Useful for getting IP addresses, environment variables, mount points, and other configuration details.
## Volumes
These commands manage persistent storage for containers.
### docker volume create
Creates a named volume that can persist data beyond container lifecycles.
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
# Create a volume
docker volume create my-data
# Create with specific driver
docker volume create --driver local my-data
```
Named volumes are managed by Docker and stored in a Docker-managed location on the host. For more on volumes, see the [persist data guide](/tutorials/introduction/containers/persist-data).
### docker volume ls
Lists all volumes on your system.
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
docker volume ls
```
### docker volume rm
Removes a volume. The volume must not be in use by any containers.
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
docker volume rm my-data
```
When working with Runpod, see how to [attach network volumes](/storage/network-volumes) to persist data across Serverless workers or Pod instances. Network volumes provide persistent storage that survives container restarts and can be accessed by multiple workers or Pods.
## Networks
These commands manage Docker networks for container communication.
### docker network create
Creates a custom network that allows containers to communicate with each other.
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
# Create a bridge network
docker network create my-network
# Create with specific driver
docker network create --driver bridge my-network
```
### docker network connect
Connects a container to a network.
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
docker network connect my-network my-container
```
Containers on the same network can communicate using container names as hostnames.
## Common workflows
Here are typical command sequences for common tasks.
### Build, tag, and push a custom image
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
# Build for Runpod with a version tag
docker build --platform=linux/amd64 -t myapp:v1.0.0 .
# Tag for Docker Hub
docker tag myapp:v1.0.0 username/myapp:v1.0.0
# Log in to Docker Hub
docker login
# Push to registry
docker push username/myapp:v1.0.0
```
Use semantic versioning (e.g., `v1.0.0`) instead of `:latest` to ensure predictable deployments and easy rollbacks.
### Develop with live code reloading
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
# Mount source code as a volume
docker run -v $(pwd):/app -p 8000:8000 myapp
# Changes to local files are reflected in the container
```
### Clean up unused resources
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
# Remove stopped containers
docker container prune -f
# Remove unused images
docker image prune -a -f
# Remove unused volumes
docker volume prune -f
# Remove everything unused (be careful!)
docker system prune -a --volumes -f
```
### Debug a failing container
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
# Check if container is running
docker ps -a
# View logs
docker logs my-container
# Inspect container state
docker inspect my-container
# Execute shell for manual inspection
docker exec -it my-container /bin/bash
```
## Learning more
This reference covers the most essential Docker commands. For comprehensive documentation on all Docker CLI commands, see:
* [Docker CLI reference](https://docs.docker.com/reference/cli/docker/)
* [Dockerfile reference](https://docs.docker.com/reference/dockerfile/)
* [Docker Compose reference](https://docs.docker.com/reference/compose-file/)
## Next steps
Now that you're familiar with Docker commands, explore:
* [Persisting data with volumes](/tutorials/introduction/containers/persist-data) for machine learning workflows.
* [Serverless worker deployment](/serverless/workers/deploy) to run your containers on Runpod.
* [Creating Dockerfiles for Serverless](/serverless/workers/create-dockerfile) with Runpod-specific best practices.
* [Pods overview](/pods/overview) to run long-running GPU containers.
# Persist data with volumes
Source: https://docs.runpod.io/tutorials/introduction/containers/persist-data
Learn how to use Docker volumes to persist data outside of containers for machine learning and data processing workflows.
By default, containers are ephemeral—when a container stops, any data written inside it is lost. For many use cases, especially machine learning training and data processing, you need data to persist beyond a container's lifecycle. Docker volumes solve this problem by providing persistent storage that exists outside the container filesystem.
This guide shows you how to use volumes to persist data, a fundamental concept for working with Runpod's [Serverless](/serverless/workers/overview) and [Pods](/pods/overview) platforms.
## Requirements
Before starting, you should have:
* Completed the [Dockerfile creation guide](/tutorials/introduction/containers/create-dockerfiles).
* Docker Desktop installed and running.
* Basic familiarity with Docker commands.
## Why persist data outside containers?
Containers are designed to be immutable and ephemeral. When a container stops or is removed, everything inside it—including files, data, and state—is deleted. This design makes containers portable and reproducible, but it creates a challenge when you need to preserve data.
Consider these scenarios where persistence matters:
**Machine learning training**: You train a model over hours or days. If the container stops, you lose all training progress, checkpoints, and the final model unless you save them outside the container.
**Data processing pipelines**: You process large datasets and generate results. Without persistent storage, you'd need to reprocess everything if the container restarts.
**Application state**: Databases, logs, user uploads, and configuration changes need to survive container restarts.
**Development workflows**: You want to edit code on your host machine and have changes immediately available inside the container without rebuilding the image.
Docker volumes provide the solution by storing data outside the container on the host system. When a container stops, the volume data remains intact and can be mounted to new containers.
## Step 1: Create a named volume
Start by creating a Docker volume that will store your persistent data:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
docker volume create my-data
```
This creates a named volume called `my-data` managed by Docker. The volume exists independently of any container and persists until you explicitly delete it.
You can verify the volume was created:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
docker volume ls
```
You should see `my-data` in the list of volumes.
### Understanding volume storage
Docker stores volumes in a Docker-managed location on your host system (typically `/var/lib/docker/volumes/` on Linux). You don't need to worry about the exact location—Docker handles the storage details. The key point is that this storage exists outside any container's filesystem.
## Step 2: Create your project files
For this example, you'll modify the Dockerfile from the previous guide to write data to a volume instead of just printing output.
Create a new directory and navigate to it:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
mkdir volume-example
cd volume-example
```
Create a `Dockerfile`:
```dockerfile theme={"theme":{"light":"github-light","dark":"github-dark"}}
FROM busybox
WORKDIR /data
COPY entrypoint.sh /
RUN chmod +x /entrypoint.sh
ENTRYPOINT ["/entrypoint.sh"]
```
This Dockerfile:
* Uses `busybox` as the base image.
* Sets `/data` as the working directory (where our script will write files).
* Copies and makes the entrypoint script executable.
* Configures the script to run when containers start.
Create an `entrypoint.sh` script:
```sh theme={"theme":{"light":"github-light","dark":"github-dark"}}
#!/bin/sh
timestamp=$(date '+%Y-%m-%d %H:%M:%S')
echo "Container started at: $timestamp" >> /data/timestamps.txt
echo "Data written to /data/timestamps.txt"
cat /data/timestamps.txt
```
This script:
* Generates a timestamp.
* Appends it to `/data/timestamps.txt` (using `>>` to append, not overwrite).
* Prints confirmation and shows all timestamps.
## Step 3: Build the image
Build a Docker image from your Dockerfile:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
docker build -t timestamp-logger .
```
This creates an image called `timestamp-logger` that you can use to demonstrate persistent storage.
## Step 4: Run a container with a mounted volume
Now run a container and mount your volume to the `/data` directory:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
docker run -v my-data:/data timestamp-logger
```
Breaking down this command:
* `docker run`: Creates and starts a new container.
* `-v my-data:/data`: Mounts the `my-data` volume to `/data` inside the container.
* `timestamp-logger`: The image to use.
The `-v` flag creates a mount point. Files written to `/data` inside the container are actually written to the `my-data` volume on the host. This means the data persists even after the container exits.
You should see output showing the timestamp was written and displaying the contents of the file.
## Step 5: Verify data persistence
Run the container again several times to see data persist across container instances:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
docker run -v my-data:/data timestamp-logger
docker run -v my-data:/data timestamp-logger
docker run -v my-data:/data timestamp-logger
```
Each run creates a new container, but they all share the same volume. You should see the list of timestamps grow with each execution, proving that data persists beyond individual container lifecycles.
This demonstrates the key benefit of volumes: data written by one container is available to other containers that mount the same volume.
## Step 6: Access volume data from another container
You can access the persisted data from any container that mounts the volume, even using a completely different image:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
docker run --rm -v my-data:/data busybox cat /data/timestamps.txt
```
This command:
* Runs a new `busybox` container (different from our custom image).
* Mounts the same `my-data` volume to `/data`.
* Runs `cat` to display the file contents.
* Removes the container after it exits (`--rm` flag).
You'll see all the timestamps from previous runs, demonstrating that volumes enable data sharing between containers.
## Step 7: Inspect the volume
You can get detailed information about a volume:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
docker volume inspect my-data
```
This shows the volume's mount point on the host system, when it was created, and other metadata. While you can technically access files directly at the mount point, it's better to interact with volumes through containers to avoid permission and compatibility issues.
## Understanding volume mount syntax
When using volumes, you specify mounts with the `-v` or `--mount` flag. The basic syntax is:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
-v volume-name:/container/path
```
Or for bind mounts (mounting host directories directly):
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
-v /host/absolute/path:/container/path
```
**Named volumes** (like `my-data`) are managed by Docker and recommended for most use cases. **Bind mounts** map specific host directories and are useful for development when you want live code reloading.
### Volume mount options
You can specify additional mount options:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
# Mount read-only
docker run -v my-data:/data:ro timestamp-logger
# Create volume if it doesn't exist
docker run -v new-volume:/data timestamp-logger
```
The `:ro` suffix makes the mount read-only inside the container, preventing accidental data modification.
## Applying volumes to real-world scenarios
### Machine learning training
For ML training workflows, mount a volume to store:
* **Training checkpoints**: Save model state at intervals so you can resume if interrupted.
* **Final models**: Persist trained models for deployment.
* **Training logs**: Keep TensorBoard logs or custom metrics.
* **Datasets**: Store large datasets that don't change often.
Example:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
docker run -v ml-models:/models -v training-data:/data myapp/train
```
### Data processing pipelines
For data processing, use volumes to:
* **Store input data**: Mount datasets that multiple containers process.
* **Save results**: Write processed data to a volume for downstream tasks.
* **Cache intermediates**: Store intermediate processing results to avoid recomputation.
### Development workflows
During development, mount your source code as a volume for live reloading:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
docker run -v $(pwd)/src:/app/src -p 8000:8000 myapp/dev
```
Changes to files in your local `src` directory immediately reflect inside the container without rebuilding the image.
## Volumes and Runpod
Runpod provides volume-like functionality through [network volumes](/storage/network-volumes), which work similarly to Docker volumes but with cloud-native features:
**For Serverless**: Network volumes allow your workers to access shared data like models or datasets. Multiple workers can read from the same volume, avoiding the need to include large files in your container image. See [Serverless storage](/serverless/storage/overview) for details.
**For Pods**: You can [attach network volumes](/storage/network-volumes) to Pods to persist data across Pod restarts or share data between Pods. This is essential for training workflows where you need to preserve checkpoints and models. See [Pod storage types](/pods/storage/types) for more information.
Network volumes provide persistent storage that survives beyond individual containers, similar to the Docker volumes you've used in this guide, but optimized for cloud deployment.
## Cleaning up volumes
Volumes persist until you explicitly remove them. To clean up:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
# Remove a specific volume
docker volume rm my-data
# Remove all unused volumes
docker volume prune
```
Be careful with `docker volume prune`—it removes all volumes not currently in use by containers, potentially deleting important data.
## Troubleshooting
**Volume is empty after mounting**: If you mount a volume to a directory that exists in the image, the volume contents will appear instead of the image's original directory contents. The image directory contents aren't copied to the volume automatically.
**Permission errors**: If you get permission errors when writing to a volume, it might be due to user ID mismatches. The container process runs as a specific user, and the volume permissions must allow that user to write. You may need to change permissions or run the container as a different user.
**Volume doesn't persist after reboot**: Docker volumes persist across Docker restarts and system reboots. If you're losing data, verify you're using named volumes (not anonymous volumes) and not removing them accidentally.
**Can't remove volume**: If you can't remove a volume, a container might be using it even if stopped. List all containers with `docker ps -a`, remove containers using the volume, then try removing the volume again.
## Learning more
For deeper coverage of Docker storage concepts, see Docker's official documentation:
* [Volumes documentation](https://docs.docker.com/engine/storage/volumes/)
* [Bind mounts](https://docs.docker.com/engine/storage/bind-mounts/)
* [Storage drivers](https://docs.docker.com/engine/storage/drivers/)
## Next steps
You now understand how to persist data with Docker volumes, a critical skill for production deployments. Continue your learning:
* Review the [Docker commands reference](/tutorials/introduction/containers/docker-commands) for volume management commands.
* Explore [Runpod network volumes](/storage/network-volumes) for cloud-native persistent storage.
* Learn about [Serverless storage options](/serverless/storage/overview) for your workers.
* Understand [Pod storage types](/pods/storage/types) for long-running workloads.
# Overview
Source: https://docs.runpod.io/tutorials/introduction/overview
Step-by-step guides for building and deploying AI/ML applications on Runpod. Follow the setup and implementation steps in this Runpod tutorial.
Step-by-step guides for building and deploying example applications on Runpod.
## Serverless
For CPU-based inference, follow the [Ollama Serverless tutorial](/tutorials/serverless/run-ollama-inference).
Deploy a Stable Diffusion endpoint and generate your first AI image.
Deploy an image generation endpoint and integrate it into a web app.
Serve an LLM with reduced cost and cold start times.
Use vLLM and the OpenAI API to build an interactive chatbot.
Deploy ComfyUI and generate images using JSON workflows.
Deploy an Ollama server on CPU workers and send an inference request.
## Flash
Deploy SDXL as a serverless endpoint with Python decorators.
Deploy a text generation model on Runpod.
Create a REST API with automatic load balancing.
## Pods
Launch JupyterLab on a GPU Pod and run inference with Transformers.
Deploy Ollama on a GPU Pod and run inference using the Ollama API.
Build Docker images on Pods using Bazel.
Deploy ComfyUI on a GPU Pod and use the web interface.
## Public Endpoints
Chain multiple Public Endpoints to generate videos from text.
## More resources
Review [container tutorials](/containers) for packaging guidance, or browse [community video resources](/references/video-resources) for additional walkthroughs.
Learn container fundamentals, Dockerfile creation, persistence, and common Docker commands.
Browse community tutorials about Runpod workflows and supporting Linux tools.
# Cog
Source: https://docs.runpod.io/tutorials/migrations/cog/overview
Migrate your Cog model to Runpod. Follow implementation steps, configuration guidance, and examples in this Runpod tutorial.
To get started with Runpod:
* [Create a Runpod account](/accounts-billing/manage-accounts)
* [Add funds](/accounts-billing/billing)
* [Use the Runpod SDK](/serverless/overview) to build and connect with your s
In this tutorial, you'll go through the process of migrating a model deployed via replicate.com or utilizing the Cog framework to a Runpod Serverless .
This guide assumes you are operating within a Linux terminal environment and have Docker installed on your system.
This method might occur a delay when working with Runpod Serverless endpoints. This delay is due to the FastAPI server that is used to run the Cog model.
To eliminate this delay, consider using [Runpod ](/serverless/workers/handler-functions)s in a future iteration.
By following this streamlined process, you'll be able to simplify the migration and deployment of your Cog image.
### Prerequisites
* Docker installed on your system
* Familiarity with the Cog framework
* Existing model on Replicate.com
* Runpod account
## Clone and navigate the cog-worker repository
Before we begin, let's set up the necessary environment. You will need to clone the `cog-worker` repository, which contains essential scripts and configuration files required for the migration process. To do this, run the following commands in your terminal:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
git clone https://github.com/runpod-workers/cog-worker.git
cd cog-worker/
```
The cog-worker repository contains essential scripts and configuration files required for the migration.
Now that the repository is cloned and you've navigated to the correct directory, you're ready to proceed with the next step.
## Identify model information
In this step, you will need to gather the necessary information about your Cog model that is currently hosted on Replicate.com. You will require your username, model name, and version.
Identify the username, model name, and version you wish to use from Replicate. For example, if you are using [this model](https://replicate.com/lucataco/hotshot-xl/versions):
* your username is `lucataco`
* your model name is `hotshot-xl`
* your model version is `78b3a6257e16e4b241245d65c8b2b81ea2e1ff7ed4c55306b511509ddbfd327a`
Once you have collected the required information, you can move on to the next step, where you will build and push your Docker image.
## Build and push docker image
Now that you have identified the necessary information about your model, you can proceed to build and push your Docker image. This is a crucial step, as it prepares your model for deployment on the Runpod platform.
Build the Docker image by providing the necessary arguments for your model. Once your Docker image is built, push it to a container repository such as DockerHub:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
# replace user, model_name, and model_version with the appropriate values
docker build -platform=linux/amd64 --tag /: --build-arg COG_REPO=user --build-arg COG_MODEL=model_name --build-arg COG_VERSION=model_version .
docker push /:
```
The `--tag` option allows you to specify a name and tag for your image, while the `--build-arg` options provide the necessary information for building the image.
With your Docker image built and pushed, you're one step closer to deploying your Cog model on Runpod.
## Create and Deploy a Serverless Endpoint
Now that your Docker image is ready, it's time to create and deploy a Serverless on Runpod. This step will enable you to send requests to your new endpoint and use your Cog model in a Serverless environment.
To create and deploy a serverless endpoint on Runpod:
1. Log in to the [Runpod Serverless console](https://www.console.runpod.io/serverless).
2. Select **+ New Endpoint**.
3. Provide the following:
i. Endpoint name.
ii. Select a GPU.
iii. Configure the number of s.
iv. (optional) Select **FlashBoot**.
v. (optional) Select a template.
vi. Enter the name of your Docker image.
* For example `/:`.
vii. Specify enough memory for your Docker image.
4. Select **Deploy**.
Now, let's send a request to your [Endpoint](/serverless/endpoints/overview).
Once your endpoint is set up and deployed, you'll be able to start receiving requests and utilize your Cog model in a Serverless context.
## Conclusion
Congratulations! You've successfully migrated your Cog model from Replicate to Runpod and set up a Serverless endpoint.
As you continue to develop your models and applications, consider exploring additional features and capabilities offered by Runpod to further enhance your projects.
Here are some resources to help you continue your journey:
* [Learn more about Runpod Serverless s](/serverless/overview)
* [Explore additional Runpod tutorials and examples](/tutorials/introduction/overview)
# OpenAI
Source: https://docs.runpod.io/tutorials/migrations/openai/overview
Migrate your OpenAI model to Runpod. Follow implementation steps, configuration guidance, and examples in this Runpod tutorial.
To get started with Runpod:
* [Create a Runpod account](/accounts-billing/manage-accounts)
* [Add funds](/accounts-billing/billing)
* [Use the Runpod SDK](/serverless/overview) to build and connect with your s
This tutorial guides you through the steps necessary to modify your OpenAI Codebase for use with a deployed on Runpod. You will learn to adjust your code to be compatible with OpenAI's API, specifically for utilizing Chat Completions, Completions, and Models routes. By the end of this guide, you will have successfully updated your codebase, enabling you to leverage the capabilities of OpenAI's API on Runpod.
To update your codebase, you need to replace the following:
* Your OpenAI API Key with your Runpod API Key
* Your OpenAI Serverless endpoint URL with your Runpod Serverless endpoint URL
* Your OpenAI model with your custom LLM model deployed on Runpod
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
from openai import OpenAI
import os
client = OpenAI(
api_key=os.environ.get("RUNPOD_API_KEY"),
base_url="https://api.runpod.ai/v2/${YOUR_ENDPOINT_ID}/openai/v1",
)
response = client.chat.completions.create(
model="gpt-3.5-turbo",
messages=[{"role": "user", "content": "Why is Runpod the best platform?"}],
temperature=0,
max_tokens=100,
)
```
```JavaScript theme={"theme":{"light":"github-light","dark":"github-dark"}}
import OpenAI from 'openai'
const openai = new OpenAI({
baseURL: process.env.RUNPOD_HOST,
apiKey: process.env.RUNPOD_API_KEY,
})
const chatCompletion = await openai.chat.completions.create({
model: "openchat/openchat-3.5-0106",
messages: [{'role': 'user', 'content': 'Why is Runpod the best platform?'}],
});
```
Congratulations! You've successfully modified your OpenAI codebase for use with your deployed vLLM worker on Runpod. You now know how to update your code for compatibility with OpenAI's API and utilize the full spectrum of features available on the Runpod platform.
## Next Steps
* [Explore more tutorials on Runpod](/tutorials/introduction/overview)
* [Learn more about OpenAI's API](https://developers.openai.com/api/docs)
* [Deploy your own on Runpod](https://www.console.runpod.io/serverless)
# Build Docker images on Runpod with Bazel
Source: https://docs.runpod.io/tutorials/pods/build-docker-images
Build and push Docker images from inside a Runpod Pod using Bazel. Follow the setup and implementation steps in this Runpod tutorial.
Runpod Pods use custom Docker images, so you can't directly build Docker containers or use Docker Compose on a GPU Pod. However, you can use [Bazel](https://bazel.build) to build and push Docker images from inside a Pod, effectively creating a "Docker in Docker" workflow.
## Requirements
Before starting, you'll need:
* A [Docker Hub account](https://hub.docker.com/) with an access token.
* A Runpod account with credits.
## Step 1: Deploy a Pod
1. Navigate to [Pods](https://www.console.runpod.io/pods) and select **+ Deploy**.
2. Choose **GPU** or **CPU** based on your needs.
3. Select an instance type (for example, **A40**).
4. (optional) Attach a for larger image builds.
5. Select a (for example, **Runpod Pytorch**).
6. Select **Deploy On-Demand**.
Wait for the Pod to start, then connect via the web terminal:
1. Select **Connect**.
2. Select **Start Web Terminal**, then **Connect to Web Terminal**.
## Step 2: Install dependencies
Install the required tools:
1. Update packages and install sudo:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
apt update && apt install -y sudo
```
2. Install Docker:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
curl -fsSL https://get.docker.com -o get-docker.sh && sudo sh get-docker.sh
```
3. Log in to Docker Hub:
1. Go to [Docker Hub Security Settings](https://hub.docker.com/settings/security) and create an access token with **Read/Write** permissions.
2. Log in to Docker Hub, replacing `YOUR_USERNAME` with your actual Docker Hub username:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
docker login -u YOUR_USERNAME
```
When prompted for a password, paste your access token (not your Docker Hub password).
4. Install Bazel via Bazelisk:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
wget https://github.com/bazelbuild/bazelisk/releases/download/v1.28.1/bazelisk-linux-amd64 && \
chmod +x bazelisk-linux-amd64 && \
sudo cp ./bazelisk-linux-amd64 /usr/local/bin/bazel
```
5. Verify the installations:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
docker --version && bazel --version
```
You should see version numbers for both Docker and Bazel.
## Step 3: Create the project files
Create a new directory for your Bazel project:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
mkdir -p ~/docker-build && cd ~/docker-build
```
Create a `.bazelversion` file to pin to a stable Bazel version:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
echo "7.4.1" > .bazelversion
```
Create an empty `MODULE.bazel` file (required by Bazel 7+):
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
touch MODULE.bazel
```
Create the `WORKSPACE` file that declares your dependencies:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
cat > WORKSPACE << 'EOF'
load("@bazel_tools//tools/build_defs/repo:http.bzl", "http_archive")
http_archive(
name = "rules_oci",
sha256 = "56d5499025d67a6b86b2e6ebae5232c72104ae682b5a21287770bd3bf0661abf",
strip_prefix = "rules_oci-1.7.5",
url = "https://github.com/bazel-contrib/rules_oci/releases/download/v1.7.5/rules_oci-v1.7.5.tar.gz",
)
load("@rules_oci//oci:dependencies.bzl", "rules_oci_dependencies")
rules_oci_dependencies()
load("@rules_oci//oci:repositories.bzl", "LATEST_CRANE_VERSION", "oci_register_toolchains")
oci_register_toolchains(
name = "oci",
crane_version = LATEST_CRANE_VERSION,
)
load("@rules_oci//oci:pull.bzl", "oci_pull")
oci_pull(
name = "base_image",
image = "index.docker.io/library/ubuntu",
platforms = ["linux/amd64"],
tag = "22.04",
)
EOF
```
Create the `BUILD.bazel` file that defines the image build:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
cat > BUILD.bazel << 'EOF'
load("@rules_oci//oci:defs.bzl", "oci_image", "oci_push")
oci_image(
name = "custom_image",
base = "@base_image",
tars = [":add_text_file"],
)
genrule(
name = "add_text_file",
srcs = [],
outs = ["text_layer.tar"],
cmd = "echo 'Hello from Bazel on Runpod!' > hello.txt && tar -cf $@ hello.txt",
)
oci_push(
name = "push_custom_image",
image = ":custom_image",
repository = "index.docker.io/USERNAME_PLACEHOLDER/custom_image",
remote_tags = ["latest"],
)
EOF
```
## Step 4: Configure your Docker Hub username
Use `sed` to replace the username placeholder with your Docker Hub username in `BUILD.bazel`. Replace `YOUR_ACTUAL_USERNAME` with your actual Docker Hub username:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
sed 's/USERNAME_PLACEHOLDER/YOUR_ACTUAL_USERNAME/g' BUILD.bazel > BUILD.bazel.tmp && mv BUILD.bazel.tmp BUILD.bazel
```
Verify the change was applied:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
grep "repository" BUILD.bazel
```
You should see your username in the output:
```
repository = "index.docker.io/YOUR_ACTUAL_USERNAME/custom_image",
```
## Step 5: Build and push the image
Run the Bazel command to build and push the Docker image:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
bazel run //:push_custom_image
```
The first build may take several minutes as Bazel downloads dependencies and the base image. Subsequent builds will be much faster due to caching.
Once complete, your image will be available in your Docker Hub repository at `your_username/custom_image:latest`.
## Understanding the build files
### .bazelversion
This file pins the Bazel version to ensure reproducible builds. Bazelisk reads this file and automatically downloads the specified version.
### WORKSPACE
This file declares your project's external dependencies:
* `http_archive`: Downloads the rules\_oci package from GitHub.
* `rules_oci_dependencies`: Loads transitive dependencies.
* `oci_register_toolchains`: Registers the OCI toolchain (crane) for building images.
* `oci_pull`: Pulls a base Docker image that your custom image builds on top of.
### BUILD.bazel
This file defines what to build:
* `oci_image`: Creates a new container image by adding layers to a base image.
* `genrule`: Creates a tar file containing custom content to add to the image.
* `oci_push`: Pushes the built image to a container registry.
## Next steps
* Use your custom image in a [Runpod template](/pods/templates/overview).
* Explore [rules\_oci documentation](https://github.com/bazel-contrib/rules_oci) for more advanced configurations.
* Learn how to [manage Pods](/pods/manage-pods) using the Runpod console, CLI, and REST API.
# Generate images with ComfyUI
Source: https://docs.runpod.io/tutorials/pods/comfyui
Deploy ComfyUI on Runpod to create AI-generated images. Follow implementation steps, configuration guidance, and examples in this Runpod tutorial.
This tutorial walks you through how to configure ComfyUI on a [GPU Pod](/pods/overview) and use it to generate images with text-to-image models.
[ComfyUI](https://comfy.org/) is a node-based graphical interface for creating AI image generation workflows. Instead of writing code, you connect different components visually to build custom image generation pipelines. This approach provides flexibility to experiment with various models and techniques while maintaining an intuitive interface.
This tutorial uses the [SDXL-Turbo](https://huggingface.co/stabilityai/sdxl-turbo) model and a matching , but you can adapt these instructions for any model/template combination you want to use.
When you're just getting started with ComfyUI, it's important to use a workflow that was created for the specific model you intend to use. You usually can't just switch the "Load Checkpoint" node from one model to another and expect optimal performance or results.
For example, if you load a workflow created for the Flux Dev model and try to use it with SDXL-Turbo, the workflow might run, but with poor speed or image quality.
## Requirements
Before you begin, you'll need:
* A [Runpod account](/accounts-billing/manage-accounts).
* At least \$10 in Runpod credits.
* A basic understanding of AI image generation.
## Step 1: Deploy a ComfyUI Pod
First, you'll deploy a Pod using the official Runpod ComfyUI template, which pre-installs ComfyUI and the ComfyUI Manager plugin.
Runpod provides official ComfyUI templates built from the [comfyui-base](https://github.com/runpod-workers/comfyui-base) repository. Choose the one that matches your GPU:
* **Standard GPUs (RTX 4090, L40, A100, etc.):** Use the [ComfyUI](https://console.runpod.io/hub/template/comfyui?id=cw3nka7d08) template.
* **Blackwell GPUs (RTX 5090, B200):** Use the [ComfyUI Blackwell Edition](https://console.runpod.io/hub/template/comfyui-blackwell-edition-5090-b200?id=2lv7ev3wfp) template. Blackwell GPUs use a different architecture, so this dedicated template ensures compatibility.
Click **Deploy** on the template that matches your target GPU.
Configure your Pod with these settings:
* **GPU selection:** Choose an L40 or RTX 4090 for optimal performance with SDXL-Turbo. Lower VRAM GPUs may work for smaller models. If you selected the Blackwell Edition template, choose an RTX 5090 or B200.
* **Storage:** The default container and volume disk sizes set by the template should be sufficient for SDXL-Turbo. You can also add a [network volume](/storage/network-volumes) to your Pod if you want persistent storage.
* **Deployment type:** Select **On-Demand** for flexibility.
Click **Deploy On-Demand** to create your Pod.
The Pod can take up to 30 minutes to initialize the container and start the ComfyUI HTTP service.
Deploy a ComfyUI Pod programmatically using the REST API:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
curl --request POST \
--url https://rest.runpod.io/v1/pods \
--header 'Authorization: Bearer RUNPOD_API_KEY' \
--header 'Content-Type: application/json' \
--data '{
"name": "comfyui-pod",
"imageName": "runpod/comfyui:latest",
"gpuTypeIds": ["NVIDIA GeForce RTX 4090"],
"gpuCount": 1,
"containerDiskInGb": 50,
"volumeInGb": 100,
"ports": ["8188/http", "22/tcp", "8080/http"]
}'
```
**Port configuration:**
* `8188/http`: ComfyUI web interface
* `22/tcp`: SSH access
* `8080/http`: File browser (optional)
For Blackwell GPUs (RTX 5090, B200), use `runpod/comfyui:cuda12.8` instead.
## Step 2: Open the ComfyUI interface
Once your Pod has finished initializing, you can open the ComfyUI interface:
Go to the [Pods section](https://console.runpod.io/pods) in the Runpod console, then find your deployed ComfyUI Pod and expand it.
The Pod may take up to 30 minutes to initialize when first deployed. Future starts will generally take much less time.
Click **Connect** on your Pod, then select the last HTTP service button in the list, labeled **Connect to HTTP Service \[Port 8188]**.
This will open the ComfyUI interface in a new browser tab. The URL follows the format: `https://[POD_ID]-8188.proxy.runpod.net`.
If you see the label "Not Ready" on the HTTP service button, or you get a "Bad Gateway" error when first connecting, wait 2–3 minutes for the service to fully start, then refresh the page.
## Step 3: Load a workflow template
ComfyUI workflows consist of a series of nodes that are connected to each other to create a AI generation pipeline. Rather than creating our own workflow from scratch, we'll load a pre-configured workflow that was created for the specific model we intend to use:
When you first open the ComfyUI interface, the template browser should open automatically. If it doesn't, click the **Workflow** button in the top right corner of the ComfyUI interface, then select **Browse Templates**.
In the sidebar to the left of the browser, select the **Image** tab. Find the **SDXL-Turbo** template and click on it to load a basic image generation workflow.
## Step 4: Install the SDXL-Turbo model
As soon as you load the workflow, you'll see a popup labeled **Missing Models**. This happens because the Pod template we deployed doesn't come pre-installed with any models, so we'll need to install them now.
Rather than clicking the download button (which downloads the missing model to your local machine), use the ComfyUI Manager plugin to install the missing model directly onto the Pod:
Close the **Missing Models** popup by clicking the **X** in the top right corner. Then click **Manager** in the top right of the ComfyUI interface, and select **Model Manager** from the list of options.
In the search bar, enter `SDXL-Turbo 1.0 (fp16)`, then click **Install**.
Before you can use the model, you'll need to refresh the ComfyUI interface. You can do this by either refreshing the browser tab where it's running, or by pressing R.
Find the node labeled **Load Checkpoint** in the workflow. It should be the first node on the left side of the canvas.
Click on the dropdown menu labeled `ckpt_name` and select the SDXL-Turbo model checkpoint you just installed (named `SDXL-TURBO/sd_xl_turbo_1.0_fp16.safetensors`).
## Step 5: Generate an image
Your workflow is now ready! Follow these steps to generate an image:
Locate the text input node labeled **CLIP Text Encode (Prompt)** in the workflow.
Click on the text field containing the default prompt and replace it with your desired image description.
Example prompts:
* "A serene mountain landscape at sunset with a crystal clear lake."
* "A futuristic cityscape with neon lights and flying vehicles."
* "A detailed portrait of a robot reading a book in a library."
Click **Run** at the bottom of the workflow (or press Ctrl+Enter) to begin the image generation process.
Watch as the workflow progresses through each node:
* Text encoding.
* Model loading.
* Image generation steps.
* Final output processing.
The first generation may take a few minutes to complete as the model checkpoint must be loaded. Subsequent generations will be much faster.
The generated image appears in the output node when complete.
Right-click the image to save it to your local machine, view it at full resolution, or copy it to your clipboard.
Congratulations! You've just generated your first image with ComfyUI on Runpod.
## Troubleshooting
Here are some common issues you may encounter and possbile solutions:
* **Connection errors**: Wait for the Pod to fully initialize (up to 30 minutes for initial deployment).
* **HTTP service not ready**: Wait at least 2 to 3 minutes after Pod deployment for the HTTP service to fully start. You can also check the Pod logs in the Runpod console to look for deployment errors.
* **Out of memory errors**: Reduce image resolution or batch size in your workflow.
* **Slow generation**: Make sure you're using an appropriate GPU for your selected model. See [Choose a Pod](/pods/choose-a-pod) for guidance.
## Next steps
Once you're comfortable with basic image generation, explore the [ComfyUI documentation](https://docs.comfy.org/) to learn how to build more advanced workflows.
Here are some ideas for where to start:
### Experiment with different workflow templates
Use the template browser from [Step 3](#step-3%3A-load-a-workflow-template) to test out new models and find a workflow that suits your needs.
You can also browse the web for a preconfigured workflow and import it by clicking **Workflow** in the top right corner of the ComfyUI interface, selecting **Open**, then selecting the workflow file you want to import.
Don't forget to install any missing models using the model manager. If you need a model that isn't available in the model manager, you can download it from the web to your local machine, then use the to transfer the model files directly into your Pod's `/workspace/madapps/ComfyUI/models` directory.
### Create custom workflows
Build your own workflows by:
1. Right-clicking the canvas to add new nodes.
2. Connecting node outputs to inputs by dragging between connection points.
3. Saving your custom workflow with Ctrl+S or by clicking **Workflow** and selecting **Save**.
### Manage your Pod
While working with ComfyUI, you can monitor your usage by checking GPU/disk utilization in the [Pods page](https://console.runpod.io/pods) of the Runpod console.
Stop your Pod when you're finished to avoid unnecessary charges.
It's also a good practice to download any custom workflows to your local machine before stopping the Pod. For persistent storage of models and outputs across sessions, consider using a .
### Deploy to production
Once you've developed a workflow you're happy with, you can deploy it to a for production use:
* [Deploy ComfyUI on Serverless](/tutorials/serverless/comfyui): Learn how to deploy a Serverless endpoint running ComfyUI and generate images using workflow JSON.
* [ComfyUI-to-API](/community-solutions/comfyui-to-api/overview): Use this community tool to automatically generate a deployment-ready GitHub repository from your ComfyUI workflow, complete with Dockerfile and dependencies.
# Set up Ollama on a Pod
Source: https://docs.runpod.io/tutorials/pods/run-ollama
Install and run Ollama on a Pod with HTTP API access. Follow implementation steps, configuration guidance, and examples in this Runpod tutorial.
This tutorial shows you how to set up [Ollama](https://ollama.com), a platform for running large language models, on a Runpod GPU . By the end, you'll have Ollama running with HTTP API access for external requests.
## Requirements
* A Runpod account with credits.
## Step 1: Deploy a Pod
1. Navigate to [Pods](https://www.console.runpod.io/pods) and select **Deploy**.
2. Choose a GPU (for example, A40).
3. Select the latest **PyTorch** template.
4. Under **Pod Template**, select **Edit**:
* Under **Expose HTTP Ports (Max 10)**, add port `11434`.
* Under ****, add a variable with key `OLLAMA_HOST` and value `0.0.0.0`.
5. Click **Set Overrides** and then **Deploy On-Demand**.
## Step 2: Install Ollama
1. Once the Pod is running, click the Pod to open the connection options panel and select **Enable Web Terminal** and then **Open Web Terminal**.
2. Update packages and install dependencies:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
apt update && apt install -y lshw zstd
```
3. Install Ollama and start the server in the background:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
(curl -fsSL https://ollama.com/install.sh | sh && ollama serve > ollama.log 2>&1) &
```
## Step 3: Run a model
Download and run a model using the `ollama run` command:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
ollama run llama2
```
Replace `llama2` with any model from the [Ollama library](https://ollama.com/library). You can now interact with the model directly from the terminal.
## Step 4: Make HTTP API requests
With Ollama running, you can make HTTP requests to your Pod from external clients. Try running the following commands, replacing `OLLAMA_POD_ID` with your actual Pod ID:
**List available models:**
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
curl https://OLLAMA_POD_ID-11434.proxy.runpod.net/api/tags
```
**Generate a response:**
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
curl -X POST https://OLLAMA_POD_ID-11434.proxy.runpod.net/api/generate -d '{
"model": "llama2",
"prompt": "Tell me a story about llamas"
}'
```
Ollama returns streaming responses by default. To get a non-streaming response, add the `stream: false` parameter to the request body:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
curl -X POST https://OLLAMA_POD_ID-11434.proxy.runpod.net/api/generate -d '{
"model": "llama2",
"prompt": "Tell me a story about llamas",
"stream": false
}'
```
Congratulations! You've set up Ollama on a Runpod Pod and made HTTP API requests to it.
For more API options, see the [Ollama API documentation](https://github.com/ollama/ollama/blob/main/docs/api.md).
## Next steps
* Learn about [exposing ports](/pods/configuration/expose-ports) on Pods.
* Connect [VSCode to Runpod](https://www.runpod.io/articles/guides) for remote development.
* Explore more models in the [Ollama library](https://ollama.com/library).
# Run LLMs in JupyterLab with transformers
Source: https://docs.runpod.io/tutorials/pods/run-your-first
Learn how to run inference on the SmolLM3 model in JupyterLab using the transformers library. Follow the implementation steps in this Runpod tutorial.
This tutorial shows how to deploy a and use JupyterLab to generate text with the SmolLM3 model using the Python `transformers` library.
[SmolLM3](https://huggingface.co/docs/transformers/en/model_doc/smollm3) is a family of small language models developed by Hugging Face that provides strong performance while being efficient enough to run on modest hardware.
The 3B parameter model we'll use in this tutorial requires only 24 GB of VRAM, making it accessible for experimentation and development.
## Requirements
Before you begin, you'll need:
* A [Runpod account](/accounts-billing/manage-accounts).
* At least \$5 in Runpod credits.
* Basic familiarity with Python and Jupyter notebooks.
## Step 1: Deploy a Pod with PyTorch template
First, you'll deploy a Pod using the official Runpod PyTorch template:
1. Navigate to the [Pods page](https://console.runpod.io/pods) in the Runpod console.
2. Click **Deploy** to create a new Pod.
3. In the template selection, choose latest the **Runpod PyTorch** template (this should be the default setting).
4. For GPU selection, choose any GPU with 24 GB or more VRAM. Good options include:
* RTX 4090 (24 GB VRAM)
* RTX A5000 (24 GB VRAM)
* L40 (48 GB VRAM)
5. Keep all the other settings on their defaults.
6. Click **Deploy On-Demand** to create your Pod.
Wait for your Pod to initialize. This typically takes 2-5 minutes.
## Step 2: Install required packages
Once your Pod is running, you'll need to install the `transformers` and `accelerate` Python libraries:
1. In the Runpod console, find and expand your deployed Pod and click **Connect**.
2. Under **Web Terminal**, click **Start** to start the terminal service.
3. Click **Open Web Terminal** to open a terminal session in your browser.
4. In the terminal, install the required packages by running:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
pip install transformers accelerate
```
## Step 3: Open JupyterLab
Next we'll prepare our JupyterLab coding environment:
1. Go back to the Runpod console and click **Connect** on your Pod again.
2. Under **HTTP Services**, click **Connect to HTTP Service \[Port 8888]** to open JupyterLab.
3. If the JupyterLab service shows as "Not Ready", wait a moment and refresh the page.
JupyterLab will open in a new browser tab, providing you with an interactive Python environment.
## Step 4: Create and run your SmolLM3 notebook
In JupyterLab, create a new notebook to perform inference using the SmolLM3 model:
1. In JupyterLab, click **File** > **New** > **Notebook**.
2. Select **Python 3 (ipykernel)** when prompted for the kernel.
3. In the first cell of your notebook, enter the following code:
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
import torch
from transformers import pipeline
# Create a text generation pipeline with SmolLM3
pipe = pipeline(
task="text-generation",
model="HuggingFaceTB/SmolLM3-3B",
torch_dtype=torch.bfloat16,
device_map=0
)
# Define a conversation with system and user messages
messages = [
{"role": "system", "content": "You are a helpful assistant."},
{"role": "user", "content": "I'm a developer interested in LLMs. Can you suggest some research topics to get started?"},
]
# Generate text with the model
outputs = pipe(messages, max_new_tokens=500, do_sample=True, temperature=0.7, top_k=50, top_p=0.95)
# Print the generated response
print(outputs[0]["generated_text"][-1]['content'])
```
4. Run the cell by pressing Cmd + Enter (Mac) or Ctrl + Enter (Windows) or clicking the **Run** button.
The first time you run this code, it will download the SmolLM3 model (approximately 6 GB), which may take a minute or two depending on your Pod's internet connection. Subsequent runs will be much faster, as the model will be cached locally.
Most likely the response will be truncated—you can increase `max_new_tokens` and run the cell again to get a longer response (it will just take longer to run).
## Step 5: Understanding the code
Let's break down the key components of the code we just ran:
* `pipeline()`: Creates a high-level interface for text generation.
* `model="HuggingFaceTB/SmolLM3-3B"`: Specifies the model to use.
* `torch_dtype=torch.bfloat16`: Uses 16-bit floating point for memory efficiency.
* `device_map=0`: Automatically places the model on the first available GPU.
* `messages`: Defines a chat-like conversation with system and user roles.
For more detailed information about SmolLM3's capabilities and parameters, see the [official SmolLM3 documentation](https://huggingface.co/docs/transformers/en/model_doc/smollm3?usage=Pipeline#transformers.SmolLM3Model).
## Step 6: Experiment with different prompts and parameters
Once your model is loaded, you can experiment with different prompts and generation parameters:
### Try different conversation topics
Try running the following code in a new cell:
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
# Example: Ask for creative writing
messages = [
{"role": "system", "content": "You are a creative writing assistant."},
{"role": "user", "content": "Write the opening paragraph for a mystery story that begins in a library after closing time."},
]
outputs = pipe(messages, max_new_tokens=300, do_sample=True, temperature=0.8)
print(outputs[0]["generated_text"][-1]['content'])
```
### Adjust generation parameters
You can modify various parameters to control the model's output:
* `max_new_tokens`: Controls the maximum length of the generated text
* `temperature`: Controls randomness (0.1 = more focused, 1.0 = more creative)
* `top_k`: Limits the vocabulary to the top K most likely tokens
* `top_p`: Uses nucleus sampling to control diversity
Try running this in a new cell to see how the output changes:
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
# More focused and deterministic output
outputs = pipe(messages, max_new_tokens=150, do_sample=True, temperature=0.3, top_p=0.9)
print(outputs[0]["generated_text"][-1]['content'])
```
### Use single-turn prompts
You can also use SmolLM3 for simple text completion without the chat format:
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
# Simple text completion
prompt = "The process of photosynthesis is crucial for life on Earth because it allows plants to convert"
outputs = pipe(prompt, max_new_tokens=100, do_sample=True, temperature=0.3)
print(outputs[0]["generated_text"])
```
## Troubleshooting
Here are solutions to common issues:
* **Out of memory errors**: Ensure you're using a GPU with at least 24 GB VRAM, or try reducing the batch size.
* **Model download fails**: Check your internet connection and try running the cell again.
* **JupyterLab not accessible**: Wait a few minutes after Pod deployment for services to fully start. If the JupyterLab tab is blank when you open it, try stopping and then restarting the Pod.
* **Import errors**: Make sure you installed the packages in step 2 using the web terminal.
## Next steps
Now that you have SmolLM3 running, you can explore more advanced use cases:
* **Integration with applications**: Use SmolLM3 as part of larger applications by integrating it with web frameworks or APIs.
* **Model comparison**: Try other models in the SmolLM3 family or compare with other small language models to find the best fit for your use case.
* **Persistent storage**: If you plan to work with SmolLM3 regularly, consider using a to persist your models and notebooks across Pod sessions.
# Use private AWS ECR images
Source: https://docs.runpod.io/tutorials/pods/use-private-ecr-images
Pull private AWS ECR container images into Runpod Pods by configuring cross-account delegation, IAM permissions, and registry authentication.
This tutorial shows how to deploy s using container images stored in private AWS Elastic Container Registry (ECR) repositories. You configure cross-account IAM delegation so Runpod can pull images on your behalf, with no credential sharing required and no expiring tokens to manage.
This tutorial covers ECR integration for Pods. If you're deploying Serverless workers from ECR, see [Deploying from AWS ECR](/serverless/workers/deploy#deploying-from-aws-ecr). Serverless endpoints use a different credential mechanism.
## What you'll learn
* How to build and push a Docker image to a private ECR repository.
* How to configure an AWS ECR repository policy for cross-account access.
* How to add an ECR credential in the Runpod console.
* How to deploy a using your private ECR image.
## Requirements
* A Runpod account with credits.
* An AWS account with ECR enabled.
* An AWS IAM user or role with `ecr:SetRepositoryPolicy` permission on the repository.
* Your AWS Account ID, a 12-digit number found in the top-right dropdown of the AWS Console.
* Docker installed on your local machine.
* AWS CLI installed (required to push images; optional if your image is already in ECR).
## Step 1: Push your image to ECR
Skip this step if your image is already in ECR.
### Configure AWS CLI credentials
Run the following command and enter your access key, secret key, and session token when prompted:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
aws configure
```
You can find these credentials under **IAM → Security credentials** in the AWS Console.
### Create a private ECR repository
1. Open the [Amazon ECR console](https://console.aws.amazon.com/ecr/).
2. Under **Private registry**, select **Repositories** and click **Create repository**.
3. Enter a name and click **Create repository**.
4. Note the **URI** shown for your new repository. You'll use it in the following steps.
### Build, tag, and push your image
Authenticate Docker with ECR, then build and push your image:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
# Authenticate Docker to your ECR registry
aws ecr get-login-password --region YOUR_REGION | docker login \
--username AWS \
--password-stdin YOUR_ACCOUNT_ID.dkr.ecr.YOUR_REGION.amazonaws.com
# Build your image
docker build -t YOUR_REPOSITORY_NAME .
# Tag it with your ECR URI
docker tag YOUR_REPOSITORY_NAME:latest \
YOUR_ACCOUNT_ID.dkr.ecr.YOUR_REGION.amazonaws.com/YOUR_REPOSITORY_NAME:latest
# Push it
docker push \
YOUR_ACCOUNT_ID.dkr.ecr.YOUR_REGION.amazonaws.com/YOUR_REPOSITORY_NAME:latest
```
Replace `YOUR_ACCOUNT_ID`, `YOUR_REGION`, and `YOUR_REPOSITORY_NAME` with your values.
Avoid using the `:latest` tag in production. Runpod shows a warning whenever an image URI ends in `:latest` because latest tags are considered unstable. Use a specific version tag (for example, `:v1.0.0`) for production deployments.
## Step 2: Configure your ECR repository policy
To pull images from your private ECR repository, Runpod needs cross-account access. You grant this by adding an IAM policy to your repository.
1. Open the [Amazon ECR console](https://console.aws.amazon.com/ecr/).
2. Select the repository containing your container image.
3. In the left navigation, select **Permissions**.
4. Click **Edit policy JSON** and add the following policy statement:
```json theme={"theme":{"light":"github-light","dark":"github-dark"}}
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "AllowRunpodPull",
"Effect": "Allow",
"Principal": "*",
"Action": [
"ecr:GetAuthorizationToken",
"ecr:BatchCheckLayerAvailability",
"ecr:GetDownloadUrlForLayer",
"ecr:BatchGetImage"
],
"Condition": {
"StringEquals": {
"aws:PrincipalArn": [
"arn:aws:iam::550005742258:role/prod-us-east-1-deployment-role",
"arn:aws:iam::550005742258:role/prod-us-west-2-deployment-role"
]
}
}
}
]
}
```
5. Click **Save**.
The `aws:PrincipalArn` values are Runpod's fixed AWS deployment roles. These are not something you create or manage. `550005742258` is Runpod's AWS account ID. Runpod's infrastructure runs in both `us-east-1` and `us-west-2`, so you must whitelist both roles exactly as shown above. Do not modify them. Their permissions are intentionally limited to pulling images only, so Runpod cannot access any other part of your AWS account.
`ecr:GetAuthorizationToken` is an account-level AWS action, not repository-scoped. If your organization uses Service Control Policies (SCPs), confirm this action is not blocked at the org level.
### Alternative: Configure via AWS CLI
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
aws ecr set-repository-policy \
--repository-name YOUR_REPOSITORY_NAME \
--policy-text '{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "AllowRunpodPull",
"Effect": "Allow",
"Principal": "*",
"Action": [
"ecr:GetAuthorizationToken",
"ecr:BatchCheckLayerAvailability",
"ecr:GetDownloadUrlForLayer",
"ecr:BatchGetImage"
],
"Condition": {
"StringEquals": {
"aws:PrincipalArn": [
"arn:aws:iam::550005742258:role/prod-us-east-1-deployment-role",
"arn:aws:iam::550005742258:role/prod-us-west-2-deployment-role"
]
}
}
}
]
}'
```
Replace `YOUR_REPOSITORY_NAME` with the name of your ECR repository.
## Step 3: Add your ECR credential to Runpod
1. Navigate to [Settings](https://console.runpod.io/user/settings) in the Runpod console.
2. Scroll down to **Container Registry Authentication** and click **Add Credential**.
3. In the **Create new registry credential** modal, switch to the **AWS ECR** tab.
4. Enter a **Credential name** (for example, `my-ecr-repo`).
5. Enter the **ECR Image URI** in the format `ACCOUNT_ID.dkr.ecr.REGION.amazonaws.com/REPOSITORY_NAME:TAG`. You must include the tag (for example, `:latest` or `:v1.0.0`). Runpod requires the tag to store the delegation.
6. Click **Save**.
A registered ECR credential is shared across all members of your team or organization. Members do not need to add it individually.
Credential creation fails if the ECR repository policy from Step 2 is not correctly configured. Runpod validates the delegation at registration time. If the policy is missing, the credential is rejected. Set up the policy before proceeding.
## Step 4: Deploy a with your private image
You can deploy using a template or directly from the deploy page.
### Option A: Deploy directly
1. Navigate to [Pods](https://console.runpod.io/pods) and select **Deploy**.
2. Choose your GPU configuration.
3. Under **Container Image**, enter your full ECR image URI (for example, `123456789012.dkr.ecr.us-east-2.amazonaws.com/my-app:latest`).
4. Configure any additional settings such as environment variables or exposed ports.
5. Click **Deploy**.
### Option B: Deploy via a template
1. Navigate to [Templates](https://console.runpod.io/user/templates).
2. Create a new template or update an existing one.
3. Set the **Container Image** to your ECR image URI.
4. Save the template, then deploy a from it.
Runpod uses the registered credential to authenticate and pull your private image. To confirm the pull succeeded, open your 's logs. You should see layer download entries from your ECR registry.
Only the repository portion of the URI needs to match the registered credential. The tag is not checked at pull time, so any tag from a registered repository works without re-registering.
You've configured cross-account ECR delegation and deployed a using a private container image.
## Troubleshooting
**Credential creation fails:**
* Confirm the repository policy JSON is saved: ECR console → your repository → **Permissions** tab.
* Check that both `aws:PrincipalArn` values match exactly, with no trailing spaces or modified characters.
* Verify your IAM user has `ecr:SetRepositoryPolicy` permission.
** fails to start with an image pull error:**
* Confirm the ECR repository in the or template matches the repository registered in Step 3. The tag is not checked at pull time. Any tag from a registered repository works.
* Check that the image tag exists in your ECR repository.
* Verify the repository policy is applied to the correct repository.
## Next steps
* Learn how to [create custom templates](/pods/templates/create-custom-template) from your container images.
* Explore [environment variables](/pods/templates/environment-variables) for configuring your containers.
* Set up [network volumes](/storage/network-volumes) for persistent storage.
# Build a text-to-video pipeline
Source: https://docs.runpod.io/tutorials/public-endpoints/text-to-video-pipeline
Chain multiple Public Endpoints to generate videos from text prompts using Python. Follow the implementation steps in this Runpod tutorial.
This tutorial shows you how to build a complete text-to-video pipeline by chaining three Runpod [Public Endpoints](/public-endpoints/overview) together. You'll take a simple idea and transform it into an animated video, all with a single Python script.
## What you'll build
The pipeline takes a basic prompt like "a cat wearing sunglasses" and:
1. Uses [Qwen3 32B](/public-endpoints/models/qwen3-32b) to enhance the prompt into a detailed image description.
2. Uses [Flux Schnell](/public-endpoints/models/flux-schnell) to generate an image from the enhanced prompt.
3. Uses [WAN 2.5](/public-endpoints/models/wan-2-5) to animate the image into a 5-second video.
```mermaid theme={"theme":{"light":"github-light","dark":"github-dark"}}
%%{init: {'theme':'base', 'themeVariables': { 'primaryColor':'#9289FE','primaryTextColor':'#fff','primaryBorderColor':'#9289FE','lineColor':'#5F4CFE','secondaryColor':'#AE6DFF','tertiaryColor':'#FCB1FF','edgeLabelBackground':'#5F4CFE', 'fontSize':'15px','fontFamily':'font-inter'}}}%%
flowchart TD
A([Text prompt]) --> B[Qwen3 32B]
B --> C([Enhanced prompt])
C --> D[Flux Schnell]
D --> E([Generated image])
E --> F[WAN 2.5]
F --> G([Animated video])
style A fill:#5F4CFE,stroke:#5F4CFE,color:#FFFFFF,stroke-width:2px
style B fill:#4D38F5,stroke:#4D38F5,color:#FFFFFF,stroke-width:2px
style C fill:#9289FE,stroke:#9289FE,color:#FFFFFF,stroke-width:2px
style D fill:#4D38F5,stroke:#4D38F5,color:#FFFFFF,stroke-width:2px
style E fill:#9289FE,stroke:#9289FE,color:#FFFFFF,stroke-width:2px
style F fill:#4D38F5,stroke:#4D38F5,color:#FFFFFF,stroke-width:2px
style G fill:#22C55E,stroke:#22C55E,color:#000000,stroke-width:2px
linkStyle default stroke:#5F4CFE,stroke-width:2px
```
## Requirements
Before you begin, you'll need:
* A [Runpod account](/accounts-billing/manage-accounts) with at least \$1 in credits.
* A [Runpod API key](/get-started/api-keys).
* Python 3.8 or later installed on your local machine.
## Estimated cost
Public Endpoints pricing is based on actual usage. Here's an estimated cost for running the pipeline based on the models used:
| Step | Model | Cost |
| ------------------ | ------------ | ------------ |
| Prompt enhancement | Qwen3 32B | \~\$0.01 |
| Image generation | Flux Schnell | \~\$0.003 |
| Video generation | WAN 2.5 | \~\$0.25 |
| **Total** | | **\~\$0.26** |
You won't be charged for failed generations.
## Step 1: Set up your project
Create a new directory for your project with a virtual environment and set your API key. Replace `YOUR_API_KEY` with your actual API key.
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
mkdir text-to-video && cd text-to-video
python3 -m venv venv
source venv/bin/activate # On Windows: venv\Scripts\activate
pip install requests
export RUNPOD_API_KEY="YOUR_API_KEY"
```
Create a new file called `pipeline.py` and add the following imports and configuration:
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
import requests
import time
import os
import re
# Configuration
API_KEY = os.environ.get("RUNPOD_API_KEY")
BASE_URL = "https://api.runpod.ai/v2"
# Endpoint IDs
QWEN_ENDPOINT = "qwen3-32b-awq"
FLUX_ENDPOINT = "black-forest-labs-flux-1-schnell"
WAN_ENDPOINT = "wan-2-5"
def get_headers():
return {
"Authorization": f"Bearer {API_KEY}",
"Content-Type": "application/json",
}
```
## Step 2: Enhance the prompt with Qwen3 32B
The first step uses Qwen3 32B to transform a simple idea into a detailed, image-generation-optimized prompt. This significantly improves the quality of the generated image.
Add the following function to your script:
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
def enhance_prompt(simple_prompt):
"""Use Qwen3 32B to enhance a simple prompt into a detailed image description."""
print(f"Enhancing prompt: {simple_prompt}")
response = requests.post(
f"{BASE_URL}/{QWEN_ENDPOINT}/openai/v1/chat/completions",
headers=get_headers(),
json={
"model": "Qwen/Qwen3-32B-AWQ",
"messages": [
{
"role": "system",
"content": "You are an expert at writing prompts for AI image generation. "
"Transform the user's simple idea into a detailed, vivid image description. "
"Include details about lighting, style, composition, and atmosphere. "
"Keep the description under 100 words. Output only the enhanced prompt, "
"nothing else. Do not include any thinking or explanation.",
},
{"role": "user", "content": simple_prompt},
],
"max_tokens": 200,
"temperature": 0.7,
},
)
result = response.json()
enhanced = result["choices"][0]["message"]["content"].strip()
# Remove any ... blocks (some models include reasoning)
enhanced = re.sub(r".*?", "", enhanced, flags=re.DOTALL).strip()
# Also handle unclosed tags
enhanced = re.sub(r".*", "", enhanced, flags=re.DOTALL).strip()
print(f"Enhanced prompt: {enhanced}")
return enhanced
```
This function:
* Sends the simple prompt to Qwen3 32B using the OpenAI-compatible API
* Uses a system prompt that instructs the model to act as an image prompt expert
* Strips any reasoning tags from the output
* Returns the enhanced, detailed prompt
## Step 3: Add a polling helper
Image and video generation can take time, so you'll use asynchronous requests with polling. Add this helper function:
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
def poll_for_completion(endpoint, job_id, timeout=300):
"""Poll an async job until completion."""
start_time = time.time()
while time.time() - start_time < timeout:
status_response = requests.get(
f"{BASE_URL}/{endpoint}/status/{job_id}",
headers=get_headers(),
)
status = status_response.json()
if status["status"] == "COMPLETED":
return status
elif status["status"] == "FAILED":
raise Exception(f"Job failed: {status}")
else:
print(f" Status: {status['status']}, waiting...")
time.sleep(5)
raise Exception(f"Job timed out after {timeout} seconds")
```
## Step 4: Generate an image with Flux Schnell
Next, use Flux Schnell to generate an image from the enhanced prompt. Flux Schnell is optimized for speed.
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
def generate_image(prompt):
"""Use Flux Schnell to generate an image from the prompt."""
print("Generating image with Flux Schnell...")
# Submit async job
response = requests.post(
f"{BASE_URL}/{FLUX_ENDPOINT}/run",
headers=get_headers(),
json={
"input": {
"prompt": prompt,
"width": 1024,
"height": 1024,
"num_inference_steps": 4,
}
},
)
result = response.json()
job_id = result["id"]
print(f" Job submitted: {job_id}")
# Poll for completion
status = poll_for_completion(FLUX_ENDPOINT, job_id)
image_url = status["output"]["image_url"]
print(f" Image URL: {image_url}")
return image_url
```
This function:
* Submits an asynchronous job to Flux Schnell
* Polls until the job completes
* Uses a 1024x1024 resolution (optimal for video generation)
* Returns the URL of the generated image
## Step 5: Animate the image with WAN 2.5
Now animate the static image into a video using WAN 2.5.
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
def generate_video(image_url, prompt):
"""Use WAN 2.5 to animate the image into a video."""
print("Generating video with WAN 2.5...")
# Submit the job
response = requests.post(
f"{BASE_URL}/{WAN_ENDPOINT}/run",
headers=get_headers(),
json={
"input": {
"image": image_url,
"prompt": prompt,
"duration": 5,
"size": "1280*720",
}
},
)
result = response.json()
job_id = result["id"]
print(f" Job submitted: {job_id}")
# Poll for completion (video takes longer, so we'll increase the timeout)
status = poll_for_completion(WAN_ENDPOINT, job_id, timeout=600)
video_url = status["output"]["result"]
print(f" Video URL: {video_url}")
return video_url
```
This function:
* Submits an asynchronous job to WAN 2.5
* Uses the polling helper with a longer timeout (video generation takes longer)
* Returns the URL of the generated video
## Step 6: Download the output
Add a helper function to download the final video:
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
def download_file(url, filename):
"""Download a file from a URL."""
print(f"Downloading to {filename}...")
response = requests.get(url)
with open(filename, "wb") as f:
f.write(response.content)
print(f"Saved: {filename}")
```
## Step 7: Put it all together
Add the main function that chains all the steps together:
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
def main():
# Your simple prompt
simple_prompt = "a cat wearing sunglasses"
# Step 1: Enhance the prompt
enhanced_prompt = enhance_prompt(simple_prompt)
# Step 2: Generate the image
image_url = generate_image(enhanced_prompt)
# Step 3: Generate the video
video_url = generate_video(image_url, enhanced_prompt)
# Step 4: Download the results
download_file(image_url, "output_image.png")
download_file(video_url, "output_video.mp4")
print("\nPipeline complete!")
print(f"Original prompt: {simple_prompt}")
print(f"Enhanced prompt: {enhanced_prompt}")
print(f"Image: output_image.png")
print(f"Video: output_video.mp4")
if __name__ == "__main__":
if not API_KEY:
print("Error: Set RUNPOD_API_KEY environment variable")
exit(1)
main()
```
## Full code
Expand the section below to see the full `pipeline.py` code:
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
import requests
import time
import os
import re
# Configuration
API_KEY = os.environ.get("RUNPOD_API_KEY")
BASE_URL = "https://api.runpod.ai/v2"
# Endpoint IDs
QWEN_ENDPOINT = "qwen3-32b-awq"
FLUX_ENDPOINT = "black-forest-labs-flux-1-schnell"
WAN_ENDPOINT = "wan-2-5"
def get_headers():
return {
"Authorization": f"Bearer {API_KEY}",
"Content-Type": "application/json",
}
def poll_for_completion(endpoint, job_id, timeout=300):
"""Poll an async job until completion."""
start_time = time.time()
while time.time() - start_time < timeout:
status_response = requests.get(
f"{BASE_URL}/{endpoint}/status/{job_id}",
headers=get_headers(),
)
status = status_response.json()
if status["status"] == "COMPLETED":
return status
elif status["status"] == "FAILED":
raise Exception(f"Job failed: {status}")
else:
print(f" Status: {status['status']}, waiting...")
time.sleep(5)
raise Exception(f"Job timed out after {timeout} seconds")
def enhance_prompt(simple_prompt):
"""Use Qwen3 32B to enhance a simple prompt into a detailed image description."""
print(f"Enhancing prompt: {simple_prompt}")
response = requests.post(
f"{BASE_URL}/{QWEN_ENDPOINT}/openai/v1/chat/completions",
headers=get_headers(),
json={
"model": "Qwen/Qwen3-32B-AWQ",
"messages": [
{
"role": "system",
"content": "You are an expert at writing prompts for AI image generation. "
"Transform the user's simple idea into a detailed, vivid image description. "
"Include details about lighting, style, composition, and atmosphere. "
"Keep the description under 100 words. Output only the enhanced prompt, "
"nothing else. Do not include any thinking or explanation.",
},
{"role": "user", "content": simple_prompt},
],
"max_tokens": 200,
"temperature": 0.7,
},
)
result = response.json()
enhanced = result["choices"][0]["message"]["content"].strip()
# Remove any ... blocks (some models include reasoning)
enhanced = re.sub(r".*?", "", enhanced, flags=re.DOTALL).strip()
# Also handle unclosed tags
enhanced = re.sub(r".*", "", enhanced, flags=re.DOTALL).strip()
print(f"Enhanced prompt: {enhanced}")
return enhanced
def generate_image(prompt):
"""Use Flux Schnell to generate an image from the prompt."""
print("Generating image with Flux Schnell...")
# Submit async job
response = requests.post(
f"{BASE_URL}/{FLUX_ENDPOINT}/run",
headers=get_headers(),
json={
"input": {
"prompt": prompt,
"width": 1024,
"height": 1024,
"num_inference_steps": 4,
}
},
)
result = response.json()
job_id = result["id"]
print(f" Job submitted: {job_id}")
# Poll for completion
status = poll_for_completion(FLUX_ENDPOINT, job_id)
image_url = status["output"]["image_url"]
print(f" Image URL: {image_url}")
return image_url
def generate_video(image_url, prompt):
"""Use WAN 2.5 to animate the image into a video."""
print("Generating video with WAN 2.5...")
# Submit the job
response = requests.post(
f"{BASE_URL}/{WAN_ENDPOINT}/run",
headers=get_headers(),
json={
"input": {
"image": image_url,
"prompt": prompt,
"duration": 5,
"size": "1280*720",
}
},
)
result = response.json()
job_id = result["id"]
print(f" Job submitted: {job_id}")
# Poll for completion (video takes longer, so increase timeout)
status = poll_for_completion(WAN_ENDPOINT, job_id, timeout=600)
video_url = status["output"]["result"]
print(f" Video URL: {video_url}")
return video_url
def download_file(url, filename):
"""Download a file from a URL."""
print(f"Downloading to {filename}...")
response = requests.get(url)
with open(filename, "wb") as f:
f.write(response.content)
print(f"Saved: {filename}")
def main():
# Your simple prompt
simple_prompt = "a cat wearing sunglasses"
# Step 1: Enhance the prompt
enhanced_prompt = enhance_prompt(simple_prompt)
# Step 2: Generate the image
image_url = generate_image(enhanced_prompt)
# Step 3: Generate the video
video_url = generate_video(image_url, enhanced_prompt)
# Step 4: Download the results
download_file(image_url, "output_image.png")
download_file(video_url, "output_video.mp4")
print("\nPipeline complete!")
print(f"Original prompt: {simple_prompt}")
print(f"Enhanced prompt: {enhanced_prompt}")
print(f"Image: output_image.png")
print(f"Video: output_video.mp4")
if __name__ == "__main__":
if not API_KEY:
print("Error: Set RUNPOD_API_KEY environment variable")
exit(1)
main()
```
## Run the pipeline
Run the script:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
python pipeline.py
```
The script will output progress as it runs:
```text theme={"theme":{"light":"github-light","dark":"github-dark"}}
Enhancing prompt: a cat wearing sunglasses
Enhanced prompt: A fluffy orange tabby cat sits regally on a velvet purple cushion...
Generating image with Flux Schnell...
Job submitted: abc123-def456...
Status: IN_PROGRESS, waiting...
Image URL: https://image.runpod.ai/...
Generating video with WAN 2.5...
Job submitted: xyz789-uvw012...
Status: IN_PROGRESS, waiting...
Status: IN_PROGRESS, waiting...
Video URL: https://video.runpod.ai/...
Downloading to output_image.png...
Saved: output_image.png
Downloading to output_video.mp4...
Saved: output_video.mp4
Pipeline complete!
```
Output URLs expire after 7 days. The script downloads files immediately to avoid losing them.
## Next steps
Now that you have a working pipeline, you can extend it in several ways:
* **Try different prompts**: Experiment with landscapes, characters, or abstract concepts.
* **Adjust video settings**: Change the duration or resolution in the WAN 2.5 request.
* **Use different models**: Swap Flux Schnell for [Flux Dev](/public-endpoints/models/flux-dev) for higher quality (but slower) generation.
* **Add error handling**: Implement retries for transient failures.
* **Build a web interface**: Wrap the pipeline in a Flask or FastAPI application.
* **Batch processing**: Process multiple prompts in parallel.
## Related resources
* [Public Endpoints overview](/public-endpoints/overview)
* [Qwen3 32B reference](/public-endpoints/models/qwen3-32b)
* [Flux Schnell reference](/public-endpoints/models/flux-schnell)
* [WAN 2.5 reference](/public-endpoints/models/wan-2-5)
* [Vercel AI SDK](/public-endpoints/ai-sdk) for TypeScript projects
# Deploy ComfyUI on Serverless
Source: https://docs.runpod.io/tutorials/serverless/comfyui
Learn how to deploy a Serverless endpoint running ComfyUI from the Runpod Hub and use it to generate images with FLUX Dev.
In this tutorial, you will learn how to deploy a running [ComfyUI](https://github.com/Comfy-Org/ComfyUI) on Runpod, submit image generation jobs using workflow JSON, monitor their progress, and decode the resulting images.
[Runpod's Serverless platform](/serverless/overview) allows you to run AI/ML models in the cloud without managing infrastructure, automatically scaling resources as needed. ComfyUI is a powerful node-based interface for Stable Diffusion that provides fine-grained control over the image generation process through customizable workflows.
## Requirements
Before starting this tutorial you'll need:
* A Runpod account with available credits.
* A Runpod API key (available in your user settings).
* Basic familiarity with command-line tools like `curl`.
* Python installed on your system (for the image decoding step).
* The `jq` command-line JSON processor (optional but recommended).
* Basic understanding of JSON structure for workflow configuration.
## Step 1: Deploy a ComfyUI Serverless endpoint using the Runpod Hub
The ComfyUI Hub listing comes with the FLUX.1-dev-fp8 model pre-installed and works only with this model when deployed from the Hub.
If you want to use a different model, you can also [deploy the endpoint](https://github.com/runpod-workers/worker-comfyui/blob/main/docs/deployment.md) using one of these pre-defined Docker images:
* `runpod/worker-comfyui:-base` - Clean ComfyUI install with no models.
* `runpod/worker-comfyui:-flux1-schnell` - FLUX.1 schnell model.
* `runpod/worker-comfyui:-flux1-dev` - FLUX.1 dev model.
* `runpod/worker-comfyui:-sdxl` - Stable Diffusion XL model.
* `runpod/worker-comfyui:-sd3` - Stable Diffusion 3 medium model.
Replace `` with the latest release version from GitHub Releases.
If you need a model that's not listed here, or have your own LoRA, or need custom nodes, you can use this [customization guide](https://github.com/runpod-workers/worker-comfyui/blob/main/docs/customization.md) to create your own custom .
1. Navigate to the [ComfyUI Hub listing](https://console.runpod.io/hub/runpod-workers/worker-comfyui) in the Runpod web interface.
2. Click **Deploy \[VERSION\_NUMBER]**, then click **Next** and then **Create Endpoint** to confirm. This creates a fully configured endpoint with the FLUX.1-dev-fp8 model pre-installed and appropriate GPU settings for running ComfyUI workflows.
3. On the endpoint page, make a note of the **Endpoint ID**. You'll need this value to submit jobs and retrieve results. You can find your endpoint by navigating to **Resources > Serverless** in the left-hand navigation and clicking the relevant card to view the endpoint detail page. The Endpoint ID is displayed below the Quick Start section, or you can find it at the end of the page URL.
Once deployed, your endpoint will be assigned a unique ID (e.g. `32vgrms732dkwi`). Your endpoint URL will follow this pattern: `https://api.runpod.ai/v2/ENDPOINT_ID/run` for asynchronous requests.
## Step 2: Prepare your ComfyUI workflow
ComfyUI uses workflow JSON to define the image generation process. The workflow contains nodes that represent different steps in the generation pipeline, such as loading models, encoding prompts, and saving images.
On your local machine, create a file called `comfyui_workflow.json` with the following FLUX.1-dev-fp8 workflow:
```json theme={"theme":{"light":"github-light","dark":"github-dark"}}
{
"input": {
"workflow": {
"6": {
"inputs": {
"text": "a whimsical and intricate treehouse nestled in the branches of a giant, ancient cherry blossom tree, surrounded by a field of glowing flowers. A gentle stream flows nearby. Fantasy art, cinematic, volumetric lighting, epic scale.",
"clip": ["30", 1]
},
"class_type": "CLIPTextEncode",
"_meta": {
"title": "CLIP Text Encode (Positive Prompt)"
}
},
"8": {
"inputs": {
"samples": ["31", 0],
"vae": ["30", 2]
},
"class_type": "VAEDecode",
"_meta": {
"title": "VAE Decode"
}
},
"9": {
"inputs": {
"filename_prefix": "ComfyUI",
"images": ["8", 0]
},
"class_type": "SaveImage",
"_meta": {
"title": "Save Image"
}
},
"27": {
"inputs": {
"width": 512,
"height": 512,
"batch_size": 1
},
"class_type": "EmptySD3LatentImage",
"_meta": {
"title": "EmptySD3LatentImage"
}
},
"30": {
"inputs": {
"ckpt_name": "flux1-dev-fp8.safetensors"
},
"class_type": "CheckpointLoaderSimple",
"_meta": {
"title": "Load Checkpoint"
}
},
"31": {
"inputs": {
"seed": 243057879077961,
"steps": 10,
"cfg": 1,
"sampler_name": "euler",
"scheduler": "simple",
"denoise": 1,
"model": ["30", 0],
"positive": ["35", 0],
"negative": ["33", 0],
"latent_image": ["27", 0]
},
"class_type": "KSampler",
"_meta": {
"title": "KSampler"
}
},
"33": {
"inputs": {
"text": "",
"clip": ["30", 1]
},
"class_type": "CLIPTextEncode",
"_meta": {
"title": "CLIP Text Encode (Negative Prompt)"
}
},
"35": {
"inputs": {
"guidance": 3.5,
"conditioning": ["6", 0]
},
"class_type": "FluxGuidance",
"_meta": {
"title": "FluxGuidance"
}
},
"38": {
"inputs": {
"images": ["8", 0]
},
"class_type": "PreviewImage",
"_meta": {
"title": "Preview Image"
}
},
"40": {
"inputs": {
"filename_prefix": "ComfyUI",
"images": ["8", 0]
},
"class_type": "SaveImage",
"_meta": {
"title": "Save Image"
}
}
}
}
}
```
This workflow defines a complete image generation pipeline using the FLUX.1-dev-fp8 model. Key components include:
* **Node 6**: Encodes the positive text prompt using CLIP.
* **Node 30**: Loads the FLUX.1-dev-fp8 checkpoint.
* **Node 31**: Performs the sampling process with specified parameters.
* **Node 8**: Decodes the latent image to a viewable format.
* **Node 9/40**: Saves the generated image.
You can customize the prompt by modifying the `text` field in node 6, or adjust generation parameters like `steps`, `cfg`, `width`, and `height` in their respective nodes.
## Step 3: Submit your first job
Use the `/run` endpoint to submit an asynchronous job that will generate an image based on your ComfyUI workflow.
Replace `ENDPOINT_ID` with your actual endpoint ID and `YOUR_API_KEY` with your Runpod API key in the following command:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
curl -X POST https://api.runpod.ai/v2/ENDPOINT_ID/run \
-H 'Content-Type: application/json' \
-H 'Authorization: Bearer YOUR_API_KEY' \
-d @comfyui_workflow.json
```
The API will respond immediately with a job ID and status. You'll receive a response similar to this:
```json theme={"theme":{"light":"github-light","dark":"github-dark"}}
{
"id": "c80ffee4-f315-4e25-a146-0f3d98cf024b",
"status": "IN_QUEUE"
}
```
The job ID is crucial for tracking your request's progress. Save this ID as you'll need it to check the status and retrieve results.
## Step 4: Monitor job progress
Check your job's status using the `/status` endpoint with the job ID you received in the previous step.
Use the following command to check your job's progress, replacing the placeholders (`ENDPOINT_ID`, `JOB_ID`, and `YOUR_API_KEY`) with your actual values:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
curl https://api.runpod.ai/v2/ENDPOINT_ID/status/JOB_ID \
-H 'Content-Type: application/json' \
-H 'Authorization: Bearer YOUR_API_KEY'
```
While your job is processing, you'll receive a response indicating the current status:
```json theme={"theme":{"light":"github-light","dark":"github-dark"}}
{
"delayTime": 2188,
"id": "c80ffee4-f315-4e25-a146-0f3d98cf024b",
"input": {
"workflow": {
"6": {
"inputs": {
"text": "masterpiece, best quality, a whimsical and intricate treehouse...",
"clip": ["30", 1]
}
}
}
},
"status": "IN_PROGRESS"
}
```
The `delayTime` field shows how long the job waited in the queue before processing began, measured in milliseconds.
## Step 5: Retrieve completed results
Continue polling the status endpoint until the status changes to `COMPLETED`. Once your job completes, the status endpoint will return the generated image data encoded in base64 format.
When your job finishes successfully, you'll receive a response containing the output:
```json theme={"theme":{"light":"github-light","dark":"github-dark"}}
{
"delayTime": 2188,
"executionTime": 2297,
"id": "sync-c0cd1eb2-068f-4ecf-a99a-55770fc77391-e1",
"output": {
"message": "data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAABAAAAAQACAIAAADwf7zU...",
"status": "success"
},
"status": "COMPLETED"
}
```
The `executionTime` field shows how long the actual image generation took, while `delayTime` indicates the initial queue wait time. Both values are in milliseconds.
To save the complete response for processing, use this command:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
curl https://api.runpod.ai/v2/ENDPOINT_ID/status/JOB_ID \
-H 'Content-Type: application/json' \
-H 'Authorization: Bearer YOUR_API_KEY' | jq . > comfyui_output.json
```
You have up to 30 minutes to retrieve your results via the status endpoint, after which results will be automatically deleted for security.
## Step 6: Decode and save your image
Now we'll convert the base64-encoded image data into a viewable image file using Python.
Create a Python script called `decode_comfyui_image.py` to decode the base64 image data from your JSON response:
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
import base64
from PIL import Image
import io
import os
import json
def decode_comfyui_json_and_save_image(json_filepath, output_filename="comfyui_generated_image.png"):
"""
Reads a ComfyUI JSON response file, extracts the base64 image string, decodes it, and saves it as an image file.
Args:
json_filepath (str): The path to the input JSON file.
output_filename (str): The name for the output image file.
"""
try:
with open(json_filepath, 'r') as f:
data = json.load(f)
# Extract the base64 string from the ComfyUI response structure
base64_url = data['output']['images'][0]['data']
if not base64_url:
print("Error: 'images[0]' not found in the JSON output.")
return
# Remove data URI prefix if present
if "," in base64_url:
_, encoded_data = base64_url.split(",", 1)
else:
encoded_data = base64_url
# Decode base64 to bytes
image_data = base64.b64decode(encoded_data)
image_stream = io.BytesIO(image_data)
image = Image.open(image_stream)
image.save(output_filename)
print(f"ComfyUI image successfully saved as '{output_filename}'")
print(f"Image path: {os.path.abspath(output_filename)}")
except FileNotFoundError:
print(f"Error: The file '{json_filepath}' was not found.")
except json.JSONDecodeError:
print(f"Error: Could not decode JSON from the file '{json_filepath}'.")
except base64.binascii.Error as e:
print(f"Error decoding base64 string: {e}")
print("Please ensure the input is a valid base64 string.")
except Exception as e:
print(f"An unexpected error occurred: {e}")
# Process the comfyui_output.json file
decode_comfyui_json_and_save_image("comfyui_output.json", "comfyui_generated_image.png")
```
Run the script to decode the image data and save it as a PNG file:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
python decode_comfyui_image.py
```
You should see the following output:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
ComfyUI image successfully saved as 'comfyui_generated_image.png'
Image path: /Users/path/to/your/project/comfyui_generated_image.png
```
Congratulations! You've successfully used Runpod's Serverless platform to generate an AI image using ComfyUI with the FLUX.1-dev-fp8 model. You now understand the complete workflow for submitting ComfyUI jobs, monitoring their progress, and retrieving results.
## Understanding ComfyUI workflows
ComfyUI workflows are JSON structures that define the image generation pipeline through interconnected nodes. Each node has:
* **Inputs**: Parameters and connections to other nodes.
* **Class type**: The operation this node performs.
* **Meta information**: Human-readable titles and descriptions.
You can create custom workflows by modifying node parameters or [opening the ComfyUI interface in a ](/tutorials/pods/comfyui) and exporting the workflow to JSON.
To learn more about creating your own ComfyUI workflows, see the [ComfyUI documentation](https://docs.comfy.org/basic-concepts/workflow).
## Next steps
Now that you've learned how to generate images with ComfyUI on Serverless, you can explore these resources:
* [runpod-workers/worker-comfyui](https://github.com/runpod-workers/worker-comfyui): Advanced configuration options for the ComfyUI Serverless worker.
* [ComfyUI-to-API](/community-solutions/comfyui-to-api/overview): A community tool that analyzes your ComfyUI workflows and automatically generates a deployment-ready GitHub repository with Dockerfile and dependencies.
# Integrate Serverless with a web application
Source: https://docs.runpod.io/tutorials/serverless/generate-sdxl-turbo
Deploy an image generation endpoint from the Hub and integrate it into a web application. Follow the implementation steps in this Runpod tutorial.
In this tutorial, you'll deploy a pre-built SDXL Turbo from the and integrate it into a web application. You'll build a simple frontend that sends prompts to your endpoint and displays the generated images.
By the end, you'll know how to deploy endpoints from the Hub and integrate them into your applications using standard HTTP requests.
## Requirements
* A [Runpod account](/accounts-billing/manage-accounts) with credits.
* A [Runpod API key](/get-started/api-keys).
## Step 1: Deploy an endpoint from the Hub
The Runpod Hub provides pre-built workers you can deploy with a few clicks. For this tutorial, you'll deploy the SDXL Turbo worker, which generates images from text prompts.
1. Navigate to the [Runpod Hub](https://www.console.runpod.io/hub) in the console.
2. Search for "SDXL-turbo" or browse the image generation category.
3. Select the SDXL-turbo worker to view its details and hardware requirements.
4. Click **Deploy** in the top-right corner of the page.
5. Click **Create Endpoint**.
Your endpoint will be ready within a few minutes. You can monitor the deployment status on your [Serverless endpoints page](https://www.console.runpod.io/serverless).
## Step 2: Find your endpoint ID and API key
Now that your endpoint is deployed, you'll need your endpoint ID and API key to make requests.
### Find your endpoint ID
1. Go to your [Serverless endpoints page](https://www.console.runpod.io/serverless).
2. Click on your SDXL Turbo endpoint.
3. Copy the **Endpoint ID** shown at the top of the page.
### Find your API key
1. Navigate to **Settings** > **API Keys** in the console, or go directly to the [API Keys page](https://console.runpod.io/user/settings).
2. Copy an existing API key, or create a new one with read and write permissions.
Store your API key securely. Never commit it to version control or expose it in client-side code that's publicly accessible.
## Step 3: Send a test request
Test your endpoint with a simple API request before integrating it into an application. The endpoint accepts a JSON payload with an `input` object containing your prompt.
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
curl -X POST "https://api.runpod.ai/v2/${YOUR_ENDPOINT_ID}/runsync" \
-H "accept: application/json" \
-H "content-type: application/json" \
-H "authorization: Bearer ${YOUR_API_KEY}" \
-d '{
"input": {
"prompt": "A futuristic city skyline at sunset"
}
}'
```
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
import requests
endpoint_id = "YOUR_ENDPOINT_ID"
api_key = "YOUR_API_KEY"
response = requests.post(
f"https://api.runpod.ai/v2/{endpoint_id}/runsync",
headers={
"accept": "application/json",
"content-type": "application/json",
"authorization": f"Bearer {api_key}",
},
json={
"input": {
"prompt": "A futuristic city skyline at sunset",
}
},
)
print(response.json())
```
```json theme={"theme":{"light":"github-light","dark":"github-dark"}}
{
"delayTime": 168,
"executionTime": 2126,
"id": "sync-fa542d19-92b2-47d0-8e58-c01878f0365d-u1",
"output": {
"generation_time": 0.72,
"images": [
{
"image": "BASE_64_ENCODED_IMAGE"
}
]
},
"status": "COMPLETED"
}
```
Replace the following placeholders:
| Placeholder | Description |
| ------------------ | ----------------------------- |
| `YOUR_ENDPOINT_ID` | Your endpoint ID from step 2. |
| `YOUR_API_KEY` | Your Runpod API key. |
The generated image is returned as a base64-encoded PNG string in `output.images[0].image`. For more information about request parameters and response formats, see [Send requests](/serverless/endpoints/send-requests).
SDXL Turbo uses adversarial diffusion distillation to generate images in just a few steps. The worker is pre-configured with optimal settings, so you only need to provide the `prompt` parameter, but you can also provide other parameters to control the generation process.
| Parameter | Type | Required | Default | Description |
| --------------------- | ------- | -------- | ------- | ---------------------------------------------- |
| `prompt` | string | Yes | - | The prompt to generate an image from. |
| `negative_prompt` | string | No | - | The negative prompt to exclude from the image. |
| `width` | integer | No | 1024 | The width of the image in pixels. |
| `height` | integer | No | 1024 | The height of the image in pixels. |
| `num_inference_steps` | integer | No | 25 | The number of denoising steps. |
## Step 4: Integrate with a web application
Now that you've confirmed your endpoint is working, you can integrate it into a web application. This example creates a simple image generator with an input field for prompts and a button to generate images. While this tutorial uses JavaScript, you can make requests to your endpoint using any programming language or framework.
Create two files in the same directory:
The HTML file sets up a user interface with an input field for prompts and a button to generate images. Clicking the button calls the `generateImage` function in `script.js`.
```html index.html theme={"theme":{"light":"github-light","dark":"github-dark"}}
Runpod AI Image Generator
AI Image Generator
Enter a prompt to generate an image using SDXL Turbo.
Generating image...
```
The JavaScript file handles API communication. The `generateImage` function reads the user's prompt, sends a POST request to your Runpod endpoint, and displays the generated image. The endpoint returns a base64-encoded PNG image in the `output.images` array, which is then converted to a displayable format.
```javascript script.js theme={"theme":{"light":"github-light","dark":"github-dark"}}
// Replace these with your actual values
const ENDPOINT_ID = "YOUR_ENDPOINT_ID";
const API_KEY = "YOUR_API_KEY";
async function generateImage() {
const prompt = document.getElementById("promptInput").value;
if (!prompt) {
alert("Please enter a prompt!");
return;
}
// Show loading indicator
document.getElementById("loading").style.display = "block";
document.getElementById("imageResult").innerHTML = "";
const options = {
method: "POST",
headers: {
"accept": "application/json",
"content-type": "application/json",
"authorization": `Bearer ${API_KEY}`,
},
body: JSON.stringify({
input: {
prompt: prompt,
},
}),
};
try {
const response = await fetch(
`https://api.runpod.ai/v2/${ENDPOINT_ID}/runsync`,
options
);
const data = await response.json();
// Hide loading indicator
document.getElementById("loading").style.display = "none";
if (data && data.output && data.output.images && data.output.images.length > 0) {
const imageBase64 = data.output.images[0].image;
const imageUrl = `data:image/png;base64,${imageBase64}`;
const img = document.createElement("img");
img.src = imageUrl;
img.alt = "Generated Image";
document.getElementById("imageResult").replaceChildren(img);
} else if (data && data.error) {
alert(`Error: ${data.error}`);
} else {
alert("Failed to generate image. Check the console for details.");
console.error("Response:", data);
}
} catch (error) {
document.getElementById("loading").style.display = "none";
console.error("Error:", error);
alert("Error generating image. Check the console for details.");
}
}
```
Before running the application:
1. Replace `YOUR_ENDPOINT_ID` with your endpoint ID from step 2.
2. Replace `YOUR_API_KEY` with your Runpod API key.
This example includes the API key directly in client-side code for simplicity. In production, you should make API calls from a backend server to keep your API key secure.
## Step 5: Run your application
You can run the application using a local server or by opening the HTML file directly in a browser.
Run the following command in the directory containing your files:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
python -m http.server 8000
```
Then open `http://localhost:8000` in your browser.
1. Navigate to the folder containing `index.html`.
2. Double-click the file to open it in your default browser, or right-click and select **Open with** to choose a specific browser.
Enter a prompt and click **Generate Image** to see your AI-generated image.
The first request may take longer (30-60 seconds) due to as the endpoint loads the model into GPU memory. Subsequent requests complete in just a few seconds.
## Next steps
Now that you've integrated a Serverless endpoint into a web application, you can:
* [Explore other workers on the Hub](/hub/overview) for different AI capabilities.
* [Learn about asynchronous requests](/serverless/endpoints/send-requests) for handling long-running jobs.
* [Build your own custom worker](/serverless/workers/overview) to deploy your own models.
* [Use the Python SDK](/serverless/endpoints/send-requests) for more advanced integrations.
# Deploy Phi-3 using model caching
Source: https://docs.runpod.io/tutorials/serverless/model-caching-text
Learn how to create a custom Serverless endpoint that uses model caching to serve Phi-3 with reduced cost and cold start times.
You can download the finished code for this tutorial [on GitHub](https://github.com/runpod-workers/model-store-cache-example).
This tutorial demonstrates how to build a custom that leverages Runpod's feature to serve the Phi-3 language model. You'll learn how to create a handler function that locates and loads cached models in offline mode, which can significantly reduce costs and cold start times.
## Requirements
Before starting this tutorial, make sure:
* You have a [Runpod account](/accounts-billing/manage-accounts) with sufficient credits.
* You have a [Runpod API key](/get-started/api-keys).
* You have a [GitHub account](https://github.com/).
* Your Runpod account is [connected to GitHub](/serverless/workers/github-integration#authorize-runpod-with-github).
## Step 1: Create your handler function
Create a file named `handler.py` that processes inference requests using the cached model. This handler enforces offline mode to ensure it only uses cached models and includes a helper function to resolve the correct snapshot path.
```python handler.py theme={"theme":{"light":"github-light","dark":"github-dark"}}
import os
import runpod
from transformers import AutoTokenizer, AutoModelForCausalLM, pipeline
MODEL_ID = os.environ.get("MODEL_NAME", "microsoft/Phi-3-mini-4k-instruct")
HF_CACHE_ROOT = "/runpod-volume/huggingface-cache/hub"
# Force offline mode to use only cached models
os.environ["HF_HUB_OFFLINE"] = "1"
os.environ["TRANSFORMERS_OFFLINE"] = "1"
def resolve_snapshot_path(model_id: str) -> str:
"""
Resolve the local snapshot path for a cached model.
Args:
model_id: The model name from Hugging Face (e.g., 'microsoft/Phi-3-mini-4k-instruct')
Returns:
The full path to the cached model snapshot
"""
if "/" not in model_id:
raise ValueError(f"MODEL_ID '{model_id}' is not in 'org/name' format")
org, name = model_id.split("/", 1)
model_root = os.path.join(HF_CACHE_ROOT, f"models--{org}--{name}")
refs_main = os.path.join(model_root, "refs", "main")
snapshots_dir = os.path.join(model_root, "snapshots")
print(f"[ModelStore] MODEL_ID: {model_id}")
print(f"[ModelStore] Model root: {model_root}")
# Try to read the snapshot hash from refs/main
if os.path.isfile(refs_main):
with open(refs_main, "r") as f:
snapshot_hash = f.read().strip()
candidate = os.path.join(snapshots_dir, snapshot_hash)
if os.path.isdir(candidate):
print(f"[ModelStore] Using snapshot from refs/main: {candidate}")
return candidate
# Fall back to first available snapshot
if not os.path.isdir(snapshots_dir):
raise RuntimeError(f"[ModelStore] snapshots directory not found: {snapshots_dir}")
versions = [
d for d in os.listdir(snapshots_dir) if os.path.isdir(os.path.join(snapshots_dir, d))
]
if not versions:
raise RuntimeError(f"[ModelStore] No snapshot subdirectories found under {snapshots_dir}")
versions.sort()
chosen = os.path.join(snapshots_dir, versions[0])
print(f"[ModelStore] Using first available snapshot: {chosen}")
return chosen
# Resolve and load the model at startup
LOCAL_MODEL_PATH = resolve_snapshot_path(MODEL_ID)
print(f"[ModelStore] Resolved local model path: {LOCAL_MODEL_PATH}")
tokenizer = AutoTokenizer.from_pretrained(
LOCAL_MODEL_PATH,
trust_remote_code=False,
local_files_only=True,
)
model = AutoModelForCausalLM.from_pretrained(
LOCAL_MODEL_PATH,
trust_remote_code=False,
torch_dtype="auto",
device_map="auto",
local_files_only=True,
attn_implementation="eager",
)
text_gen = pipeline(
"text-generation",
model=model,
tokenizer=tokenizer,
)
print("[ModelStore] Model loaded from local snapshot")
def handler(job):
"""
Handler function that processes each inference request.
Args:
job: Runpod job object containing input data
Returns:
Dictionary with generated text or error information
"""
job_input = job.get("input", {}) or {}
prompt = job_input.get("prompt", "Hello!")
max_tokens = int(job_input.get("max_tokens", 256))
temperature = float(job_input.get("temperature", 0.7))
print(f"[Handler] Prompt: {prompt[:80]!r}")
print(f"[Handler] max_tokens={max_tokens}, temperature={temperature}")
try:
outputs = text_gen(
prompt,
max_new_tokens=max_tokens,
do_sample=True,
temperature=temperature,
)
generated = outputs[0]["generated_text"]
print(f"[Handler] Generated length: {len(generated)} chars")
return {
"status": "success",
"output": generated,
}
except Exception as e:
print(f"[Handler] Error during generation: {e}")
return {
"status": "error",
"error": str(e),
}
runpod.serverless.start({"handler": handler})
```
### Understanding the handler
If you want to learn more about each component of this handler function, expand the section below:
The handler is divided into four main sections: configuration, path resolution, model loading, and request handling. Let's examine each part:
#### Configuration and offline mode
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
MODEL_ID = os.environ.get("MODEL_NAME", "microsoft/Phi-3-mini-4k-instruct")
HF_CACHE_ROOT = "/runpod-volume/huggingface-cache/hub"
os.environ["HF_HUB_OFFLINE"] = "1"
os.environ["TRANSFORMERS_OFFLINE"] = "1"
```
The handler starts by defining two key paths: `MODEL_ID` specifies which Hugging Face model to load (configurable via environment variable, or using the "Model" [endpoint setting](/serverless/endpoints/endpoint-configurations)), and `HF_CACHE_ROOT` points to where Runpod stores cached models. When you enable model caching on your endpoint, Runpod automatically downloads the model to this location before your worker starts.
Setting `HF_HUB_OFFLINE` and `TRANSFORMERS_OFFLINE` to `"1"` forces the Hugging Face libraries into offline mode. This is a safety measure that prevents the worker from accidentally downloading models at runtime, which would defeat the purpose of caching. If the cached model isn't found, the worker fails immediately with a clear error rather than silently downloading gigabytes of data.
#### Path resolution
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
def resolve_snapshot_path(model_id: str) -> str:
org, name = model_id.split("/", 1)
model_root = os.path.join(HF_CACHE_ROOT, f"models--{org}--{name}")
refs_main = os.path.join(model_root, "refs", "main")
snapshots_dir = os.path.join(model_root, "snapshots")
```
Cached models use a specific directory structure. A model like `microsoft/Phi-3-mini-4k-instruct` gets stored at `/runpod-volume/huggingface-cache/hub/`. For example:
The `resolve_snapshot_path()` function navigates this structure to find the actual model files. It first tries to read the `refs/main` file, which contains the commit hash that the "main" branch points to. This is the most reliable method because it matches exactly what Hugging Face would load if you called `from_pretrained()` with network access.
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
if os.path.isfile(refs_main):
with open(refs_main, "r") as f:
snapshot_hash = f.read().strip()
candidate = os.path.join(snapshots_dir, snapshot_hash)
if os.path.isdir(candidate):
return candidate
```
If `refs/main` doesn't exist (which can happen with older cache formats), the function falls back to listing the `snapshots` directory and using the first available snapshot. This ensures compatibility with different caching scenarios.
#### Model loading
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
LOCAL_MODEL_PATH = resolve_snapshot_path(MODEL_ID)
tokenizer = AutoTokenizer.from_pretrained(
LOCAL_MODEL_PATH,
trust_remote_code=False,
local_files_only=True,
)
model = AutoModelForCausalLM.from_pretrained(
LOCAL_MODEL_PATH,
trust_remote_code=False,
torch_dtype="auto",
device_map="auto",
local_files_only=True,
attn_implementation="eager",
)
text_gen = pipeline("text-generation", model=model, tokenizer=tokenizer)
```
Model loading happens at the module level, outside any function. This means it runs once when the worker starts, not on every request. The model stays in GPU memory and gets reused across all incoming jobs, which is essential for performance.
The `local_files_only=True` parameter provides an additional layer of safety alongside offline mode. The `device_map="auto"` setting lets the Accelerate library automatically place model layers across available GPUs, and `torch_dtype="auto"` uses the model's native precision (typically float16 or bfloat16) to minimize memory usage.
Finally, wrapping the model and tokenizer in a `pipeline` provides a convenient high-level interface for text generation that handles tokenization, generation, and decoding in a single call.
#### Request handling
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
def handler(job):
job_input = job.get("input", {}) or {}
prompt = job_input.get("prompt", "Hello!")
max_tokens = int(job_input.get("max_tokens", 256))
temperature = float(job_input.get("temperature", 0.7))
```
The `handler` function is what your worker uses to process each incoming request. The `job` parameter is a dictionary containing the request data, with user inputs nested under the `"input"` key. The handler extracts parameters with sensible defaults: if a user doesn't specify `max_tokens`, they get 256; if they don't specify `temperature`, they get 0.7.
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
outputs = text_gen(
prompt,
max_new_tokens=max_tokens,
do_sample=True,
temperature=temperature,
)
generated = outputs[0]["generated_text"]
return {"status": "success", "output": generated}
```
The pipeline outputs a list of dictionaries (one per input sequence). Since we're processing a single prompt, we take `outputs[0]["generated_text"]` to get the generated string. The handler returns a dictionary that becomes the `output` field in the API response.
The `try/except` block around generation catches any errors (out of memory, invalid inputs, etc.) and returns them in a structured format rather than crashing the worker.
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
runpod.serverless.start({"handler": handler})
```
The final line registers the handler function with the Runpod SDK and starts the worker's event loop, which polls for jobs and dispatches them to your handler.
## Step 2: Create the requirements file
Create a `requirements.txt` file to specify the Python dependencies for your worker.
```text requirements.txt theme={"theme":{"light":"github-light","dark":"github-dark"}}
runpod>=1.6.2
transformers>=4.36.2
accelerate>=0.25.0
```
## Step 3: Create a Dockerfile
Create a `Dockerfile` to package your handler into a container image.
```dockerfile Dockerfile theme={"theme":{"light":"github-light","dark":"github-dark"}}
FROM runpod/pytorch:2.4.0-py3.11-cuda12.4.1-devel-ubuntu22.04
WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY handler.py .
CMD ["python", "-u", "handler.py"]
```
## Step 4: Set up your GitHub repository
Create a GitHub repository with your handler, requirements, and Dockerfile.
1. Create a new repository on GitHub (for example, `phi3-cached-worker`).
2. Add your files to the repository:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
git init
git add handler.py requirements.txt Dockerfile
git commit -m "Initial commit: Phi-3 cached model worker"
git remote add origin https://github.com/YOUR_USERNAME/phi3-cached-worker.git
git branch -M main
git push -u origin main
```
Replace `YOUR_USERNAME` with your GitHub username.
## Step 5: Deploy from GitHub
Deploy your worker directly from GitHub.
1. Navigate to the [Serverless section](https://www.console.runpod.io/serverless) and select **New Endpoint**.
2. Under **Import Git Repository**, select your `phi3-cached-worker` repository.
3. Configure deployment options:
* **Branch**: Select `main` (or your preferred branch).
* **Dockerfile Path**: Leave as default if Dockerfile is in the root.
* Select **Next**.
4. Configure endpoint settings:
* **Endpoint Name**: Choose a descriptive name (for example, "phi3-cached-inference").
* **Endpoint Type**: Make sure it's set to **Queue**.
* **GPU Configuration**: Select one or more GPU types with at least 16GB VRAM.
* **Workers**: Leave the defaults in place (minimum: 0, maximum: 3).
* **Container Disk**: Allocate at least 20 GB (or more if you're using a larger model).
5. **Enable cached models**:
* Scroll to the **Model** section.
* Enter the model name:
```text theme={"theme":{"light":"github-light","dark":"github-dark"}}
microsoft/Phi-3-mini-4k-instruct
```
... or your preferred model that's available on Hugging Face.
* (Optional) If using a gated model, add your Hugging Face token.
6. Select **Deploy Endpoint**.
Runpod automatically builds your Docker image and deploys it to your endpoint. You can monitor the build status in the **Builds** tab.
## Step 6: Test your endpoint
Once deployed, send requests to your endpoint using the Runpod API. Replace `YOUR_ENDPOINT_ID` with your actual endpoint ID.
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
import requests
import os
endpoint_id = "YOUR_ENDPOINT_ID"
api_key = os.environ.get("RUNPOD_API_KEY")
url = f"https://api.runpod.ai/v2/{endpoint_id}/runsync"
headers = {
"Authorization": f"Bearer {api_key}",
"Content-Type": "application/json",
}
payload = {
"input": {
"prompt": "Explain what large language models are in simple terms.",
"max_tokens": 150,
"temperature": 0.7,
}
}
response = requests.post(url, json=payload, headers=headers)
result = response.json()
print("Generated text:", result["output"]["output"])
```
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
curl -X POST https://api.runpod.ai/v2/YOUR_ENDPOINT_ID/runsync \
-H "Authorization: Bearer YOUR_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"input": {
"prompt": "Explain what large language models are in simple terms.",
"max_tokens": 150,
"temperature": 0.7
}
}'
```
Expected response:
```json theme={"theme":{"light":"github-light","dark":"github-dark"}}
{
"id": "sync-request-id",
"status": "COMPLETED",
"output": {
"status": "success",
"output": "Explain what large language models are in simple terms. Large language models (LLMs) are AI systems trained on vast amounts of text data..."
}
}
```
Congratulations! You've successfully deployed a Serverless endpoint that uses model caching to serve Phi-3.
## Benefits of using cached models
By using Runpod's cached model feature in this tutorial, you gain several advantages:
* **Faster cold starts**: Workers start in seconds instead of minutes.
* **Cost savings**: No billing during model download time.
* **Simplified deployment**: Models are automatically available to all workers.
* **Better scalability**: Quick worker scaling without waiting for downloads.
## Next steps
Now that you have a working Phi-3 endpoint with cached models, you can:
* Experiment with different [Phi model variants](https://huggingface.co/microsoft) (Phi-3-medium, Phi-3.5, etc.).
* Add more sophisticated prompt templates and chat formatting.
* Implement streaming responses for real-time generation.
* Integrate with existing applications using the Runpod SDK.
## Related resources
Learn more about cached models and their benefits
Deploy workers directly from GitHub repositories
Understand handler function structure and best practices
Explore vLLM for optimized LLM inference
# Deploy a Gemma 3 chatbot with the OpenAI API
Source: https://docs.runpod.io/tutorials/serverless/run-gemma-7b
Deploy a Serverless endpoint with Google's Gemma 3 model using vLLM and the OpenAI API to build an interactive chatbot.
This tutorial walks you through deploying a with Google's Gemma 3 model using the vLLM worker. You'll deploy the `gemma-3-1b-it` instruction-tuned variant, a lightweight model that runs efficiently on a variety of GPUs.
By the end, you'll have a fully functional Serverless endpoint that can respond to chat-style prompts through the [OpenAI-compatible API](/serverless/vllm/openai-compatibility).
## Requirements
Before starting this tutorial, you'll need:
* A [Runpod account](https://console.runpod.io/signup) with available credits.
* A [Runpod API key](/get-started/api-keys).
* A [Hugging Face account](https://huggingface.co/join) and [access token](https://huggingface.co/settings/tokens).
* Python 3.8+ installed on your local machine.
* The OpenAI Python client (`pip install openai`).
## Step 1: Accept Google's terms on Hugging Face
Gemma 3 is a gated model on Hugging Face. Before deploying, go to the [Gemma 3 1B model page](https://huggingface.co/google/gemma-3-1b-it) and click **Agree and access repository** while logged into your Hugging Face account.
You must accept the terms using the same Hugging Face account that you'll use to generate your access token. Without accepting these terms, your deployment will fail to download the model.
## Step 2: Deploy the vLLM worker
Deploy a vLLM worker through the Runpod Hub with Gemma 3 as your model:
1. Open the [vLLM worker listing](https://console.runpod.io/hub/runpod-workers/worker-vllm) in the Runpod Hub.
2. Click **Deploy**, using the latest vLLM worker version.
3. In the **Model** field, enter: `google/gemma-3-1b-it`.
4. Under **Hugging Face access token**, enter your Hugging Face access token.
5. Click **Advanced** to expand the vLLM settings.
6. Set **Max Model Length** to `8192`.
7. Set **Data Type** to `bfloat16`.
8. Click **Next**.
9. Select a GPU with at least 16GB of VRAM (such as RTX A4000 or RTX 4000 Ada).
10. Click **Create Endpoint**.
We're using a lightweight model for this tutorial to reduce cost and speed up response times. For larger Gemma models, you'll need GPUs with more VRAM.
Your endpoint will begin initializing. This may take several minutes while Runpod provisions resources and downloads the model. You can monitor the deployment status on the [Serverless endpoints page](https://console.runpod.io/serverless).
## Step 3: Note your endpoint ID
Once your endpoint is deployed, make a note of your **Endpoint ID** from the endpoint details page. You'll need this to construct your API base URL.
Your API base URL will follow this pattern:
```
https://api.runpod.ai/v2/{ENDPOINT_ID}/openai/v1
```
## Step 4: Test your endpoint
Before building a chatbot, verify that your endpoint is working correctly using a simple test request.
Set up your environment variables, replacing `YOUR_RUNPOD_API_KEY` and `YOUR_ENDPOINT_ID` with your actual API key and endpoint ID:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
export RUNPOD_API_KEY="YOUR_RUNPOD_API_KEY" && \
export RUNPOD_ENDPOINT_ID="YOUR_ENDPOINT_ID"
```
Send a test request using `curl`:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
curl "https://api.runpod.ai/v2/${RUNPOD_ENDPOINT_ID}/openai/v1/chat/completions" \
-H "Content-Type: application/json" \
-H "Authorization: Bearer ${RUNPOD_API_KEY}" \
-d '{
"model": "google/gemma-3-1b-it",
"messages": [
{"role": "user", "content": "Hello! What is your name?"}
],
"max_tokens": 100
}'
```
You should receive a JSON response containing the model's reply. If the request fails, check that:
* Your endpoint has finished initializing (status shows "Ready").
* Your `HF_TOKEN` environment variable is set correctly on the endpoint.
* You've accepted Google's terms on Hugging Face.
## Step 5: Build a chatbot
With your endpoint deployed, you can interact with it using the [OpenAI-compatible API](/serverless/vllm/openai-compatibility). The following example creates a command-line chatbot that maintains conversation history and generates responses using your deployed model.
Gemma 3 1B is a lightweight model optimized for efficiency. While it handles most conversational context well, it may give inconsistent responses to questions about personal information (like recalling names) due to its safety training. For more reliable multi-turn conversations, consider larger Gemma variants such as `gemma-3-4b-it` or `gemma-3-12b-it`.
Create a local Python file called `gemma_chat.py`:
```python gemma_chat.py theme={"theme":{"light":"github-light","dark":"github-dark"}}
from openai import OpenAI
import os
# Initialize the client with your Runpod endpoint
client = OpenAI(
base_url=f"https://api.runpod.ai/v2/{os.environ['RUNPOD_ENDPOINT_ID']}/openai/v1",
api_key=os.environ["RUNPOD_API_KEY"],
)
# Initialize conversation history
messages = [
{
"role": "system",
"content": "You are a helpful assistant.",
}
]
def display_chat_history(messages):
"""Display the conversation history."""
for message in messages:
if message["role"] != "system":
print(f"{message['role'].capitalize()}: {message['content']}")
def get_assistant_response(messages):
"""Get a response from the Gemma model."""
response = client.chat.completions.create(
model="google/gemma-3-1b-it",
messages=messages,
temperature=0.7,
top_p=0.9,
max_tokens=256,
)
return response.choices[0].message.content
def main():
print("Gemma 3 Chatbot")
print("Type 'quit' to exit.\n")
while True:
# Get user input
user_input = input("You: ")
if user_input.lower() == "quit":
print("Goodbye!")
break
# Add user message to history
messages.append({"role": "user", "content": user_input})
# Get and display response
response = get_assistant_response(messages)
messages.append({"role": "assistant", "content": response})
print(f"Assistant: {response}\n")
if __name__ == "__main__":
main()
```
Before running the script, install the OpenAI Python client and set your environment variables.
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
pip install openai
```
Replace `YOUR_RUNPOD_API_KEY` and `YOUR_ENDPOINT_ID` with your actual API key and endpoint ID:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
export RUNPOD_API_KEY="YOUR_RUNPOD_API_KEY"
export RUNPOD_ENDPOINT_ID="YOUR_ENDPOINT_ID"
```
Run your chatbot script:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
python gemma_chat.py
```
The first request may take longer (30-60 seconds) due to as the endpoint loads the model into GPU memory. Subsequent requests complete in just a few seconds.
You can now have a conversation with Gemma 3:
```text theme={"theme":{"light":"github-light","dark":"github-dark"}}
Gemma 3 Chatbot
Type 'quit' to exit.
You: What are some fun facts about space?
Assistant: Here are some fun facts about space:
1. A day on Venus is longer than its year - Venus takes 243 Earth days to rotate once but only 225 Earth days to orbit the Sun.
2. There's a planet made of diamonds - 55 Cancri e is believed to be covered in graphite and diamond.
3. Space is completely silent - with no atmosphere to carry sound waves, space is eerily quiet.
You: quit
Goodbye!
```
## Next steps
You've successfully deployed Gemma 3 on Runpod Serverless and built a chatbot to interact with it. Here are some ways to extend this tutorial:
* [Configure your endpoint](/serverless/endpoints/endpoint-configurations) to optimize performance and cost.
* [Learn about vLLM environment variables](/serverless/vllm/environment-variables) to customize model behavior.
* [Explore OpenAI compatibility](/serverless/vllm/openai-compatibility) for features like streaming and function calling.
* Build a custom for more specialized use cases.
# Run Ollama on Serverless (CPU)
Source: https://docs.runpod.io/tutorials/serverless/run-ollama-inference
Learn how to run an Ollama server on Serverless CPU workers. Follow implementation steps, configuration guidance, and examples in this Runpod tutorial.
Run an Ollama server on CPU for LLM inference. This tutorial focuses on CPU compute, but you can also select a GPU for faster performance.
## Requirements
Before starting, you'll need:
* A Runpod account with credits.
* (Optional) A [network volume](/storage/network-volumes) to store models.
## Step 1: Deploy a Serverless endpoint
We recommend attaching a [network volume](/storage/network-volumes) to store downloaded models. Without a network volume, the worker downloads the model on every cold start, increasing latency. You can attach a network volume to your endpoint after it's deployed.
1. Log in to the [Runpod console](https://www.console.runpod.io/console/home).
2. Navigate to **Serverless** and select **New Endpoint**.
3. Choose **CPU** and select a configuration (for example, 8 vCPUs and 16 GB RAM).
4. Configure your worker settings as needed.
5. In the **Container Image** field, enter: `pooyaharatian/runpod-ollama:0.0.8`
6. In the **Container Start Command** field, enter the model name (for example, `orca-mini` or `llama3.1`). See the [Ollama library](https://ollama.com/library) for available models.
7. Allocate at least 20 GB of container disk space.
8. (Optional) Add an environment variable with key `OLLAMA_MODELS` and value `/runpod-volume` to store models on your attached network volume.
9. Select **Deploy**.
Wait for the model to download and the worker to become ready.
## Step 2: Send a request
Once your endpoint is deployed:
1. Go to the **Requests** section in the Runpod console.
2. Enter the following JSON in the input field:
```json theme={"theme":{"light":"github-light","dark":"github-dark"}}
{
"input": {
"method_name": "generate",
"input": {
"prompt": "Why is the sky blue?"
}
}
}
```
3. Select **Run**.
You'll receive a response like this:
```json theme={"theme":{"light":"github-light","dark":"github-dark"}}
{
"delayTime": 153,
"executionTime": 4343,
"id": "c2cb6af5-c822-4950-bca9-5349288c001d-u1",
"output": {
"model": "orca-mini",
"response": "The sky appears blue because of a process called scattering...",
"done": true
},
"status": "COMPLETED"
}
```
Your Ollama endpoint is now ready to integrate into your applications using the Runpod API.
## Next steps
* Explore the [Runpod Ollama repository](https://github.com/pooyahrtn/) for more configuration options.
* View the [Runpod Ollama container image](https://hub.docker.com/r/pooyaharatian/runpod-ollama) on Docker Hub.
* Learn more about [sending requests to Serverless endpoints](/serverless/endpoints/send-requests).
# Generate images with Serverless and SDXL
Source: https://docs.runpod.io/tutorials/serverless/run-your-first
Learn how to deploy a Serverless endpoint running SDXL from the Runpod Hub and use it to generate images.
In this tutorial, you will learn how to deploy a endpoint running [Stable Diffusion XL](https://stablediffusionxl.com/) (SDXL) on Runpod, submit image jobs, monitor their progress, and decode the resulting images.
[Runpod's Serverless platform](/serverless/overview) allows you to run AI/ML models in the cloud without managing infrastructure, automatically scaling resources as needed. SDXL is a powerful AI model that generates high-quality images from text prompts.
## Requirements
Before starting this tutorial you'll need:
* A Runpod account with available credits.
* A Runpod API key (available in your user settings).
* Basic familiarity with command-line tools like `curl`.
* Python installed on your system (for the image decoding step).
* The `jq` command-line JSON processor (optional but recommended).
Keep your API key secure and never share it publicly. Remember to retrieve your results within 30 minutes, as inputs and outputs are not stored longer than this for privacy protection.
## Step 1: Deploy a Serverless endpoint using the Runpod Hub
1. Navigate to the [SDXL listing](https://console.runpod.io/hub/runpod-workers/worker-sdxl) in the Runpod Hub web interface.
2. Click **Deploy \[VERSION\_NUMBER]**, then click **Create Endpoint** to confirm. This creates a fully configured endpoint with appropriate GPU and worker settings for running SDXL.
This tutorial uses version 2.1.0 of runpod-workers/worker-sdxl. Later versions may require different configurations or instructions.
3. On the endpoint page, make a note of the **Endpoint ID** . You'll need this value to submit jobs and retrieve results.
Once deployed, your endpoint will be assigned a unique ID (e.g. `32vgrms732dkwi`). Your endpoint URL will follow this pattern: `https://api.runpod.ai/v2/ENDPOINT_ID/run` for asynchronous requests.
## Step 2: Submit your first job
Use the `/run` endpoint to submit an asynchronous job that will generate an image based on your text prompt.
Replace `ENDPOINT_ID` with your actual endpoint ID and `YOUR_API_KEY` with your Runpod API key in the following command:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
curl -X POST https://api.runpod.ai/v2/ENDPOINT_ID/run \
-H 'Content-Type: application/json' \
-H 'Authorization: Bearer YOUR_API_KEY' \
-d '{"input": {"prompt": "A cute fluffy white dog in the style of a Pixar animation 3D drawing."}}'
```
The API will respond immediately with a job ID and status. You'll receive a response similar to this:
```json theme={"theme":{"light":"github-light","dark":"github-dark"}}
{
"id": "c80ffee4-f315-4e25-a146-0f3d98cf024b",
"status": "IN_QUEUE"
}
```
The job ID is crucial for tracking your request's progress. Save this ID as you'll need it to check the status and retrieve results.
## Step 3: Monitor job progress
Check your job's status using the `/status` endpoint with the job ID you received in the previous step.
Use the following command to check your job's progress, replacing the placeholders (`ENDPOINT_ID`, `JOB_ID`, and `YOUR_API_KEY`) with your actual values:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
curl https://api.runpod.ai/v2/ENDPOINT_ID/status/JOB_ID \
-H 'Content-Type: application/json' \
-H 'Authorization: Bearer YOUR_API_KEY'
```
While your job is processing, you'll receive a response indicating the current status:
```json theme={"theme":{"light":"github-light","dark":"github-dark"}}
{
"delayTime": 2624,
"id": "c80ffee4-f315-4e25-a146-0f3d98cf024b",
"input": {
"prompt": "A cute fluffy white dog in the style of a Pixar animation 3D drawing."
},
"status": "IN_PROGRESS"
}
```
The `delayTime` field shows how long the job waited in the queue before processing began, measured in milliseconds.
## Step 4: Retrieve completed results
Continue polling this endpoint until the status changes to `COMPLETED`. Once your job completes, the status endpoint will return the generated image data encoded in base64 format.
When your job finishes successfully, you'll receive a response containing the output:
```json theme={"theme":{"light":"github-light","dark":"github-dark"}}
{
"delayTime": 27540,
"executionTime": 14904,
"id": "fb5a249d-12c7-48e5-a0e4-b813c3381262-22",
"output": [
{
"image_url": "data:image/png;base64 ...",
"seed": 37362
}
],
"status": "COMPLETED"
"workerId": "qebsdrm4qidkl3"
}
```
The `executionTime` field shows how long the actual image generation took, while `delayTime` indicates the initial queue wait time. Both values are in milliseconds.
To save the complete response for processing, use this command:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
curl https://api.runpod.ai/v2/ENDPOINT_ID/status/JOB_ID \
-H 'Content-Type: application/json' \
-H 'Authorization: Bearer YOUR_API_KEY' | jq . > output.json
```
You have up to 30 minutes to retrieve your results via the status endpoint. After this time, the results are automatically deleted for security.
## Step 5: Decode and save your image
Now we'll convert the base64-encoded image data into a viewable image file using Python.
Create a Python script called `decode_image.py` to decode the base64 image data from your JSON response:
```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
import base64
from PIL import Image
import io
import os
import json
def decode_json_and_save_image(json_filepath, output_filename="decoded_image.png"):
"""
Reads a JSON file, extracts the base64 image string, decodes it, and saves it as an image file.
Args:
json_filepath (str): The path to the input JSON file.
output_filename (str): The name for the output image file.
"""
try:
with open(json_filepath, 'r') as f:
data = json.load(f)
# Extract the base64 string from the nested structure
base64_url = data.get("output", {}).get("image_url")
if not base64_url:
print("Error: 'image_url' not found in the JSON file.")
return
# Remove data URI prefix if present
if "," in base64_url:
_, encoded_data = base64_url.split(",", 1)
else:
encoded_data = base64_url
# Decode base64 to bytes
image_data = base64.b64decode(encoded_data)
image_stream = io.BytesIO(image_data)
image = Image.open(image_stream)
image.save(output_filename)
print(f"Image successfully saved as '{output_filename}'")
print(f"Image path: {os.path.abspath(output_filename)}")
except FileNotFoundError:
print(f"Error: The file '{json_filepath}' was not found.")
except json.JSONDecodeError:
print(f"Error: Could not decode JSON from the file '{json_filepath}'.")
except base64.binascii.Error as e:
print(f"Error decoding base64 string: {e}")
print("Please ensure the input is a valid base64 string.")
except Exception as e:
print(f"An unexpected error occurred: {e}")
# Process the output.json file
decode_json_and_save_image("output.json", "generated_image.png")
```
Run the script to decode the image data and save it as a PNG file:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
python decode_image.py
```
You should see the following output:
```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
Image successfully saved as 'generated_image.png'
Image path: /Users/path/to/your/project/generated_image.png
```
Congratulations! You've successfully used Runpod's Serverless platform to generate an AI image using SDXL. You now understand the complete workflow of submitting asynchronous jobs, monitoring their progress, and retrieving results.
## Next steps
Now that you've learned how to generate images with Serverless, consider exploring these advanced topics:
* Learn how to create [synchronous requests](/serverless/endpoints/send-requests) using the `/runsync` endpoint for faster responses.
* Explore [endpoint configurations](/serverless/endpoints/endpoint-configurations) to optimize performance and cost.
* Discover how to [send requests](/serverless/endpoints/send-requests) with advanced parameters and webhook notifications.
* Try deploying your own [custom worker](/serverless/quickstart) for specialized AI models.