Pod migration is currently in beta. Join our Discord if you’d like to provide feedback.
Your options when GPUs are unavailable
When prompted to migrate your Pod, you have three options:- Do nothing: If you don’t want to migrate your data, you can wait and try again later. The GPU will become available once another user stops their Pod on that machine.
- Start Pod with CPUs: If you don’t need GPU access immediately, you can start your Pod with CPUs only. This lets you access your data and manually migrate files if needed, but the Pod will have limited CPU resources and is not suitable for compute-intensive tasks.
- Automatically migrate Pod data: This option spins up a new Pod with the same specifications as your current one and automatically migrates your data to a machine with available GPUs. The migration process finds a new machine with your requested GPU type, provisions the instance, and transfers your network volume data from the old Pod to the new one.
Important considerations for migration
When you trigger an automatic Pod migration, you’ll receive a new Pod with a new ID and IP address. This is because Pod IDs are architecturally tied to specific physical machines. This may impact your workload if you have:- A Pod ID hardcoded in an API call.
- A proxy URL hardcoded (e.g.,
http://b63b243b47bd340becc72fbe9b3e642c.proxy.runpod.net). - A firewall or VPN configured with a specific Pod ID.
- A firewall or VPN configured with a specific Pod IP address.
- A specific URL for your server (when you start a new Pod, you’ll get a new URL for any UI or server you’ve set up).
Preventing Pod migration scenarios
The most effective way to avoid the need for Pod migrations is to use network volumes. Network volumes decouple your data from specific physical machines, storing your/workspace data on a separate, persistent volume that can be attached to any Pod. If you need to terminate a Pod, you can deploy a new one and attach the same network volume, giving you immediate access to your data on any machine with an available GPU.