Configuration best practices

​

Production

• Pin GPUs to a specific pool or device instead of GpuGroup.ANY, so performance and cost are predictable.
• Keep workers warm for latency-sensitive work by setting a minimum of 1 or more.
• Set execution_timeout_ms so a stuck job can’t run indefinitely.
• Store secrets with runpod.Secret, not as plain values in env.
• Cache model weights with model or a network volume, so workers don’t download them on every cold start.

​

Queue resources

import runpod

app = runpod.App("batch")
weights = runpod.NetworkVolume("weights", size=100)

@app.queue(
    gpu=runpod.GpuType.NVIDIA_A100_80GB_PCIe,
    workers=(1, 10),
    idle_timeout=1200,            # 20 minutes
    execution_timeout_ms=600000,  # 10 minutes per job
    mounts={"/runpod-volume": weights},
    env={"HF_TOKEN": runpod.Secret("hf-token")},
)
def process_batch(items: list):
    ...

​

API resources

@app.api(gpu="4090", workers=(3, 20), idle_timeout=1800)
class Inference:
    @runpod.init
    def setup(self):
        self.model = load_model()  # Runs once per worker

    @runpod.post("/generate")
    async def generate(self, body: dict):
        return {"output": self.model(body["prompt"])}

    @runpod.get("/health")
    async def health(self):
        return {"status": "ok"}

​

Development

• Use GpuGroup.ANY or a list of GPUs so workers start quickly.
• Keep the maximum low, for example workers=(0, 2), to limit spend while testing.
• Use a short idle_timeout, such as 300 seconds, so workers scale down between runs.
• Iterate with rp flash dev, which deletes its temporary endpoints when you exit. See Dev sessions.

@app.queue(gpu=runpod.GpuGroup.ANY, workers=(0, 2), idle_timeout=300)
def experiment(data: dict):
    ...

​

Cost control

• Scale to zero with a minimum of 0 workers. The default, (0, 3), already does this.
• Use the smallest GPU that fits, or a CPU instance when you don’t need a GPU.
• Shorten idle_timeout for infrequent traffic.
• Batch work into fewer, larger jobs.
• Delete volumes you no longer need. Volumes keep billing after you undeploy an app.

​

Trade-offs

​

Checklist before deploying

• GPUs are pinned for production resources.
• Latency-sensitive resources have a minimum of at least 1 worker.
• execution_timeout_ms is set.
• Secrets use runpod.Secret, and every variable workers need is passed through env.
• Large models are cached with model or a volume.
• API resources have a health route.
• The app runs cleanly with rp flash dev main.py --once.

​

Next steps

• See every resource parameter.
• Deploy your app.