Configuration best practices

​

Production workloads

​

General recommendations

• Pin specific GPU types instead of using GpuGroup.ANY for predictable performance and costs.
• Use network volumes for large models to avoid downloading on each worker startup. Wrap the model load in VolumeCache so workers load the weights from local disk instead of reading them over the network mount.
• Set appropriate execution_timeout_ms to prevent runaway jobs and control costs.
• Use environment variables for configuration and secrets, not hardcoded values.

​

Queue-based endpoints

from runpod_flash import Endpoint, GpuType, NetworkVolume

@Endpoint(
    name="production-batch",
    gpu=GpuType.NVIDIA_A100_80GB_PCIe,  # Specific GPU for predictable performance
    workers=(1, 10),        # At least 1 worker, scale up to 10
    idle_timeout=1200,      # 20 minutes - keep workers longer for variable traffic
    execution_timeout_ms=600000,  # 10 minute timeout
    volume=NetworkVolume(name="my-volume"),
    env={"MODEL_PATH": "/runpod-volume/models"}
)
def process_batch(data): ...

• workers=(1, n): Set min to 1 to avoid cold starts for first job in queue.
• workers=(n, max): Set max based on expected peak concurrent jobs.
• idle_timeout: 900-1800 seconds (15-30 minutes) for production workloads.

​

Load-balanced endpoints

from runpod_flash import Endpoint, GpuType, NetworkVolume

api = Endpoint(
    name="production-api",
    gpu=GpuType.NVIDIA_GEFORCE_RTX_4090,  # Specific GPU for consistent performance
    workers=(3, 20),        # Always keep 3 workers ready, scale to 20
    idle_timeout=1800,      # 30 minutes - keep workers active longer
    execution_timeout_ms=60000,  # 60 second timeout per request
    volume=NetworkVolume(name="my-volume")
)

@api.post("/process")
async def process_request(data: dict) -> dict:
    return {"result": "processed"}

@api.get("/health")
async def health_check() -> dict:
    return {"status": "healthy"}

• workers=(n, max): Set min ≥ 1 for production APIs to avoid cold starts. Unlike queue-based endpoints where jobs can wait, API clients expect immediate responses.
• workers=(min, n): Set max based on expected peak concurrent requests.
• idle_timeout: 1200-1800 seconds (20-30 minutes) to keep workers ready.
• Include health check routes (e.g., GET /health) for monitoring.

​

Development

​

General recommendations

• Use GpuGroup.ANY for fastest GPU provisioning during development.
• Set workers=(0, n) to minimize costs when not actively testing.
• Keep max workers low (1-3) to control development expenses.
• Use short idle_timeout (300 seconds / 5 minutes) to scale down quickly between test runs.
• Test locally with flash dev before deploying to production.

​

Example configuration

from runpod_flash import Endpoint, GpuGroup

@Endpoint(
    name="dev-testing",
    gpu=GpuGroup.ANY,       # Fast provisioning
    workers=(0, 2),         # Scale to zero, limit to 2 concurrent
    idle_timeout=300        # 5 minutes - quick scale-down
)
def test_function(data): ...

​

Cost optimization

​

General recommendations

• Set workers=(0, n) to scale to zero when there are no requests (no running workers = no compute cost).
• Use smaller GPU types when workload allows (e.g., GpuType.NVIDIA_GEFORCE_RTX_4090 instead of GpuType.NVIDIA_A100_80GB_PCIe).
• Use CPU endpoints when GPU acceleration isn’t needed.
• Reduce idle_timeout for sporadic workloads (300-600 seconds / 5-10 minutes).
• Batch operations into fewer job submissions when possible.

​

Cost-optimized queue-based endpoint

from runpod_flash import Endpoint, GpuType, NetworkVolume

@Endpoint(
    name="batch-job",
    gpu=GpuType.NVIDIA_GEFORCE_RTX_4090,  # Cost-effective GPU
    workers=(0, 5),         # Scale to zero, controlled max
    idle_timeout=300,       # 5 minutes - fast scale-down
    volume=NetworkVolume(name="my-volume")  # Avoid re-downloading models
)
def batch_process(data): ...

​

Cost-optimized CPU endpoint

from runpod_flash import Endpoint

@Endpoint(
    name="cpu-batch",
    cpu="cpu5c-4-8",        # 4 vCPU, 8GB RAM
    workers=(0, 3),         # Scale to zero, limit to 3
    idle_timeout=300        # 5 minutes - fast scale-down
)
def cpu_process(data): ...

​

Configuration trade-offs

​

Configuration checklist

• GPU selection: Using specific GPU types (not GpuGroup.ANY) for predictable performance
• Worker scaling: workers=(1, n) or higher min for load balancers and latency-sensitive workloads
• Timeouts: execution_timeout_ms set appropriately for your workload
• Storage: Network volume attached if using large models or datasets
• Environment variables: All configuration and secrets passed via env parameter
• Monitoring: Health check routes implemented (load balancers)
• Testing: Tested locally with flash dev before production deployment