Run Ollama on Serverless (CPU)

​

Requirements

• A Runpod account with credits.
• (Optional) A network volume to store models.

​

Step 1: Deploy a Serverless endpoint

1. Log in to the Runpod console.
2. Navigate to Serverless and select New Endpoint.
3. Choose CPU and select a configuration (for example, 8 vCPUs and 16 GB RAM).
4. Configure your worker settings as needed.
5. In the Container Image field, enter: pooyaharatian/runpod-ollama:0.0.8
6. In the Container Start Command field, enter the model name (for example, orca-mini or llama3.1). See the Ollama library for available models.
7. Allocate at least 20 GB of container disk space.
8. (Optional) Add an environment variable with key OLLAMA_MODELS and value /runpod-volume to store models on your attached network volume.
9. Select Deploy.

​

Step 2: Send a request

1. Go to the Requests section in the Runpod console.
2. Enter the following JSON in the input field:
   {
     "input": {
       "method_name": "generate",
       "input": {
         "prompt": "Why is the sky blue?"
       }
     }
   }
   
3. Select Run.

{
  "delayTime": 153,
  "executionTime": 4343,
  "id": "c2cb6af5-c822-4950-bca9-5349288c001d-u1",
  "output": {
    "model": "orca-mini",
    "response": "The sky appears blue because of a process called scattering...",
    "done": true
  },
  "status": "COMPLETED"
}

​

Next steps

• Explore the Runpod Ollama repository for more configuration options.
• View the Runpod Ollama container image on Docker Hub.
• Learn more about sending requests to Serverless endpoints.