Generator Handler

RunPod provides a robust streaming feature that enables users to receive real-time updates on job outputs, mainly when dealing with Language Model tasks. We support two types of streaming generator functions: regular generator and async generator.

import runpod


def generator_handler(job):
    for count in range(3):
        result = f"This is the {count} generated output."
        yield result


runpod.serverless.start(
    {
        "handler": generator_handler,  # Required
        "return_aggregate_stream": True,  # Optional, results available via /run
    }
)

Return aggregate Stream

By default, when a generator handler is running, the fractional outputs will only be available at the /stream endpoint, if you would also like the outputs to be available from the /run and /runsync endpoints you will need to set return_aggregate_stream to True when starting your handler.