Generator Exhaustion in Cached Methods

We had a method returning API responses as generators, with a TTL cache wrapping it. Simple enough — or so I thought.

The Bug

The method signature looked something like this:

@cache(ttl=3600)
def fetch_foo_data() -> Generator[dict, None, None]:
    response = call_external_api()
    for item in response:
        yield item

For the first call after the cache expired, this worked fine. But subsequent calls returned empty — nothing iterated. Then, when the TTL expired again and we called it once more, data appeared again.

The cache decorator was storing the generator object itself, not the results. Generators in Python are stateful iterables that exhaust after one full iteration. Once you've iterated through a generator, it's done. Subsequent attempts to iterate over the same generator object yield nothing.

So the flow was:

1. Call 1 (cache miss): Generator created, cache stores the generator object, we iterate it, data returns. Generator is now exhausted.
2. Call 2-N (cache hit): Same exhausted generator object returned from cache. Iteration yields nothing.
3. Call N+1 (cache expired): New generator created, cached, iterated successfully. Cycle repeats.

How I Found It

The symptoms were initially confusing. On my local server, the first request worked — data came back clean. After restarting the server, it worked again. But sustained traffic revealed the issue: second request onward returned empty. The behavior was inconsistent between local dev and production on sustained load.

Comparing the two environments, I noticed:

• Local: Single request → works. Restart server → works. Second request → empty.
• Server: Consistent empty responses after first request.

The pattern screamed "state not resetting" rather than a logic error. The cache decorator was the obvious suspect. Once I realized the generator was being stored in the cache (not the results), the picture clicked: generators exhaust after one iteration. The cache was serving the same exhausted generator on every subsequent call.

The Fix

Convert the generator to a list before caching:

@cache(ttl=3600)
def fetch_foo_data() -> list[dict]:
    response = call_external_api()
    return [item for item in response]

Or if you need to keep the generator interface (e.g., for streaming large datasets), don't cache the generator. Cache the raw response or the computation, then generate on each call:

@cache(ttl=3600)
def foo() -> list[dict]:
    response = call_external_api()
    return [item for item in response]

def fetch_foo_data() -> Generator[dict, None, None]:
    data = foo()
    for item in data:
        yield item

Lesson

Caching is straightforward for immutable values (ints, strings, dicts). But stateful objects — generators, file handles, database cursors — don't cache well. They're designed to be consumed, not reused. If you're caching something that isn't a pure value, ask: will this object still be valid when I pull it from the cache?

The generator bug was subtle because the cache decorator didn't fail or warn. It silently returned the exhausted object. The symptom (intermittent empty results) didn't obviously point to the root cause until you traced the cache behavior.