Iterators, Generators & Comprehensions

A list holds all its items in memory at once. A generator produces items one at a time, only when asked — so you can process a 10 GB file or an endless stream with tiny memory use.

Write a generator function with yield instead of return, or a generator expression with round brackets: (x * 2 for x in data). Chain generators into a pipeline where each step handles one item at a time.

The itertools module has fast building blocks: islice (take the first n), groupby, chain, count and more. collections.Counter counts things in one line.

generators.pyPython

from collections import Counter
from collections.abc import Iterable, Iterator
from itertools import islice


def read_scores(lines: Iterable[str]) -> Iterator[tuple[str, int]]:
    for line in lines:
        name, _, score = line.strip().partition(",")
        if score.isdigit():
            yield name, int(score)


def grades(rows: Iterable[tuple[str, int]]) -> Iterator[str]:
    for _, score in rows:
        yield "A" if score >= 75 else "B" if score >= 65 else "C" if score >= 45 else "D" if score >= 30 else "F"


def numbers_from(start: int) -> Iterator[int]:
    while True:              # infinite, but lazy
        yield start
        start += 1


lines = ["Amina,88", "Juma,42", "bad line", "Neema,71", "Ali,29"]
print(Counter(grades(read_scores(lines))))    # Counter({'A': 1, 'D': 1, 'B': 1, 'F': 1})

print(list(islice(numbers_from(10), 5)))      # [10, 11, 12, 13, 14]

total = sum(n * n for n in range(1_000_000))  # generator expression: no big list in memory
print(total)

Runs in your browser · Python

Key points

Generators (yield) produce items lazily — ideal for big or endless data.
Use (...) generator expressions inside sum, max, any instead of building lists.
itertools and collections.Counter solve many problems in one line.

Exercise

Write a generator chunks(items, size) that yields lists of size items. Use it to process a list of 1,000 student IDs in batches of 100 and print each batch's first and last ID.

Show solution

Try the exercise yourself first — then compare your approach with this one.

chunks is a generator: it slices the list size items at a time and yields each batch, so only one batch is handled at once.

batches.pyPython

from collections.abc import Iterator, Sequence
from typing import TypeVar

T = TypeVar("T")


def chunks(items: Sequence[T], size: int) -> Iterator[list[T]]:
    for start in range(0, len(items), size):
        yield list(items[start:start + size])


student_ids = [f"S{n:04d}" for n in range(1, 1001)]

for number, batch in enumerate(chunks(student_ids, 100), start=1):
    print(f"Batch {number:>2}: {batch[0]} .. {batch[-1]} ({len(batch)} ids)")

Runs in your browser · Python

Key points

Exercise

Check your understanding