Generators in Python

Before starting with this tutorial you should learn what is yield Keyword and what are iterables & iterators as those concepts will be used in here.

A Generator is nothing but a function which returns value using the yield keyword and not using the return statement.

If we have to write a custom iterator you will have to implement the __iter__() and __next__() method, define the mechanism of returning the next value by maintaining the state and raise the StopIteration exception when no more values are left for iterating.

Well, generator makes it super easy to create an iterator in which state is maintained, all you have to do is use the yield keyword to return data.

Any function that contains the yield keyword becomes a generator and will return an iterator object which can be iterated over using a for loop.

Creating a Python Generator

The magic recipe to convert a simple function into a generator function is the yield keyword. We can have a single or multiple yield statements to return some data from the generator where each time the generator is called the yield statement stores the state of the local variables and yields a result.

Here is a simple example,

In above example, we have defined a python generator called new_gen() in which we take a variable n assign it some value, print something and then yield a value. Inside the function, we have repeated it 3 times.

If we try to call a generator function like a normal function it won't get executed because it returns an iterator object. Hence we have taken a variable x to store the iterator object and then used the next() method to execute the method. And every time the execution of the generator starts from where it last left i.e. the last yield statement.

Yielding into Python List

We can directly use a generator to form a list by generating values as shown in the example below, where we have used a generator to generate a series of values and stored the values yielded by the generator into the list mylist.

# defining a generator
def generator():
    x = 0
    while x < 5:
        yield x
        x += 1

# yielding values directly into a list
mylist = list(generator())

# use for loop to print mylist items
for x in mylist:
    print(x)

0 1 2 3 4

Generator vs. Normal Function vs. Python List

The major difference between a generator and a simple function is that a generator yields values instead of returning values. In simpler words, a simple function just returns a single variable and then it ends but a generator can yield a series of values by providing an iterator object on which we can iterate.

A generator must have a yield statement, else it will just be a normal function.

As the generator yields it stores its local variable state and returns the control to the caller. And when it is again called, it starts from where it left in the last call. A normal function doesn't support this behaviour.

Now let's talk about the differences between a generator and a list. The biggest difference which is also the biggest advantage of a generator is that the generator never stores any values; it just yields values one by one therefore saving space as well as time, which is not in the case of a list.

Python Generators with a Loop

As a generator return an iterator object hence we can use loops like for loop to iterate over it and print the yielded values.

Let's take a simple example:

Generator Expressions in Python

Just like list comprehension, we can use expressions to create python generator's shorthand. For example,

data = [0, 1, 2, 3, 4]
# creating a generator using generator expression
new_generator = (x*x for x in range(5))

for each in new_generator:
    print(each)

0 1 4 9 16

In the above example, the generator expression generates an iterator which is similar to the list data with each of its values squared. Instead of creating a list of squares we have directly printed the result without the need to store a single value.