Generator in Python

Introduction

Generators allows to create functions that behave like an iterator, and because it creates a sequence of values over time, there is no need to allocate memory to store the entire sequence. This article discusses implementation of generators, and also compares the memory usage.

Sample Scenario

Lets take a simple program, that takes in a list of numbers and returns their square values. The generic code will be as follows,

[1, 4, 9, 16, 25, 36, 49, 64, 81, 100] 
520 bytes

pympler is used to find the memory allocated to the variable. If we do need to result in one sequence, we can implement a generator that does not store the results as a list, and simply yields the squared result for each entry in the list.

1 4 9 16 25 36 49 64 81 100  
128 bytes

Here we can do away with storing the list in memory, which reduces memory requirement by almost 76% in this case. But if the size of input list were to increase, we would still need the 128 bytes as at any given point the generator yeilds only one value.

Lets modify the program to randomly create an input list of ten thousand integers, to further understand impact on memory

without generator

[3878549284, 329495104, ...  ... ...  ,7644879225, 6411525184] 
407632 bytes

with generator,

[3878549284, 329495104, ...  ... ...  ,7644879225, 6411525184] 
128 bytes

Understanding the Generator

#1:  <generator object square at 0x7f11e82bb6d0>
#2:  1
#3:  4
--------------------------------------------------------------------StopIteration                     Traceback (most recent call last)

printing the variable does not return a list, what we get is an generator object as it does not hold the entire result in memory, and only yields one result at a time.

We can however get the result of each yield and that's achieved using the next function, however once the result items are exhausted, a StopIteration error is thrown.

Generator in List Comprehension

An easier way to code the square of a list discussed above is to use the list comprehension

[1, 4, 9, 16, 25, 36, 49, 64, 81, 100]

520 bytes

This can easily be converted to a generator function by simply replacing the outer square brackets, with regular parenthesis

1 4 9 16 25 36 49 64 81 100  
128 bytes

The output of a generator can be converted back to a list using the list keyword, however this again impacts memory

[1, 4, 9, 16, 25, 36, 49, 64, 81, 100] 
488 bytes

Conclusion

Generators can be implemented when the complete result is not required in a single sequence. It allows for lazy evaluation, here the next element of the iteration is only processed when requested, this improves performance in large data sets.