What are Python Itertools Functions?

According to Python’s documentation, Itertools is a Python module that provides a set of fast and memory-efficient tools for working with Python iterators. These tools can be used by themselves or in combination, and they make it possible to succinctly and efficiently create and work with iterators in a fast and memory-efficient manner.

The Itertools module contains functions that make it easier to work with iterators, particularly when handling large sets of data. Itertools functions can work on existing iterators to create even more complex Python iterators.

Additionally, Itertools can help developers reduce errors when working with iterators and write cleaner, readable, and maintainable code.

Types of Iterators in Python Itertools

Based on the functionality that the iterators in the Itertools module provide, they can be classified into the following types:

#1. Infinite Iterators

These are iterators that allow you to work with infinite sequences and run a loop infinitely if there’s no condition put in to break out of the loop. Such iterators are useful when simulating infinite loops or generating an unbounded sequence. Itertools has three infinite iterators, which include count(), cycle(), and repeat().

#2. Combinatoric Iterators

Combinatoric Iterators comprise functions that can be used to work on cartesian products and perform combinations and permutations of elements contained within an iterable. These are the go-to functions when trying to find all possible ways to arrange or combine elements in an iterable. Itertools has four combinatoric iterators. These are product(), permutations(), combinations() and combinations_with_replacement().

#3. Iterators Terminating On The Shortest Input Sequence

These are terminating iterators that are used on finite sequences and generate an output based on the type of function used. Examples of these terminating iterators include: accumulate(), chain(), chain.from_iterable(), compress(), dropwhile(), filterfalse(), groupby(), islice(), pairwise(), starmap(), takewhile(), tee(), and zip_longest().

Let us look at how different Itertools functions work according to their type:

Infinite Iterators

The three infinite iterators include:

#1. count()

The count(start, step) function generates an infinite sequence of numbers starting from the start value. The function takes two optional arguments: start and step. The argument start sets where the sequence of numbers should start. By default, it starts at 0 if a start value is not provided. step sets the difference between each consecutive number. The default step value is 1.

import itertools
# count starting at 4, making steps of 2  
for i in itertools.count(4, 2):
    # condition to end the loop avoiding infinite looping
    if i == 14:
        break
    else:
        print(i) # output - 4, 6, 8, 10, 12

Output

4
6
8
10
12

#2. cycle()

cycle(iterable) function takes an iterable as an argument and then cycles through the iterable allowing access to items in the iterable in the order they appear.

For instance, if we pass in [“red”, “green”, “yellow”] into cycle(), in the first cycle, we’ll have access to “red”; in the second cycle we’ll have access to “green”, then “yellow”. In the fourth cycle, since all elements have been exhausted in the iterable, we’ll start over at “red” and then go on infinitely.

When calling cycle() you store its result in a variable to create an iterator that maintains its state. This ensures the cycle does not start all over every time, giving you access to only the first element.

import itertools

colors = ["red", "green", "yellow"]
# pass in colors into cycle()
color_cycle = itertools.cycle(colors)
print(color_cycle)

# range used to stop the infinite loop once we've printed 7 times
# next() used to return the next item from the iterator
for i in range(7):
    print(next(color_cycle))

Output:

red
green
yellow
red
green
yellow
red

#3. repeat()

repeat(elem,n) takes two arguments, an element to repeat (elem), and the number of times you want to repeat the element(n). The element you want to repeat can be a single value or an iterable. If you don’t pass in, n, the element will be repeated infinitely.

import itertools
   
for i in itertools.repeat(10, 3):
    print(i)

Output:

10 
10
10

Combinatoric Iterators

The combinatoric iterators include:

#1. product()

product() is a function used to compute the cartesian product of the iterable passed to it. If we have two iterables or sets, for example, x = {7,8} and y = {1,2,3}, the cartesian product of x and y will contain all possible combinations of elements from x and y, where the first element is from x and the second one from y. The cartesian product of x and y in this case is [(7, 1), (7, 2), (7, 3), (8, 1), (8, 2), (8, 3)].

product() takes an optional parameter called repeat which is used to compute the cartesian product of an iterable with itself. repeat specifies the number of repetitions for each element from the input iterables when computing the Cartesian product.

For instance, calling product(‘ABCD’, repeat=2) yields combinations such as (‘A’, ‘A’), (‘A’, ‘B’), (‘A’, ‘C’), and so on. If repeat was set to 3, the function would yield combinations such as (‘A’, ‘A’, ‘A’), (‘A’, ‘A’, ‘B’), (‘A’, ‘A’, ‘C’), (‘A’, ‘A’, ‘D’) and so on.

from itertools import product
# product() with the optional repeat argument
print("product() with the optional repeat argument ")
print(list(product('ABC', repeat = 2)))

# product with no repeat
print("product() WITHOUT an optional repeat argument")
print(list(product([7,8], [1,2,3])))

Output

product() with the optional repeat argument 
[('A', 'A'), ('A', 'B'), ('A', 'C'), ('B', 'A'), ('B', 'B'), ('B', 'C'), ('C', 'A'), ('C', 'B'), ('C', 'C')]
product() WITHOUT an optional repeat argument
[(7, 1), (7, 2), (7, 3), (8, 1), (8, 2), (8, 3)]

#2. permutations()

permutations(iterable, group_size) returns all possible permutations of the iterable passed into it. A permutation represents the number of ways elements in a set can be ordered. permutations() takes an optional argument group_size. If group_size is not specified, the permutations generated will be the same size as the length of the iterable passed into the function

import itertools
numbers = [1, 2, 3]
sized_permutations = list(itertools.permutations(numbers,2))
unsized_permuatations = list(itertools.permutations(numbers))

print("Permutations with a size of 2")
print(sized_permutations)
print("Permutations with NO size argument")
print(unsized_permuatations)

Output

Permutations with a group size of 2
[(1, 2), (1, 3), (2, 1), (2, 3), (3, 1), (3, 2)]
Permutations with NO size argument
[(1, 2, 3), (1, 3, 2), (2, 1, 3), (2, 3, 1), (3, 1, 2), (3, 2, 1)]

#3. combinations()

combinations(iterable, size) returns all possible combinations of an iterable of a given length from the elements in the iterable passed into the function. The size argument specifies the size of each combination.

The results are ordered. Combination differs slightly from permutations. With permutation, the order matters, but with combination, the order doesn’t matter. For instance, in [A, B, C] there are 6 permutations: AB, AC, BA, BC, CA, CB but only 3 combinations AB, AC, BC.

import itertools
numbers = [1, 2, 3,4]
size2_combination = list(itertools.combinations(numbers,2))
size3_combination = list(itertools.combinations(numbers, 3))

print("Combinations with a size of 2")
print(size2_combination)
print("Combinations with a size of 3")
print(size3_combination)

Output:

Combinations with a size of 2
[(1, 2), (1, 3), (1, 4), (2, 3), (2, 4), (3, 4)]
Combinations with a size of 3
[(1, 2, 3), (1, 2, 4), (1, 3, 4), (2, 3, 4)]

#4. combinations_with_replacement()

combinations_with_replacement(iterable, size) generates all possible combinations of an iterable of a given length from the iterable passed into the function and allows for repeated elements in the output combinations. The size determines the size of the combinations generated.

This function differs from combinations() in that it gives combinations where an element can be repeated more than once. For instance, you can get a combination such as (1,1) which you can’t with combination().

import itertools
numbers = [1, 2, 3,4]

size2_combination = list(itertools.combinations_with_replacement(numbers,2))
print("Combinations_with_replacement => size 2")
print(size2_combination)

Output

Combinations_with_replacement => size 2
[(1, 1), (1, 2), (1, 3), (1, 4), (2, 2), (2, 3), (2, 4), (3, 3), (3, 4), (4, 4)]

Terminating iterators

This includes iterators such as:

#1. accumulate()

accumulate(iterable, function) takes an iterable and a second optional argument which is a function. It then returns the accumulated result of applying the function in each iteration on elements on the iterable. If no function is passed, addition is done and the accumulated results are returned.

import itertools
import operator
numbers = [1, 2, 3, 4, 5]

# Accumulate the sum of numbers
accumulated_val = itertools.accumulate(numbers)
accumulated_mul = itertools.accumulate(numbers, operator.mul)
print("Accumulate with no function")
print(list(accumulated_val))
print("Accumulate with multiplication")
print(list(accumulated_mul))

Output:

Accumulate with no function
[1, 3, 6, 10, 15]
Accumulate with multiplication
[1, 2, 6, 24, 120]

#2. chain()

chain(iterable_1, iterable_2, …) takes multiple iterables and chains them together producing a single iterable containing values from the iterables passed to the chain() function

import itertools

letters = ['A', 'B', 'C', 'D']
numbers = [1, 2, 3]
colors = ['red', 'green', 'yellow']

# Chain letters and numbers together
chained_iterable = list(itertools.chain(letters, numbers, colors))
print(chained_iterable)

Output:

['A', 'B', 'C', 'D', 1, 2, 3, 'red', 'green', 'yellow']

#3. chain.from_iterable()

chain.from_iterable(iterable)  this function is similar to chain(). However, it differs from the chain in that it only takes a single iterable containing sub-iterables and chains them together.

import itertools

letters = ['A', 'B', 'C', 'D']
numbers = [1, 2, 3]
colors = ['red', 'green', 'yellow']

iterable = ['hello',colors, letters, numbers]
chain = list(itertools.chain.from_iterable(iterable))
print(chain)

Output:

['h', 'e', 'l', 'l', 'o', 'red', 'green', 'yellow', 'A', 'B', 'C', 'D', 1, 2, 3]

#4. compress()

compress(data, selectors) takes in two arguments, data which is an iterable, and selectors which is an iterable containing booleans values true, and false. 1, 0 can also be used as alternatives to the boolean values true and false. compress() then filters the passed data using the corresponding elements passed in the selector.

Values in data that correspond to the value true or 1 in the selector are selected, while the rest which correspond to false or 0 are ignored. If you pass fewer booleans in selectors than the number of items in data all the elements beyond the passed booleans in selectors are ignored

import itertools

# data has 10 items
data = ['A', 'B', 'C', 'D', 'E', 'F', 'G', 'H', 'I', 'J']
# passing in 9 selector items
selectors = [True, False, 1, False, 0, 1, True, False, 1]

# Select elements from data based on selectors
filtered_data = list(itertools.compress(data, selectors))
print(filtered_data)

Output:

['A', 'C', 'F', 'G', 'I']

#5. dropwhile()

dropwhile(function, sequence) takes in a function with the condition that returns true or false and a sequence of values. It then drops all values until the condition passed returns False. Once the condition returns false, the rest of the elements are included in its results regardless of whether they’d return True or False.

import itertools

numbers = [1, 2, 3, 4, 5, 1, 6, 7, 2, 1, 8, 9, 0, 7]

# Drop elements until the passed condition is False
filtered_numbers = list(itertools.dropwhile(lambda x: x < 5, numbers))
print(filtered_numbers)

Output:

[5, 1, 6, 7, 2, 1, 8, 9, 0, 7]

#6. filterfalse()

filterfalse(function, sequence) takes in a function, with a condition that evaluates to true or false and a sequence. It then returns values from the sequence which do not satisfy the condition in the function. 

import itertools

numbers = [1, 2, 3, 4, 2, 3 5, 6, 5, 8, 1, 2, 3, 6, 2, 7, 4, 3]

# Filter elements for which condition is False
filtered_numbers = list(itertools.filterfalse(lambda x: x < 4, numbers))
print(filtered_numbers)

Output:

[4, 5, 6, 5, 8, 6, 7, 4]

#7. groupby()

groupby(iterable, key) takes in an iterable and a key, then makes an iterator that returns consecutive keys and groups. For it to work, the iterable passed to it needs to be sorted on the same key function. The key function computer a key value for each element in the iterable.

import itertools

input_list = [("Domestic", "Cow"), ("Domestic", "Dog"), ("Domestic", "Cat"),("Wild", "Lion"), ("Wild", "Zebra"), ("Wild", "Elephant")]
classification = itertools.groupby(input_list,lambda x: x[0])
for key,value in classification:
  print(key,":",list(value))

Output:

Domestic : [('Domestic', 'Cow'), ('Domestic', 'Dog'), ('Domestic', 'Cat')]
Wild : [('Wild', 'Lion'), ('Wild', 'Zebra'), ('Wild', 'Elephant')]

#8. islice()

islice(iterable, start, stop, step) allows you to slice an iterable using the start, stop, and step values passed. The step argument is optional. Counting starts from 0 and the item on the stop number is not included.

import itertools

numbers = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18]

# Select elements within a range
selected_numbers = list(itertools.islice(numbers, 2, 10))
selected_numbers_step= list(itertools.islice(numbers, 2, 10,2))
print("islice without setting a step value")
print(selected_numbers)
print("islice with a step value of 2")
print(selected_numbers_step)

Output:

islice without setting a step value
[3, 4, 5, 6, 7, 8, 9, 10]
islice with a step value of 2
[3, 5, 7, 9]

#9. pairwise()

pairwise(iterable) returns successive overlapping pairs taken from the iterable passed to it in the order they appear in the iterable. If the iterable passed to it has less than two values, the result from pairwise() will be empty.

from itertools import pairwise

numbers = [1, 2, 3, 4, 5, 6, 7, 8]
word = 'WORLD'
single = ['A']

print(list(pairwise(numbers)))
print(list(pairwise(word)))
print(list(pairwise(single)))

Output:

[(1, 2), (2, 3), (3, 4), (4, 5), (5, 6), (6, 7), (7, 8)]
[('W', 'O'), ('O', 'R'), ('R', 'L'), ('L', 'D')]
[]

#10. starmap()

starmap(function, iterable) is a function used instead of map() when argument parameters are already grouped in tuples. startmap() applies a function to the elements of the iterable passed to it. The iterable should have elements grouped in tuples.

import itertools

iter_starmap = [(123, 63, 13), (5, 6, 52), (824, 51, 9), (26, 24, 16), (14, 15, 11)]
print (list(itertools.starmap(min, iter_starmap)))

Output:

[13, 5, 9, 16, 11]

#11. takewhile()

takewhile(function, iterable) works in the opposite way to dropwhile(). takewhile() takes in a function with a condition to be evaluated and an iterable. It then includes all elements in the iterable which satisfy the condition in the function until False is returned. Once False is returned, all the following elements in the iterable are ignored.

import itertools

numbers = [1, 2, 3, 4, 5, 1, 6, 7, 2, 1, 8, 9, 0, 7]

# Drop elements until the passed condition is False
filtered_numbers = list(itertools.takewhile(lambda x: x < 5, numbers))
print(filtered_numbers)

Output:

[1, 2, 3, 4]

#12. tee()

tee(iterable, n) takes in an iterable and returns multiple independent iterators. The number of iterators to return is set by n, which by default is 2.

import itertools

numbers = [1, 2, 3, 4, 5]

# Create two independent iterators from numbers
iter1, iter2 = itertools.tee(numbers, 2)
print(list(iter1))
print(list(iter2))

Output:

[1, 2, 3, 4, 5]
[1, 2, 3, 4, 5]

#13. zip_longest()

zip_longest(iterables, fillvalue) takes in multiple iterators and a fillvalue. It then returns an iterator that aggregates elements from each of the iterators passed to it. If the iterators are not of the same length, the missing values are replaced by the fillvalue passed to the function until the longest iterable has been exhausted.

import itertools

names = ['John', 'mathew', 'mary', 'Alice', 'Bob', 'Charlie', 'Fury']
ages = [25, 30, 12, 13, 42]

# Combine name and ages, filling in missing ages with a dash
combined = itertools.zip_longest(names, ages, fillvalue="-")

for name, age in combined:
    print(name, age)

Output:

John 25
mathew 30
mary 12
Alice 13
Bob 42
Charlie -
Fury -

Conclusion

Python itertools are an important toolset for a Python developer. Python itertools are used extensively in functional programming, data processing, and transformation, data filtering and selection, grouping and aggregation, combining iterables, combinatorics, and when working with infinite sequences.

As a Python developer, you’ll benefit greatly by learning about itertools so make sure to use this article to familiarize yourself with Python Itertools.