Advanced: Comprehensions

Welcome to the Week 3 Advanced Python Notebook. This notebook is designed for students who already have substantial experience with writing loops in Python and are confidence on the material from the Beginner and Intermediate material.

Your task today is to carefully read through the content and complete the exercises at the end. These exercises are more challenging and are intended to deepen your understanding of how Python handles data behind the scenes.

Important: This notebook is only recommended if you are already very confident with Python. Before beginning, you must have attempted at least \(4\) exercises from both the Beginner and Intermediate notebooks. If you have not done so, please return to those notebooks first, as the material here builds directly on the Beginner material and is substantially more complex than the Intermediate.

In this notebook, you will explore comprehensions. Comprehensions are a short-hand syntax for creating collections from for loops and are extremely useful for condensing long code and improving readbility.

Work through the examples carefully, and take your time with the exercises. They are designed to stretch your understanding and prepare you for advanced applications of Python.

Table of Contents

List Comprehensions

In this section, we shall look at a nice short-hand for creating a list using a for loop known as a list comprehension. To get started let’s consider the following for loop:

Given a list of numbers, numbers, this code looks through the list one-by-one, squaring each element and saving it in the numbers_squared list. This code is nice, but quite long for a fairly simple operation.

The same logic can be written more concisely with a list comprehension as follows:

As can be seen, we still have a lot of the syntax from the original for loop (we keep the keywords for and in for instance), but the code is now much shorter and potentially more readable.

In general the syntax for a list comprehension is as follows:

result = [expression for item in iterable]

Let’s break this down a little. In this syntax, we have: - iterable: The sequence we loop over (e.g. a list, string, range,… etc). - item: The variable representing the current element from the iterable. - expression: What we want to calculate using item. - for... in: These express the for loop itself. - []: This indicates that we are building a list. - result: The variable we save the output to.

Here are two more examples of a list comprehensions:

Test your understanding: Using your knowledge of list comprehensions, explain what the above code is doing. What are the list_comprehension1 and list_comprehension2 lists counting? Why is the last element of list_comprehension2 equal to zero?

List comprehensions are not the only type of comprehension in Python. The same concise syntax can be used to build sets and dictionaries.

Set Comprehensions

A set comprehension looks almost identical to a list comprehension, except that it uses curly braces {} instead of square brackets [].

Notice that sets automatically remove duplicates. For instance, even both \(-2\) and \(2\) appear in the list, \(4=2^2=(-2)^2\) appears only once in the set.

The general syntax for a set comprehension is given by:

result = {expression for item in iterable}

This is identical to the expression for list comprehensions, except the square brackets have been replaced by curley braces.

Dictionary Comprehensions

A dictionary comprehension allows us to generate key–value pairs in a single line. The syntax is very similar to list and set comprehensions, but instead of just an expression, we provide a key: value pair for each item.

To see what’s happening under the hood, here’s the same example written using a standard for loop:

Both versions create a dictionary that stores the fruit name as the key and its length as the value. The comprehension just combines the loop and assignment into a single, concise expression.

Again the syntax is similar here, but this time we have:

result = {key_expression: value_expression for item in iterable}

Notice that we are still using curly braces {}, just like in a set comprehension. The difference here is the colon (:) inside the braces. The colon tells Python we are building key-value pairs for a dictionary rather than a values for a set.

Combining Conditionals and Comprehensions

So far, we’ve seen comprehensions that apply the same operation to every element of an iterable. But often we want to be more selective. We might instead want to:

  • Only include items that meet some condition.
  • Or transform items differently depending on a condition.

Comprehensions let us do both of these by combining them with conditional expressions.

Filtering Items: A conditional at the end of a comprehension can be keep items which satisfy a certain criteria and disregard others. For instance, the below list comprehension keeps only the even numbers:

This comprehension says:

  • loop through the numbers 0–9,
  • check if each one is divisible by 2,
  • and only keep those which are divisible by 2.

The equivalent for loop looks like this:

This same idea also works for sets and dictionaries:

Test your understanding: Make sure you understand the above code before moving on. How would you modify the set comprehension to select odd numbers instead?

Conditional Expressions Inside a Comprehension: Instead of filtering, you can also use a conditional expression inside the comprehension to decide how to transform each item. Here every element is included, but its value depends on the condition.

This comprehension says:

  • loop through the numbers 0–5,
  • use the number as the key,
  • and store "even" or "odd" as the value depending on whether it divides by 2.

The equivalent for loop is given by:

Note: Further information on conditional expressions can be found in the week 2 advanced notebook.

Generator Comprehensions

All the comprehensions we have seen so far (lists, sets, and dictionaries) build the entire collection as soon as you define them. Python loops through the iterable right away, computes every result, and stores them in memory.

That is fine for small data, but if the sequence is very large the computation can use a lot of memory. Often we do not need to keep everything at once. Instead, it is better to produce values only when they are actually needed.

A generator comprehension allows us to do exactly that. It looks just like a list comprehension, but uses round parentheses () instead of square brackets [].

Here, squares is not a list. It is a generator object. Instead of computing and storing all the values at once, it creates them one at a time as you iterate over it. This makes generator comprehensions memory efficient, since nothing is stored unnecessarily. They only ever evaluate on demand, producing values when the program asks for them.

For example:

This calculates the sum of a million squares efficiently because Python generates each square only when the sum() function requests it, and then discards it right away.

Generator comprehensions are preferable when you need to perform computations that are larger than you can fit in memory. However, generator comprehensions can also be slower than list comprehensions, as they require more access to physical memory. In practice, which tool is best will depend upon the task you are doing.

Exercises

Question 1: From range(1, 21), use a comprehension to create a set of the squares of the even numbers only.

Question 2: Below is a large integer.

Using a list comprehension and the sum function, work out the sum of the digits of my_integer.

Hint: Convert the integer to a string first and recall that strings are iterable!

Question 3: In this question we shall call an integer that equals the sum of the cubes of it’s digits “pluriperfect”. For example \(153\) is pluriperfect as \(153=1^3+5^3+3^3\). Using the range function together with list comprehensions, compute all pluriperfect numbers between \(0\) and \(999\).

Hint: For this question, you can use the sum function, which sums the items in a list. You may find that you need more than one list comprehension for this task!

Question 4: Below is a long string:

Using an appropriate choice of comprehension and conditional statements, compute and print the unique vowels in my_string. Your code should account for spaces, punctuation, numbers and case.

Hint: You might want to consider the lower function from week 1 and the membership test in "aeiou".

Question 5: The built-in function max() can be applied to any generator or iterable covered in this notebook. It returns the largest value in a numerical collection. Suppose you want to modify your code from question 3 to find the largest pluriperfect number less than 10000000000. Which comprehension would be most appropriate for this task, and why? You do not need to implement your answer for this question.