List comprehensions and exceptions

The five fundamentals get you reading code. These two features get you reading real code, because every data codebase is full of them.

List comprehensions

The filter-and-accumulate loops from topic 4, in one line:

salaries = [95000, 72000, 110000, 68000]

# transform every item
in_cad = [s * 1.35 for s in salaries]

# keep some items
high = [s for s in salaries if s > 90000]

# both
high_in_cad = [s * 1.35 for s in salaries if s > 90000]

Read it left to right: “s * 1.35 for each s in salaries, if s > 90000”. It is the same as:

high_in_cad = []
for s in salaries:
    if s > 90000:
        high_in_cad.append(s * 1.35)

Pull a column out of a list of dicts, the most common use in data work:

companies = [o["company"] for o in offers]

The same shape works for dicts and sets:

{o["company"]: o["salary_usd"] for o in offers}     # dict comprehension
{o["country"] for o in offers}                       # set of unique countries

Rule of thumb: one condition, one transform, one line. If it needs two lines to read, write a loop. Interviewers like comprehensions. Colleagues like readable code. Do both.

sum, len, min, max, sorted

Built-ins that replace whole loops:

sum(salaries)                        # 345000
len(salaries)                        # 4
max(salaries)                        # 110000
sum(salaries) / len(salaries)        # the mean
sorted(salaries, reverse=True)       # new list, biggest first
sorted(offers, key=lambda o: o["salary_usd"])

Exceptions: when something goes wrong

Bad data is normal. One row has "n/a" where a number should be. The API returns an error page. A file is missing. Without handling, the job crashes on row 4,012 of 100,000.

try:
    salary = int(text)
except ValueError:
    salary = None

try runs the risky code. If it raises ValueError, the except block runs instead. Nothing crashes.

Catch the specific exception you expect. except Exception: catches everything, including the bugs you would want to see. Common ones:

Exception	When
ValueError	int("n/a")
KeyError	offer["bonus"] and the key is missing
TypeError	"85000" + 1
FileNotFoundError	opening a file that does not exist
ZeroDivisionError	total / 0

finally runs no matter what. Use it to close connections:

try:
    rows = fetch_from_api()
except ConnectionError as e:
    print(f"API failed: {e}")
    rows = []
finally:
    close_connection()

Raise your own when data is wrong and continuing would hide it:

if not rows:
    raise ValueError("Source returned zero rows, refusing to overwrite the table")