Lesson 06
Just Enough Python
Python for people who have never programmed, on real orders data: read and write simple Python in two browser labs, then run your first pandas script on your own computer.
Essential Python for data work, written for people who have never programmed.
What you’ll be able to do
By the end of this lesson you’ll answer simple questions about a table of orders with Python: how many orders came in, which ones were big, and how much money each country brought in. You’ll:
- know where to write Python and how to run it,
- read short Python code and say what it does,
- clean a messy CSV file and save the result as a JSON report,
- run a pandas script on your own computer,
- know what to learn now and what can wait.
This is not a full Python course. It’s the slice you need to start, plus a map for later.
About the video
The video is a fast tour of the whole Python world for data work, from core Python to Spark and cloud tools. This written lesson goes slower and in a different order, built for someone who has never programmed. If the video feels fast, that’s fine: follow the text and the labs, and come back to the video later for the big picture.
How this lesson works
You’ll switch between reading and doing:
- Read Where Python code runs and Python basics.
- Do Lab 1 in your browser.
- Read Files, CSV and JSON.
- Do Lab 2.
- Set up Python on your computer, learn pandas basics, and run your summary script.
The labs open in a new tab, so this page stays where you left it. They need nothing installed, and they tick tasks off automatically. Everything after the summary script is optional reference.
Pacing. If Python is new to you, plan on roughly 2 to 3 hours for the browser part (steps 1 to 4), and less if you’ve programmed before. Setting up Python and running the summary script (step 5) is a separate 20 to 30 minutes. You don’t have to do it all at once: the ends of Lab 1 and Lab 2 are natural places to stop. Your lab code and ticks are saved in this browser, and labs you mark complete are saved to your account.
Start with the problem, not the code
Before any Python, one mindset shift: your job is to solve business problems, not to write code. Python (or SQL, or any other tool) is only a way to get there.
When the business comes to you with a problem, first understand what they actually need and why. Then pick the simplest way to solve it. Sometimes that’s a Python script, sometimes a SQL query, sometimes a spreadsheet. Nobody gets promoted for lines of code; they get promoted because the problem went away.
Why Python for data work
Python is the most common language in data jobs. Its code is short and fairly readable, and there’s
a library for most tasks: pandas for tables, requests for web APIs, matplotlib for charts,
scikit-learn for machine learning. Tools you’ll meet later in this course, such as Airflow, dbt,
PySpark and Jupyter, are built on Python or work with it, so the same skill carries over.
There are other languages. Scala, for example, shows up in some big-data teams, mostly around Spark, but it’s harder to learn and less common in job postings. Python is the best place to start. If you ever need a second language, the ideas carry over.
Where Python code runs
Python code is plain text. To run it, you hand it to the Python program, which reads it from top to bottom and does what each line says. You’ll meet three places to do that.
1. An editor with a Run button (the labs). Each lab has an editor holding a file called
main.py. You type code, press Run (or Ctrl+Enter, ⌘+Enter on a Mac), and anything your code
prints appears in the Output area. Every Run starts the file again from the top.
2. A .py file and a terminal (your own computer). You save code in a file such as
summary.py, then run it by typing the command python summary.py in a terminal. The terminal is
where you type commands like cd, ls, python and pip (the CLI
lesson covers them). The .py file is where Python code goes. If you type
Python code such as print("hi") straight into a terminal, you’ll get an error, because the terminal
isn’t Python.
3. A notebook (Jupyter, Databricks, Google Colab). Code lives in cells that you run one at a time, with the output under each cell. Notebooks are popular for exploring data. You’ll meet them later in the course; this lesson uses scripts.
Scripts need print()
A script only shows what you ask it to show.
Python
2 + 2
print(2 + 2)
Output
4
The first line works out 4 and throws it away. Only print(...) puts a value on the screen.
Notebooks show the value of a cell’s last line automatically, which is why notebook code often has
no print. In scripts and in the labs, print whatever you want to see.
How to read the code blocks here
Each code block in this lesson has a label:
- Python: code for the lab editor or a
.pyfile. - Terminal: commands for a terminal, not for Python.
- Output: what you should see after running the code. Don’t type these.
Where to try the examples
- Python basics examples run in any Python editor. The cleanest place is the free playground at the top of the Just Enough Python course page: replace its demo code with the example and press Run code. Nothing you do there affects the labs.
- Files, CSV and JSON examples need
orders.csv, which only Lab 2 has. - pandas examples need pandas, which the browser labs don’t have. They belong in
explore.pyandsummary.pyon your own computer, after the setup.
Each example is standalone unless its label says add below. Those build on the example before them, so run them together, in order.
To try an example inside a lab, paste it at the very bottom of main.py, press Run, read the
Output, then delete it before you go on. Some examples create their own small orders list. While
pasted below the lab’s data, that list replaces the lab’s orders for the run, so the lab’s tasks
won’t add up until you delete it. Your earlier answers stay as they are. A few examples are close to
a lab task, so pasting one may tick that task. You’ll learn more by writing the lab answers yourself.
Python basics
Everything Lab 1 needs, in the order you need it.
Comments, print and variables
Python
# A line that starts with # is a comment: a note for humans. Python skips it.
customer = "Ana"
amount = 120.0
print(customer)
print(customer, "paid", amount)
Output
Ana
Ana paid 120.0
customer = "Ana" creates a variable: a name (customer) that points to a value ("Ana").
Think of it as a labelled cell in a spreadsheet. print can take several values separated by commas
and prints them with a space between each.
Basic types
Every value has a type:
"Ana"is text, called a string (str). Text always goes in quotes, single or double.3is a whole number (int).120.0is a decimal number (float).TrueandFalseare booleans (bool): the answer to a yes/no question.Nonemeans “no value”, like an empty cell.
Quotes matter. "120" is text and 120 is a number, and they behave differently:
Python
print("120" + "5")
print(120 + 5)
print(float("120.5") + 1)
Output
1205
125
121.5
Adding two strings glues them together. Data read from files arrives as text, so you’ll often
convert it first: float("120.5") turns the text into the number 120.5.
Arithmetic and comparisons
Python
price = 80.0
quantity = 3
total = price * quantity
print(total)
print(total > 200)
print(total == 240)
Output
240.0
True
True
+,-,*and/add, subtract, multiply and divide.>,<,>=(at least) and<=(at most) compare two values and giveTrueorFalse.!=means “not equal”.
= is not ==. A single = stores a value: total = 240 means “put 240 in total”. A
double == asks a question: total == 240 means “is total equal to 240?”. Mixing them up is one
of the most common beginner mistakes. Writing if total = 240: gives a SyntaxError.
Lists: several values in order
Python
customers = ["Ana", "Ben", "Chloe"]
print(len(customers))
print(customers[0])
print(customers[2])
Output
3
Ana
Chloe
A list is written in square brackets, with values separated by commas. Think of it as one column
of a spreadsheet. len(...) counts the items.
customers[0] gets an item by its position. Positions start at 0, so [0] is the first item and
[2] is the third. Asking for customers[3] gives an IndexError, because there is no fourth item.
Dictionaries: values with names
Python
order = {"order_id": 1, "customer": "Ana", "amount": 120.0}
print(order["customer"])
print(order["amount"])
Output
Ana
120.0
A dictionary (dict) is written in curly braces and holds key: value pairs. Think of it as one
row of a spreadsheet: the keys are the column names, the values are the cells. You read a value by
putting its key in square brackets. Asking for a key that doesn’t exist gives a KeyError.
A list of dicts is a table
Put rows (dicts) in a list and you have a table. Lab 1 uses this shape, and it’s also what you get from CSV files and web APIs.
Python
orders = [
{"order_id": 1, "customer": "Ana", "amount": 120.0},
{"order_id": 2, "customer": "Ben", "amount": 35.5},
]
print(orders[1])
print(orders[1]["customer"])
Output
{'order_id': 2, 'customer': 'Ben', 'amount': 35.5}
Ben
Read orders[1]["customer"] from left to right: take orders, get the row at position 1 (the
second row), then get the value under "customer".
Conditions: if, colons and indentation
Python
amount = 150.0
if amount > 100:
print("big order")
else:
print("small order")
print("done")
Output
big order
done
- The
ifline ends with a colon (:). It means “the lines that belong to me start on the next line”. - Those lines are indented by 4 spaces. Python uses indentation, not brackets, to know which lines
belong to the
if. In the labs, Tab inserts 4 spaces. else:runs when the condition is false. For more cases, addelif another_condition:between them.print("done")isn’t indented, so it runs either way.
A missing colon gives a SyntaxError, and uneven indentation gives an IndentationError. Both are
normal while learning, and the error message tells you the line number.
Loops: do something for every row
Python
orders = [
{"order_id": 1, "customer": "Ana", "country": "US", "amount": 120.0},
{"order_id": 2, "customer": "Ben", "country": "DE", "amount": 35.5},
{"order_id": 3, "customer": "Chloe", "country": "US", "amount": 210.0},
]
for order in orders:
print(order["customer"])
Output
Ana
Ben
Chloe
for order in orders: runs the indented lines once for every item in the list. Each time round,
order holds the current row. You choose the name: for row in orders: works the same way.
Put an if inside the loop to pick out some rows:
Python (add below the previous example)
for order in orders:
if order["amount"] > 100:
print(order["order_id"])
Output
1
3
The print is indented twice: it belongs to the if, which belongs to the loop.
A running total. Start at 0 and add each amount:
Python (add below the previous example)
total = 0
for order in orders:
total = total + order["amount"]
print(total)
Output
365.5
total = total + order["amount"] means: take the old total, add this order’s amount, and store the
result back in total. You’ll also see the shorter form total += order["amount"], which does the
same.
A new list. Start with an empty list [] and .append() the rows you want to keep:
Python (add below the previous example)
big = []
for order in orders:
if order["amount"] > 100:
big.append(order)
print(len(big))
Output
2
A count per group. Use a dictionary as the running total, with one entry per country:
Python (add below the previous example)
counts = {}
for order in orders:
country = order["country"]
counts[country] = counts.get(country, 0) + 1
print(counts)
Output
{'US': 2, 'DE': 1}
counts.get(country, 0) returns the count so far, or 0 if this country hasn’t appeared yet. Plain
counts[country] would give a KeyError the first time a country appears.
Functions
Python
def is_big(order):
return order["amount"] > 100
print(is_big({"amount": 150.0}))
print(is_big({"amount": 50.0}))
Output
True
False
defdefines a function: a named piece of code you can reuse.orderis a parameter, the name the function gives to whatever input you pass in.returnsends a result back. Withoutreturn, a function gives backNone.- Defining a function doesn’t run it. Calling it with brackets, as in
is_big(...), does. printshows a value on the screen.returnhands it back to your code so you can store or reuse it.
Optional: list comprehensions
You’ll often see a shorter way to build a filtered list. You don’t need it for the labs; an ordinary loop works everywhere.
Python
amounts = [120.0, 35.5, 210.0]
big_amounts = [a for a in amounts if a > 100]
print(big_amounts)
Output
[120.0, 210.0]
Read it as “a, for each a in amounts, if a is over 100”. It does the same as the loop with
.append() above. Use whichever you find clearer.
Lab 1: practise on orders data
Now do Lab 1
Nine small tasks on a list of orders: count them, read values, loop, filter, add up a total, keep the paid orders, write a function, and count orders per country. Each task has a Cheatsheet with the syntax, a hint, the expected result, and a solution. Try each task before opening the solution. Open Lab 1 in a new tab →
When all tasks are ticked, mark the lab complete and come back here for files. This is a good place to stop for the day; next time, start at Files, CSV and JSON.
Files, CSV and JSON
Real data usually arrives in files. Lab 2 works with this file, orders.csv:
Output (the contents of orders.csv)
order_id,customer,country,amount,status
1,Ana,US,120.00,paid
2,Ben,DE,35.50,paid
3,Chloe,FR,210.00,refunded
4,Ana,US,80.00,paid
5,Dev,DE,N/A,paid
6,Ben,DE,12.50,cancelled
7,Emma,US,99.00,paid
8,Chloe,FR,300.00,paid
9,Farid,UK,45.00,paid
A CSV (comma-separated values) file is a table stored as plain text. The first line is the
header with the column names, each following line is one row, and commas separate the cells. Notice
order 5: its amount is N/A, not a number. Real exports often look like this.
Why the numbers change between exercises
This file is close to Lab 1’s data but not the same. Orders 1 to 8 have the same customers, countries
and statuses, but order 5’s amount is N/A here instead of 150.0, and there’s a ninth order (Farid,
UK, 45.00). So Lab 1 adds up 8 amounts to 1007.0, while Lab 2 has 8 valid amounts out of 9 rows,
adding up to 902.0. The orders.csv you download for pandas later is Lab 1’s clean eight orders
again. The small examples in Python basics use their own two- or three-order samples.
The examples in this section expect orders.csv in the same folder as the script. In the Lab 2
editor it’s already there.
Open a file with with
Python
with open("orders.csv") as f:
print(f.read())
Output
order_id,customer,country,amount,status
1,Ana,US,120.00,paid
2,Ben,DE,35.50,paid
3,Chloe,FR,210.00,refunded
4,Ana,US,80.00,paid
5,Dev,DE,N/A,paid
6,Ben,DE,12.50,cancelled
7,Emma,US,99.00,paid
8,Chloe,FR,300.00,paid
9,Farid,UK,45.00,paid
open("orders.csv")opens the file calledorders.csvin the current folder.with ... as f:opens the file, calls itfinside the indented lines, and closes it when those lines end, even if an error happens.f.read()returns the whole file as one string.
Read rows with csv.DictReader
Python
import csv
with open("orders.csv") as f:
rows = list(csv.DictReader(f))
print(len(rows))
print(rows[0])
Output
9
{'order_id': '1', 'customer': 'Ana', 'country': 'US', 'amount': '120.00', 'status': 'paid'}
import csvloads thecsvmodule. It comes with Python, so there’s nothing to install.csv.DictReader(f)reads the header, then turns every following line into a dict keyed by column name: the same list-of-dicts shape as in Lab 1.list(...)collects all the rows into a list while the file is still open.- Every value is text:
'120.00'is in quotes. To do maths with it, convert it first.
Convert text to numbers and handle errors
Python
print(float("120.00") + 1)
print(float("N/A"))
Output
121.0
Traceback (most recent call last):
File "main.py", line 2, in <module>
print(float("N/A"))
^^^^^^^^^^^^
ValueError: could not convert string to float: 'N/A'
The second line fails. Read an error from the bottom up: the last line says what went wrong (a
ValueError, because "N/A" isn’t a number), and the lines above say where: line 2 of main.py.
On your computer the file path is longer, and older Python versions don’t show the ^^^ markers. The labs show a
one-line version, such as ValueError: could not convert string to float: 'N/A' (main.py, line 2).
To skip bad values instead of crashing, use try and except:
Python
for text in ["120.00", "N/A", "35.50"]:
try:
amount = float(text)
except ValueError:
print("skipping", text)
continue
print("amount:", amount)
Output
amount: 120.0
skipping N/A
amount: 35.5
- Python runs the lines under
try:. If aValueErrorhappens there, it jumps to the lines underexcept ValueError:instead of stopping the script. continueskips the rest of this round of the loop and moves on to the next item.- Name the error you expect, here
ValueError, rather than catching everything. That way real bugs still show up.
Write JSON
JSON is a text format for data. It looks almost like a Python dict, and it’s what most web APIs send and receive.
Python
import json
summary = {"customers": 3, "countries": ["US", "DE"]}
with open("summary.json", "w") as f:
json.dump(summary, f, indent=2)
with open("summary.json") as f:
print(f.read())
Output
{
"customers": 3,
"countries": [
"US",
"DE"
]
}
open("summary.json", "w")opens the file for writing. It creates the file, or replaces it if it already exists.json.dump(data, f, indent=2)writes the dict into the file.indent=2spreads it over several lines so people can read it. A Python list becomes a JSON list.json.load(f)does the opposite: it reads a JSON file back into a dict.
Lab 2: clean a CSV and save a report
Now do Lab 2
Six small tasks on the orders.csv above: print the file, read it with csv.DictReader, spot the
N/A, skip it with try/except, add up the clean amounts, and save report.json. Files your script
writes appear as new tabs in the lab.
Open Lab 2 in a new tab →
When all tasks are ticked, mark the lab complete and come back here to set up Python on your own computer. This is another good place to stop; next time, start at Set up Python on your computer.
Set up Python on your computer
The labs run Python inside your browser. Real work happens on your own computer or on a server, and pandas, the library for tables, isn’t available in the labs. This part sets up Python the standard way: Python from python.org, a virtual environment (venv), and pip. Expect it to take 20 to 30 minutes the first time.
On Windows, if you set up WSL in the CLI lesson, you can follow the Mac and Linux commands inside WSL instead of using PowerShell.
1. Check whether Python is installed
Terminal (Mac, Linux, WSL)
python3 --version
Terminal (Windows PowerShell)
py --version
You should see something like Python 3.12.4. Any version from 3.10 up is fine. If you get
“command not found” or “not recognized”, install Python from
python.org/downloads. On Windows, tick Add python.exe to
PATH on the installer’s first screen. Then close the terminal, open a new one, and check again.
2. Make a project folder
Terminal (any system)
mkdir python-orders
cd python-orders
Keep everything for this exercise in this folder.
3. Create a virtual environment
A virtual environment is a private set of packages for one project. Packages you install for this project won’t clash with other projects.
Terminal (Mac, Linux, WSL)
python3 -m venv .venv
Terminal (Windows PowerShell)
py -m venv .venv
This creates a folder called .venv inside python-orders. You only do this once per project.
4. Activate it
Terminal (Mac, Linux, WSL)
source .venv/bin/activate
Terminal (Windows PowerShell)
.venv\Scripts\Activate.ps1
Your prompt now starts with (.venv). While it’s active, python and pip refer to this project’s
copies on every system. Activate again each time you open a new terminal for this project.
deactivate switches it off.
5. Install pandas
Terminal (any system, with (.venv) active)
python -m pip install pandas
python -m pip makes sure pip installs into the Python you’re using, which is the venv’s.
6. Get the data
Download orders.csv and save it into the
python-orders folder. It holds the same eight orders as Lab 1, without Lab 2’s broken row. Or
download it from the terminal, inside the python-orders folder:
Terminal (Mac, Linux, WSL)
curl -O https://surfalytics.com/files/python/orders.csv
Terminal (Windows PowerShell)
curl.exe -O https://surfalytics.com/files/python/orders.csv
Check that it’s there with ls (Mac, Linux, WSL) or dir (PowerShell).
7. Write and run a script
Open the python-orders folder in a code editor, such as VS Code from the
VS Code lesson. Create a file called explore.py and paste the examples
from pandas basics into it as you go. Run it from the terminal, in the
python-orders folder with (.venv) active:
Terminal (any system)
python explore.py
pandas basics
pandas is the most widely used Python library for tables. Its main object is the DataFrame: a
table with named columns and numbered rows, like one sheet of a spreadsheet. This is the
orders.csv you just downloaded:
| order_id | customer | country | amount | status |
|---|---|---|---|---|
| 1 | Ana | US | 120.0 | paid |
| 2 | Ben | DE | 35.5 | paid |
| 3 | Chloe | FR | 210.0 | refunded |
| 4 | Ana | US | 80.0 | paid |
| 5 | Dev | DE | 150.0 | paid |
| 6 | Ben | DE | 12.5 | cancelled |
| 7 | Emma | US | 99.0 | paid |
| 8 | Chloe | FR | 300.0 | paid |
Load and look
Python · explore.py
import pandas as pd
orders = pd.read_csv("orders.csv")
print(orders.head(3))
print(len(orders))
Output
order_id customer country amount status
0 1 Ana US 120.0 paid
1 2 Ben DE 35.5 paid
2 3 Chloe FR 210.0 refunded
8
import pandas as pdloads pandas under the short namepd. Almost everyone uses that name.pd.read_csv("orders.csv")reads the whole file into a DataFrame. Unlike thecsvmodule, it turns numbers into numbers for you..head(3)gives the first 3 rows. The numbers on the left (0, 1, 2) are row labels that pandas adds.len(orders)counts the rows.
Work with a whole column
orders["amount"] picks one column. Maths on a column applies to every row at once, with no loop:
Python · explore.py (add below)
print(orders["amount"].sum())
print(orders["amount"].max())
Output
1007.0
300.0
Filter rows in two steps
Keeping only the big orders takes two steps. Step 1: compare the column with 100. pandas answers the question for every row:
Python · explore.py (add below)
is_big = orders["amount"] > 100
print(is_big)
Output
0 True
1 False
2 True
3 False
4 True
5 False
6 False
7 True
Name: amount, dtype: bool
Step 2: put those answers inside orders[...]. pandas keeps the rows marked True:
Python · explore.py (add below)
big = orders[is_big]
print(big)
Output
order_id customer country amount status
0 1 Ana US 120.0 paid
2 3 Chloe FR 210.0 refunded
4 5 Dev DE 150.0 paid
7 8 Chloe FR 300.0 paid
You’ll usually see both steps on one line: big = orders[orders["amount"] > 100]. It’s the same
thing: the inner part makes the True/False answers, the outer orders[...] keeps the matching rows.
If you know SQL, this is WHERE amount > 100.
Group rows and add them up
“Revenue per country” means: split the orders into one pile per country, add up the amounts in each pile, and show one line per country. A spreadsheet pivot table does the same. In pandas:
Python · explore.py (add below)
paid = orders[orders["status"] == "paid"]
groups = paid.groupby("country")
by_country = groups["amount"].sum()
print(by_country)
Output
country
DE 185.5
FR 300.0
US 299.0
Name: amount, dtype: float64
- The first line keeps only the paid orders, with the same filter as above.
paid.groupby("country")makes the piles, one per country.groups["amount"].sum()adds up theamountcolumn within each pile. The result is sorted by country.
In SQL, this is GROUP BY country. In code you’ll often see it on one line:
paid.groupby("country")["amount"].sum().
Your summary script
This is the last step of the lesson. Create a new file, summary.py, in the python-orders folder:
Python · summary.py
import pandas as pd
orders = pd.read_csv("orders.csv")
paid = orders[orders["status"] == "paid"]
print("Orders:", len(orders))
print("Paid orders:", len(paid))
print("Paid revenue:", paid["amount"].sum())
print()
print("Paid revenue by country:")
groups = paid.groupby("country")
print(groups["amount"].sum())
Terminal (in python-orders, with (.venv) active)
python summary.py
Output
Orders: 8
Paid orders: 6
Paid revenue: 784.5
Paid revenue by country:
country
DE 185.5
FR 300.0
US 299.0
Name: amount, dtype: float64
Check this yourself
If your terminal shows this summary, you’ve done the local part of Done when. The site can’t see your computer, so this step isn’t tracked: you check it yourself. The two labs are tracked automatically. When both are done and this script runs, mark the lesson complete.
When something goes wrong
python3: command not found, or'py' is not recognized. Python isn’t installed, or the terminal was opened before you installed it. Install it, then open a new terminal.Activate.ps1 cannot be loaded because running scripts is disabled on this system(Windows). RunSet-ExecutionPolicy -Scope CurrentUser RemoteSignedonce in PowerShell, answerY, and activate again.ensurepip is not availablewhen creating the venv (Ubuntu or WSL). Install the venv package withsudo apt install python3-venv, then create the venv again.ModuleNotFoundError: No module named 'pandas'. The venv isn’t active in this terminal: there’s no(.venv)at the start of the prompt. Activate it and run the script again. If the error stays, runpython -m pip install pandaswith the venv active.FileNotFoundError: [Errno 2] No such file or directory: 'orders.csv'. The terminal is in a different folder from the CSV. Check where you are withpwd, list files withlsordir, andcdintopython-orders.SyntaxErrororIndentationError. Look at the line number in the message. Check for a missing colon, an unclosed quote or bracket, or indentation that isn’t 4 spaces.- A
>>>prompt appears. You typedpythonon its own, which starts Python’s interactive mode. Typeexit()to get back to the terminal, then runpython summary.py. - Python code typed into the terminal gives a strange error. Lines such as
import pandas as pdbelong in the.pyfile, not in the terminal.
Other setup tools you’ll hear about (optional)
You don’t need any of these for this lesson. venv and pip work everywhere, and the other tools solve the same problems in different ways:
- uv: a newer, fast tool that installs packages, creates virtual environments, and records exact versions in a lockfile. Increasingly common in new projects.
- Poetry: manages a project’s packages in
pyproject.tomlwith a lockfile. Common in existing projects. - Conda: environments that also include Python itself and non-Python libraries. Common in data science.
- pyenv: installs several Python versions side by side and switches between them per folder.
Terminal (illustration: each line needs that tool installed first)
uv add pandas # uv project: install and record pandas
poetry add pandas # Poetry project: the same
conda create -n orders python=3.12 pandas # Conda: new environment with Python and pandas
pyenv install 3.12 && pyenv local 3.12 # pyenv: this folder uses Python 3.12Later: the wider Python toolbox
You now have the part of Python that the rest of the course builds on. Everything below is reference: none of it is needed for the labs or the next lessons. Skim the names now so they’re familiar, and come back when a project needs one.
The code examples below are illustrations. They need extra packages, and some need a database, cloud account, credentials or running service, so they won’t work if pasted as they are. Each one says what it needs.
Know these exist
Databases
Most company data lives in a database, not a CSV. These libraries let Python query it:
- SQLAlchemy: the standard toolkit for SQL databases.
pd.read_sqlaccepts a SQLAlchemy engine. - psycopg2: the PostgreSQL driver.
- pymongo: the MongoDB driver.
- pyodbc: connects to databases through ODBC drivers (SQL Server and others).
Python (illustration: needs pandas, sqlalchemy and psycopg2-binary installed, plus a running
PostgreSQL database and real connection details)
import pandas as pd
from sqlalchemy import create_engine
engine = create_engine("postgresql+psycopg2://user:password@host:5432/db")
df = pd.read_sql("SELECT * FROM orders", engine)Web APIs
Many data sources are web APIs that return JSON. requests is the usual library for calling them.
Python (illustration: needs requests installed; the URL is a placeholder, and real APIs usually
need an API key)
import requests
response = requests.get("https://api.example.com/orders", timeout=30)
orders = response.json() # the JSON text turned into Python lists and dictsOther file formats
The same table can be saved in different formats, and the choice affects size, speed and how easy it is to share:
- CSV: plain-text table. Opens anywhere, but has no column types and gets big.
- JSON: nested key-value text, and what APIs return.
- Parquet: columnar and compressed, so it’s usually much smaller and faster to read than CSV. The standard for analytics and data lakes.
- Avro: row-based, with the schema stored in the file. Common in streaming systems such as Kafka.
Python (illustration: needs pandas and pyarrow installed, and an orders.json file)
import pandas as pd
df = pd.read_json("orders.json")
df.to_parquet("orders.parquet")Revisit later
More Python syntax: tuples, sets, lambda, sorting, default parameters
Python (runnable)
point = (40.7, -74.0) # tuple: like a list, but can't be changed
countries = {"US", "DE", "US"} # set: unique values only
print(countries) # {'US', 'DE'} (order may vary)
def add_tax(amount, rate=0.2): # rate has a default value
return amount * (1 + rate)
print(add_tax(100)) # 120.0
print(add_tax(100, rate=0.1)) # 110.00000000000001 (decimals aren't exact in floats)
items = [{"name": "pen", "price": 3}, {"name": "cup", "price": 1}]
cheapest_first = sorted(items, key=lambda item: item["price"])
print(cheapest_first[0]["name"]) # cupA lambda is a one-line function without a name. It’s most often used to tell sorted() which
value to sort by.
Faster table tools: NumPy, Polars, PyArrow
- NumPy: fast arrays of numbers. An operation applies to the whole array at once with no Python loop (“vectorization”). pandas is built on top of it.
- Polars: an alternative to pandas that uses all your CPU cores and is often faster on large data. The syntax differs, but the ideas (filter, group, join) are the same.
- PyArrow: the Python side of Apache Arrow, a columnar format for data in memory. You rarely call it directly, but it’s behind Parquet support in pandas and Polars.
Python (illustration: needs numpy and polars installed, and orders.csv)
import numpy as np
import polars as pl
prices = np.array([10, 25, 40])
print(prices * 1.2) # every element at once, no loop
orders = pl.read_csv("orders.csv")
print(orders.filter(pl.col("amount") > 100))Bigger data and pipelines: PySpark, Dask, Airflow
pandas runs on one machine and needs the data to fit in memory. When data outgrows that, or a job has to run every night without you, you need tools for scale and scheduling.
- PySpark is the Python API for Apache Spark, which splits data across many machines (a cluster) and processes the parts in parallel. Spark DataFrames feel like pandas or SQL. Go deeper with the hands-on PySpark course.
- Dask processes data larger than memory, on a laptop or a cluster, with an API that mirrors
pandas. It does nothing until you call
.compute(). - Apache Airflow schedules and runs pipelines. A pipeline is a DAG: a set of tasks with an order between them, plus a schedule.
Python (illustration: needs a Spark installation or cluster, and access to the S3 bucket)
from pyspark.sql import SparkSession, functions as F
spark = SparkSession.builder.appName("orders").getOrCreate()
orders = spark.read.parquet("s3://bucket/orders/")
orders.filter(F.col("amount") > 100).groupBy("country").agg(F.sum("amount")).show()Python (illustration: needs dask[dataframe] installed and a set of orders-*.csv files)
import dask.dataframe as dd
orders = dd.read_csv("orders-*.csv")
print(orders.groupby("country")["amount"].sum().compute())Python (illustration: a DAG file for an Airflow 2 installation, which runs it on schedule; the two functions are placeholders for your own extract and load code)
from datetime import datetime
from airflow import DAG
from airflow.operators.python import PythonOperator
def extract_orders():
... # e.g. download yesterday's orders from an API
def load_to_warehouse():
... # e.g. write them into a database table
with DAG("daily_orders", start_date=datetime(2026, 1, 1), schedule="@daily") as dag:
extract = PythonOperator(task_id="extract", python_callable=extract_orders)
load = PythonOperator(task_id="load", python_callable=load_to_warehouse)
extract >> load # extract runs first, then loadCloud SDKs and Docker
Each cloud has its own Python library:
- boto3: AWS (S3 files, Redshift, Glue, Lambda and more).
- google-cloud-storage: files in Google Cloud Storage.
- azure-storage-blob: files in Azure Blob Storage.
- Docker packages a script with everything it needs so it runs the same anywhere (see Just Enough Docker).
Python (illustration: needs boto3 installed, AWS credentials configured on your machine, and a
bucket you can read)
import boto3
s3 = boto3.client("s3")
s3.download_file("my-bucket", "orders.csv", "orders.csv")Data validation and testing
A wrong number in a dashboard costs trust that’s hard to win back. These tools catch bad data and broken code before anyone sees them:
- pytest: tests for your own functions, so a change doesn’t silently break the logic.
- Pydantic: checks individual records (API responses, configs) against a typed model.
- Great Expectations: checks on whole datasets (“never empty”, “between 0 and 100”) with reports.
- Pandera: schema checks for pandas and Polars DataFrames.
- DQX: Databricks Labs’ data quality framework for PySpark.
Python (illustration: needs pydantic installed)
from pydantic import BaseModel
class Order(BaseModel):
order_id: int
amount: float
Order(order_id=1, amount="abc") # raises ValidationError: "abc" isn't a numberAdvanced Python: decorators, generators, type hints, async, logging
You won’t write these at first, but you’ll read them in other people’s code.
Decorators wrap a function to add behaviour such as timing, retries or logging. The @ line is
common in Airflow, FastAPI and pytest.
Python (runnable)
import time
def timer(func):
def wrapper():
start = time.time()
result = func()
print(f"{func.__name__} took {time.time() - start:.2f}s")
return result
return wrapper
@timer
def load_data():
time.sleep(0.5)
load_data() # prints: load_data took 0.50sGenerators use yield to hand back values one at a time instead of building a whole list, so a
file far bigger than your memory can be processed line by line.
Python (illustration: needs a file at path)
def read_rows(path):
with open(path) as f:
for line in f:
yield line.strip() # only one line in memory at a timeType hints say which types a function expects and returns. Python doesn’t enforce them, but
editors and tools like mypy use them to catch mistakes.
def add_tax(amount: float, rate: float = 0.2) -> float:
return amount * (1 + rate)async/await lets one program wait on many slow operations, such as API calls, at the same time instead of one after another.
Python (illustration: needs httpx installed and real URLs)
import asyncio
import httpx
async def fetch_all(urls):
async with httpx.AsyncClient() as client:
return await asyncio.gather(*(client.get(u) for u in urls))Logging is print for programs that run without you: messages get levels and timestamps, and
can go to files or monitoring tools.
Python (runnable)
import logging
logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(message)s")
logging.info("Loaded %d rows", 1200)Habits for team code: style, Git, CI/CD, secrets, config
- PEP 8: Python’s official style guide (naming, indentation, line length). A formatter such as
rufforblackapplies it for you. - Git: every script belongs in version control (see the GitHub lesson).
- CI/CD: run tests and checks automatically on every pull request, for example with GitHub Actions.
- Environment variables: keep passwords and API keys out of code. Read them with
os.environ, and load a local.envfile withpython-dotenv. Never commit.env. - Config files: keep settings such as paths and table names in a YAML, TOML or INI file instead of hard-coding them.
Python (illustration: needs python-dotenv installed and a .env file containing DB_PASSWORD)
import os
from dotenv import load_dotenv
load_dotenv() # reads .env into the environment
db_password = os.environ["DB_PASSWORD"]Books, if you want to go deeper
Books are optional. You don’t need to read one before this lesson or before moving on in the course. They’re useful alongside it: when a topic here feels shaky, the matching chapter explains it more slowly, with more exercises. Typing out the examples yourself is what makes the syntax stick.
- Think Python by Allen B. Downey. Teaches you to think like a programmer, not just memorize syntax. Free to read online.
- Python Crash Course by Eric Matthes. Hands-on: the basics first, then small projects that use them.
If you pick one, dip into the chapters that match what you’re working on rather than trying to finish it cover to cover first.
Cheatsheet
| When | What | Where in this lesson |
|---|---|---|
| Learn now | print, variables, types, = vs ==, lists, dicts, if, for, functions | Python basics, Lab 1 |
| Learn now | with open, csv.DictReader, float(), try/except, json.dump | Files, CSV and JSON, Lab 2 |
| Learn now | venv + pip, pandas read_csv, filtering, groupby | Setup, pandas basics, summary script |
| Know they exist | databases (SQLAlchemy), web APIs (requests), Parquet and Avro | Later: Know these exist |
| Revisit later | comprehensions, lambda, NumPy, Polars, PySpark, Dask, Airflow, cloud SDKs, validation, decorators, generators, async, uv/Poetry/Conda/pyenv | Later: Revisit later |
Next steps
Continue with Getting Started with AI in IDE, where an AI assistant helps you write and explain the Python from this lesson.
This lesson is for members only
Join Surfalytics to unlock all 7 lessons in this module, plus every module, project, mock interview, and the private community.
7-day free trial · Cancel any time