Just Enough Python

Essential Python skills for data work, data analysis, machine learning, and AI.

What this video covers

Python is a huge language, and you only need a slice of it for data work. This lesson maps out that slice: why Python, how to learn it, and the ten areas you’ll actually use on the job, from basic syntax to Spark, cloud storage, and clean project setup. Each area below gets a short explanation and a small code example you can copy. The examples mostly use one imaginary orders table, so you can see how the pieces connect.

Start with the problem, not the code

Before any Python, one mindset shift: your job is to solve business problems, not to write code. Python (or SQL, or any other tool) is only a way to get there.

When the business comes to you with a problem, first understand what they actually need and why. Then pick the simplest way to solve it. Sometimes that’s a Python script, sometimes a SQL query, sometimes a spreadsheet. Nobody gets promoted for lines of code; they get promoted because the problem went away.

Why Python for data roles

Python is the default language for data work. It reads almost like English, it’s quick to learn, and there’s a library for nearly everything: pandas for tables, requests for APIs, matplotlib for charts, scikit-learn for machine learning, and SDKs for every cloud and AI service. Airflow, dbt, PySpark, and Jupyter all speak Python, so the same skill carries you through analytics, data engineering, and ML.

There are alternatives. Scala, for example, shows up in some big-data teams, mostly around Spark. It’s powerful, but harder to learn and far less common in job postings. Python is the best place to start. It covers most data jobs today, and if you ever need a second language, the concepts carry over.

Learn from books

There are plenty of Python video courses, but the recommendation here is to read a book and work through it. A good book goes deeper and in a better order, and typing out the examples yourself is what makes the syntax stick. Two to start with:

• Think Python by Allen B. Downey. Teaches you to think like a programmer, not just memorize syntax. Free to read online.
• Python Crash Course by Eric Matthes. Hands-on and project-based: the basics first, then small projects that put them to use.

Pick one and finish it rather than starting both.

Key topics for data work

Here’s the slice of Python that matters for data roles. Don’t try to learn it all at once: core concepts and pandas come first and cover most day-to-day work. The rest you pick up when a project needs it, and it’s enough to know they exist and what they’re for.

1. Core Python concepts
2. Data manipulation
3. Data processing
4. Database connectivity
5. Data validation and quality
6. Cloud and infrastructure
7. Data formats and serialization
8. Package and environment managers
9. Advanced concepts
10. Best practices

Core Python concepts

The foundation everything else builds on. If a book chapter covers one of these, it’s worth your time.

Variables and data types. A variable is a name for a value. The basic types are text (str), whole numbers (int), decimals (float), True/False (bool), and “nothing” (None).

name = "Surfalytics"   # str
rows = 1200            # int
price = 9.99           # float
is_active = True       # bool

Data structures. Ways to hold many values:

tables = ["orders", "users"]          # list: ordered, changeable
point = (40.7, -74.0)                 # tuple: ordered, can't change
user = {"id": 1, "name": "Ann"}       # dict: key → value lookup
countries = {"US", "CA", "US"}        # set: unique values only → {"US", "CA"}

Control flow. Run code only when a condition is true:

if rows == 0:
    print("empty table")
elif rows < 1000:
    print("small table")
else:
    print("big table")

Functions. Reusable blocks of code. def defines one, parameters go in, return sends a result back. A lambda is a one-line function without a name, handy for short sorts and filters.

def add_tax(price, rate=0.2):
    return price * (1 + rate)

add_tax(100)                          # 120.0

items = [{"name": "pen", "price": 3}, {"name": "cup", "price": 1}]
by_price = sorted(items, key=lambda x: x["price"])   # cup first

List comprehensions. Build a new list from an existing one in a single line:

prices = [10, 25, 40]
with_tax = [p * 1.2 for p in prices]          # [12.0, 30.0, 48.0]
expensive = [p for p in prices if p > 20]     # [25, 40]

Exception handling. Catch errors instead of crashing. try runs the risky code, except handles a failure, finally always runs (for example, to close a connection).

try:
    value = int("abc")
except ValueError:
    value = None
finally:
    print("done")

Modules and packages. import code from libraries or other files. Put your own reusable functions in a .py file and every script in the project can import them.

import pandas as pd
from datetime import date
from my_utils import add_tax   # your own my_utils.py

Virtual environments. Give each project its own set of packages so versions don’t clash between projects. More on the tools in Package and environment managers.

python3 -m venv venv && source venv/bin/activate

Data manipulation

This is where most analysts spend their Python time: loading a table, cleaning it, slicing it, and summarizing it.

pandas is the main library for this. Its core object is the DataFrame: rows and columns, like a spreadsheet or a SQL table, but in code.

import pandas as pd

orders = pd.read_csv("orders.csv")
orders.head()        # first 5 rows
orders.info()        # columns, types, missing values

If you know SQL, you already know the ideas. Only the syntax is new.

Filtering keeps only the rows you need, like SQL’s WHERE:

big = orders[orders["amount"] > 100]
us_paid = orders[(orders["country"] == "US") & (orders["status"] == "paid")]

Grouping aggregates by a column, like GROUP BY:

orders.groupby("country")["amount"].sum()
orders.groupby("country").agg(total=("amount", "sum"), orders=("id", "count"))

Merging combines two tables on a key, like JOIN:

users = pd.read_csv("users.csv")
orders_with_users = orders.merge(users, left_on="user_id", right_on="id", how="left")

how takes the usual join types: "inner", "left", "right", "outer".

pandas isn’t the only option. Three more libraries you’ll run into:

NumPy handles fast arrays of numbers. Its key idea is vectorization: one operation applies to the whole array at once, with no Python for loop, which is much faster on large data. pandas is built on top of it.

import numpy as np

prices = np.array([10, 25, 40])
prices * 1.2            # [12. 30. 48.], every element, no loop
prices.mean()           # 25.0
np.sqrt(prices)         # math on the whole array

Polars is a faster alternative to pandas. It uses all your CPU cores and stays quick on datasets that make pandas crawl. The syntax is different, but the ideas (filter, group, join) are the same.

import polars as pl

orders = pl.read_csv("orders.csv")
orders.filter(pl.col("amount") > 100).group_by("country").agg(pl.col("amount").sum())

PyArrow is the Python side of Apache Arrow, a fast columnar format for data in memory. You rarely call it directly, but it’s the engine behind reading Parquet files, Polars, and newer pandas, and it lets these tools hand data to each other without copying it.

import pyarrow.parquet as pq

table = pq.read_table("orders.parquet")
df = table.to_pandas()

Data processing

pandas runs on one machine and needs the data to fit in memory. When the data outgrows that, or a job has to run every night without you, you need tools for scale and scheduling.

PySpark is the Python API for Apache Spark, which does distributed processing: the data is split across many machines (a cluster) and processed in parallel. You’ll mostly work with Spark DataFrames, which feel like pandas or SQL. RDDs (Resilient Distributed Datasets) are the lower-level building block underneath; you rarely need them directly.

from pyspark.sql import SparkSession, functions as F

spark = SparkSession.builder.appName("orders").getOrCreate()
orders = spark.read.parquet("s3://bucket/orders/")
orders.filter(F.col("amount") > 100).groupBy("country").agg(F.sum("amount")).show()

Go deeper with the hands-on PySpark course.

Dask brings parallel computing to datasets larger than memory, on your laptop or a cluster. Its DataFrame mirrors the pandas API, so it’s a smaller step up than Spark. It splits the data into chunks and does nothing until you call .compute().

import dask.dataframe as dd

orders = dd.read_csv("orders-*.csv")        # many files, more than fits in RAM
orders.groupby("country")["amount"].sum().compute()

Apache Airflow schedules and runs data pipelines. A pipeline is a DAG (directed acyclic graph): a set of tasks with an order between them. Each task is built from an operator (PythonOperator, BashOperator, a SQL operator…), and the DAG’s schedule says when it runs.

from airflow import DAG
from airflow.operators.python import PythonOperator
from datetime import datetime

with DAG("daily_orders", start_date=datetime(2026, 1, 1), schedule="@daily") as dag:
    extract = PythonOperator(task_id="extract", python_callable=extract_orders)
    load = PythonOperator(task_id="load", python_callable=load_to_warehouse)
    extract >> load        # extract runs first, then load

Database connectivity

Most company data lives in a database, not a CSV. These libraries let Python query it directly:

• SQLAlchemy: the standard toolkit for SQL databases, plus an ORM (tables as Python classes). pd.read_sql uses it under the hood.
• psycopg2: the PostgreSQL driver.
• pymongo: the MongoDB driver.
• pyodbc: connects to any database with an ODBC driver (SQL Server, Snowflake, and others).

from sqlalchemy import create_engine

engine = create_engine("postgresql+psycopg2://user:password@host:5432/db")
df = pd.read_sql("SELECT * FROM orders", engine)

Data validation and quality

Bad data is worse than no data: a wrong number in a dashboard costs trust that’s hard to win back. These tools catch problems before anyone sees them:

• Pydantic: validates individual records (API responses, configs) against a typed model.
• Great Expectations: declarative checks on whole datasets (“column is never null”, “values between 0 and 100”) with readable reports.
• Pandera: schema checks for pandas and Polars DataFrames, right in your code.
• Unit testing: test your own functions with pytest, so a change doesn’t silently break the logic.
• DQX: Databricks Labs’ data quality framework for PySpark DataFrames.

from pydantic import BaseModel

class Order(BaseModel):
    id: int
    amount: float

Order(id=1, amount="abc")   # raises ValidationError

Cloud and infrastructure

Modern data lives in the cloud and arrives through APIs. Each cloud has its own Python library:

• boto3: the AWS SDK for S3 files, Redshift, Glue, Lambda, and more.
• google-cloud-storage: read and write files in Google Cloud Storage.
• azure-storage-blob: the same for Azure Blob Storage.
• Docker: package your script with its dependencies so it runs the same anywhere (see Just Enough Docker).
• API requests: pull data from REST APIs with requests.

import boto3, requests

boto3.client("s3").download_file("my-bucket", "orders.csv", "orders.csv")
data = requests.get("https://api.example.com/orders", timeout=30).json()

Data formats and serialization

The same table can be saved in very different ways, and the choice affects size, speed, and how easy it is to share:

• JSON: nested key-value text, and what APIs return. Built-in json module.
• CSV: plain-text table. Opens anywhere, but it’s slow, has no types, and gets big.
• Parquet: columnar and compressed. The standard for analytics and data lakes.
• Avro: row-based with a built-in schema. Common in streaming (Kafka).

df = pd.read_json("orders.json")
df.to_parquet("orders.parquet")    # much smaller and faster to read than CSV

Package and environment managers

“It works on my machine” usually means different library versions. These tools keep every project’s setup clean and repeatable.

Package managers install libraries:

• pip: the default installer that comes with Python.
• uv: a much faster modern replacement for pip, written in Rust. It also creates virtual environments and pins exact versions in a lockfile.
• Poetry: manages a project’s dependencies in pyproject.toml with a lockfile. Popular before uv, still common in existing projects.

pip install pandas
uv add pandas              # install + record in pyproject.toml, many times faster
poetry add pandas          # the same in a Poetry project

Environment managers keep projects isolated from each other:

• venv: built-in virtual environments for packages (the command is in Core Python concepts).
• Conda: environments that also include the Python version and non-Python libraries. Common in data science.
• pyenv: installs and switches between several Python versions on one machine.

pyenv install 3.11 && pyenv local 3.11    # this folder uses Python 3.11
python -m venv venv                        # then a venv on top of it

Advanced concepts

You won’t write these on day one, but you’ll read them in other people’s code sooner than you think.

Decorators wrap a function to add behavior (timing, retries, logging) without changing its code. That @ line is all over Airflow, FastAPI, and pytest.

import time

def timer(func):
    def wrapper(*args, **kwargs):
        start = time.time()
        result = func(*args, **kwargs)
        print(f"{func.__name__} took {time.time() - start:.2f}s")
        return result
    return wrapper

@timer
def load_data():
    ...

Generators use yield to hand back values one at a time instead of building the whole list in memory. That’s how you read a 50 GB file on a laptop with 16 GB of RAM.

def read_rows(path):
    with open(path) as f:
        for line in f:
            yield line.strip()     # one line in memory at a time

Context managers are the with statement: it opens a resource and closes it automatically, even if an error happens. Used for files, database connections, and locks.

with open("orders.csv") as f:
    header = f.readline()
# file is closed here

Type hints say what types a function expects and returns. Python doesn’t enforce them, but your editor and tools like mypy catch mistakes before you run the code.

def add_tax(price: float, rate: float = 0.2) -> float:
    return price * (1 + rate)

Async/await runs many I/O tasks (API calls, downloads) at the same time instead of waiting for each one in turn. Fetching 100 URLs takes about as long as the slowest one, not all 100 added up.

import asyncio, httpx

async def fetch_all(urls):
    async with httpx.AsyncClient() as client:
        return await asyncio.gather(*(client.get(u) for u in urls))

Logging replaces print. Messages get levels and timestamps, and can go to files or monitoring tools, so when a pipeline fails at 3 a.m. you can see why.

import logging

logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(message)s")
logging.info("Loaded %d rows", 1200)
logging.error("Upload failed")

Best practices

The habits that separate a script that works once from code a team can rely on:

• PEP 8: Python’s official style guide (naming, indentation, line length). Let a formatter like ruff or black apply it for you.
• Git: not Python, but essential. Every script belongs in version control (see the GitHub lesson).
• CI/CD: automatically run tests and linters on every pull request, and deploy on merge (for example, with GitHub Actions).
• Environment variables: keep passwords and API keys out of code. Read them with os.environ, and load a local .env file with python-dotenv. Never commit .env.
• Configuration management: keep settings (paths, table names, schedules) in a config file instead of hard-coding them. YAML, TOML, or INI files read with configparser.

import os
from dotenv import load_dotenv

load_dotenv()                          # reads .env into the environment
db_password = os.environ["DB_PASSWORD"]

Cheatsheet

Area	Start with	Also worth knowing
Core Python	types, lists/dicts, if, functions, try/except	comprehensions, lambda
Data manipulation	pandas	NumPy, Polars, PyArrow
Data processing	PySpark	Dask, Airflow
Databases	SQLAlchemy + pd.read_sql	psycopg2, pymongo, pyodbc
Data quality	pytest, Pydantic	Great Expectations, Pandera, DQX
Cloud	requests, boto3	google-cloud-storage, azure-storage-blob, Docker
File formats	CSV, JSON, Parquet	Avro
Packages and environments	venv + pip, or uv	Poetry, Conda, pyenv
Advanced	with, logging, type hints	decorators, generators, async
Best practices	Git, .env for secrets	PEP 8 + ruff, CI/CD, config files

Next steps

Continue with Getting Started with AI in IDE, where an AI assistant helps you write and explain the Python from this lesson.