DataFrame basics I · Topic 3 Free
Built-in functions (F.*)
Same tasks, other language: the SQL version of this topic.
Data model — the tables you will query
How the tables join
| Relationship | Cardinality | Join condition |
|---|---|---|
| students.cohort_id → cohorts.id | many students → one cohort | JOIN cohorts ON students.cohort_id = cohorts.id |
| enrollments.student_id → students.id | many enrollments → one student | JOIN students ON enrollments.student_id = students.id |
| enrollments.course_id → courses.id | many enrollments → one course | JOIN courses ON enrollments.course_id = courses.id |
| job_offers.student_id → students.id | many job offers → one student | JOIN students ON job_offers.student_id = students.id |
In short: each student belongs to one cohort. Students enroll in courses through enrollments, and their job offers land in job_offers.
Everything SQL does with functions, PySpark does through the pyspark.sql.functions module — imported as F by convention.
Text functions
SQL → PySpark:
SELECT UPPER(name), LOWER(country), LENGTH(name) FROM students;students.select(
F.upper(col("name")),
F.lower(col("country")),
F.length(col("name")),
)Also useful: F.trim(), F.regexp_replace(col, pattern, replacement), F.concat_ws(sep, ...).
Date functions
students.select(
"name",
F.year(col("joined_at")).alias("join_year"),
F.month(col("joined_at")).alias("join_month"),
)F.year, F.month, F.dayofmonth mirror SQL’s EXTRACT(YEAR FROM ...) family.
Conditional logic: when / otherwise
A preview of SQL’s CASE WHEN — in PySpark it reads like a sentence:
students.select(
"name",
F.when(col("country") == "Canada", "local").otherwise("international").alias("kind"),
)Functions compose
Exactly like SQL, output of one feeds another:
students.select(F.upper(F.trim(col("name"))).alias("clean_name"))Read from the inside out.
Playground note
This playground runs the PySpark API on a DuckDB engine, so a few niche functions are missing (for example F.round and F.substring). Everything in this course works exactly as it does in real Spark.
Practice
Real Python runs in your browser with the PySpark DataFrame API. Solve the tasks, or just experiment.