DataFrame basics I · Topic 5 Free
Filtering rows: filter()
Same tasks, other language: the SQL version of this topic.
Data model — the tables you will query
How the tables join
| Relationship | Cardinality | Join condition |
|---|---|---|
| students.cohort_id → cohorts.id | many students → one cohort | JOIN cohorts ON students.cohort_id = cohorts.id |
| enrollments.student_id → students.id | many enrollments → one student | JOIN students ON enrollments.student_id = students.id |
| enrollments.course_id → courses.id | many enrollments → one course | JOIN courses ON enrollments.course_id = courses.id |
| job_offers.student_id → students.id | many job offers → one student | JOIN students ON job_offers.student_id = students.id |
In short: each student belongs to one cohort. Students enroll in courses through enrollments, and their job offers land in job_offers.
SQL’s WHERE becomes .filter() (or its exact synonym .where()).
SELECT name, city FROM students WHERE country = 'Canada';students.filter(col("country") == "Canada").select("name", "city")Comparison operators
| SQL | PySpark |
|---|---|
= | == (double equals — Python) |
<> | != |
> < >= <= | same |
They work on numbers, text, and dates:
job_offers.filter(col("salary_usd") >= 100000)
students.filter(col("joined_at") >= "2025-01-01")
courses.filter(col("difficulty") != "beginner")Combining conditions: & and |
This is the number one PySpark trap. Python’s words and / or do not work on columns. Use the symbols, and wrap every condition in parentheses:
# AND
job_offers.filter((col("city") == "Berlin") & (col("salary_usd") > 90000))
# OR
students.filter((col("country") == "Germany") | (col("country") == "Poland"))
# NOT
job_offers.filter(~(col("remote") == True))Without the parentheses Python misreads the expression and you get a confusing error. When in doubt: parentheses around each comparison, always.
SQL strings inside filter()
.filter() also accepts a plain SQL condition as a string — fully valid PySpark:
job_offers.filter("salary_usd >= 100000 AND remote = false")Handy when a condition is easier to say in SQL. Both styles appear in real codebases; the Column style is more common because it survives refactoring better.
Practice
Real Python runs in your browser with the PySpark DataFrame API. Solve the tasks, or just experiment.