DataFrame basics I · Topic 2 Free
Expressions and lit()
Same tasks, other language: the SQL version of this topic.
Data model — the tables you will query
How the tables join
| Relationship | Cardinality | Join condition |
|---|---|---|
| students.cohort_id → cohorts.id | many students → one cohort | JOIN cohorts ON students.cohort_id = cohorts.id |
| enrollments.student_id → students.id | many enrollments → one student | JOIN students ON enrollments.student_id = students.id |
| enrollments.course_id → courses.id | many enrollments → one course | JOIN courses ON enrollments.course_id = courses.id |
| job_offers.student_id → students.id | many job offers → one student | JOIN students ON job_offers.student_id = students.id |
In short: each student belongs to one cohort. Students enroll in courses through enrollments, and their job offers land in job_offers.
In SQL you computed values right in the SELECT list. PySpark does the same inside .select() — with Column objects.
Arithmetic
SQL:
SELECT title, hours * 60 AS minutes FROM courses;PySpark:
courses.select("title", (col("hours") * 60).alias("minutes"))+, -, *, /, % all work on col(...). Note the parentheses around the math before .alias() — you alias the whole expression, not the number 60.
Literals: lit()
A fixed value must be wrapped in lit() so PySpark knows it is a value, not a column name:
from pyspark.sql.functions import lit
students.select("name", lit(2026).alias("current_year"))Compare with SQL, where you just wrote 2026. This is the most common beginner error in PySpark: writing col("name") == "x" works for comparisons, but building expressions from raw values usually needs lit().
Joining strings
SQL’s name || ', ' || country becomes concat_ws (“concat with separator”):
students.select(F.concat_ws(", ", col("name"), col("country")).alias("profile"))F is the conventional alias for pyspark.sql.functions — in real projects you will see this import at the top of every file:
from pyspark.sql import functions as FName your expressions
Same rule as SQL: every computed column gets an .alias(). Without it you get generated names that break downstream code.
Practice
Real Python runs in your browser with the PySpark DataFrame API. Solve the tasks, or just experiment.