DataFrame basics I · Topic 9 Free
Sorting: orderBy() and limit()
Same tasks, other language: the SQL version of this topic.
Data model — the tables you will query
How the tables join
| Relationship | Cardinality | Join condition |
|---|---|---|
| students.cohort_id → cohorts.id | many students → one cohort | JOIN cohorts ON students.cohort_id = cohorts.id |
| enrollments.student_id → students.id | many enrollments → one student | JOIN students ON enrollments.student_id = students.id |
| enrollments.course_id → courses.id | many enrollments → one course | JOIN courses ON enrollments.course_id = courses.id |
| job_offers.student_id → students.id | many job offers → one student | JOIN students ON job_offers.student_id = students.id |
In short: each student belongs to one cohort. Students enroll in courses through enrollments, and their job offers land in job_offers.
Same rule as SQL: without orderBy, row order is not guaranteed. Spark makes this even more true — data lives in partitions across machines, so unsorted order is genuinely random.
orderBy()
SELECT company, salary_usd FROM job_offers ORDER BY salary_usd DESC;job_offers.select("company", "salary_usd").orderBy(col("salary_usd").desc())Direction is a method on the column: .asc() (default) or .desc(). A plain string sorts ascending:
students.orderBy("name") # A → Z
students.orderBy(col("name").desc()) # Z → A.sort() is an exact synonym of .orderBy() — you will see both in real code.
Several sort keys
courses.orderBy(col("category").asc(), col("hours").desc())Rows equal on the first key are sorted by the second — same as SQL.
limit()
SQL’s LIMIT 5 is the .limit(5) method. With sorting it gives top-N:
students.select("name", "joined_at").orderBy(col("joined_at").desc()).limit(5)The pipeline reads naturally: pick columns → sort newest first → keep 5.
A note on cost
In distributed Spark, a global sort is one of the expensive operations — data must be shuffled between machines. Sorting 30 rows here is free; sorting 30 billion rows is a design decision. Interviewers love this question.
Practice
Real Python runs in your browser with the PySpark DataFrame API. Solve the tasks, or just experiment.