Regular expressions: rlike()

SQL’s regex operator ~ becomes the Column method .rlike() (“regex like”).

SELECT name FROM students WHERE name ~ '^[AM]';

students.filter(col("name").rlike("^[AM]")).select("name")

The same regex syntax

Everything you learned in the SQL regex topic carries over unchanged:

Pattern	Matches
.	any single character
^ / $	start / end of the string
[abc], [a-z], [0-9]	character sets and ranges
+ / * / ?	one or more / zero or more / optional
(ab|cd)	ab or cd

courses.filter(col("title").rlike("[0-9]"))                  # contains a digit
job_offers.filter(col("role").rlike("(Analyst|Engineer)"))   # either word
students.filter(col("name").rlike("^A.*a$"))                 # starts A, ends a

Like in SQL, an unanchored regex matches anywhere in the string — use ^ and $ to pin it down.

Cleaning data with regex

The matching operator’s best friend is the replacement function — a data engineering workhorse:

students.select(F.regexp_replace(col("name"), "[aeiou]", "*").alias("masked"))

You will use regexp_replace for stripping currency signs, fixing phone formats, and normalizing messy text in almost every pipeline.

When to use what

• Exact value → ==
• Starts with / contains / ends with → like(), startswith(), contains()
• Anything more complex → rlike()