The Essential Guide to Regular Expressions for Data Scientists

As a data scientist, you’ll frequently encounter messy, unstructured text data. Before you can analyze this data, you need to clean it, extract relevant information, and transform it into a structured format. This is where regular expressions come in useful.

Think of regex as a specialized mini-language for describing patterns in text. Once you understand the core concepts, you’ll be able to perform complex text operations with just a few lines of code that would otherwise require dozens of lines using standard string methods.

🔗 Link to the code on GitHub. You can also check out this quick reference regex table.

How to Think About Regex

 
The key to mastering regex is developing the right mental model. At its core, a regular expression is simply a pattern that moves through text from left to right, trying to find matches.

Imagine you’re looking for a specific pattern in a book. You scan each page, looking for that pattern. That’s essentially what regex does—it scans through your text character by character, checking if the current position matches your pattern.

Let’s start by importing Python’s built-in re module:

1. Literal Characters: Building Your First Regex Pattern

 
The simplest regex patterns match exact text. If you want to find the word “data” in a text, you can use:

text = "Data science is cool as you get to work with real-world data"
matches = re.findall(r"data", text)
print(matches)  

Notice that this only found the lowercase “data” and missed “Data” at the beginning.

Regular expressions are case-sensitive by default. This brings us to our first important lesson: be specific about what you want to match.

matches = re.findall(r"data", text, re.IGNORECASE)
print(matches)  

Output >>> ['Data', 'data']

The r before the string creates a “raw string.” This is important in regex because backslashes are used for special sequences, and raw strings prevent Python from interpreting these backslashes.

2. Metacharacters: Beyond Literal Matching

 
What makes regex useful is its ability to define patterns using metacharacters. These are special characters that have meaning beyond their literal representation.

The Wildcard: The Dot (.)

The dot matches any character except a newline. This is particularly useful when you know part of a pattern but not everything:

text = "The cat sat on the mat. The bat flew over the rat."
pattern = r"The ... "
matches = re.findall(pattern, text)
print(matches)  

Here, we’re finding “The” followed by any three characters and a space.

Output >>> ['The cat ', 'The bat ']

The dot is powerful, but sometimes too powerful—it matches anything! This is where character classes come in.

Character Classes: Getting Specific with []

Character classes let you define a set of characters to match:

text = "The cat sat on the mat. The bat flew over the rat."
pattern = r"[cb]at"
matches = re.findall(pattern, text)
print(matches) 

This pattern finds “cat” or “bat”—any character in the set [cb] followed by “at”.

Output >>> ['cat', 'bat']

Character classes are perfect when you have a limited set of characters that could appear in a certain position.

You can also use ranges in character classes:

# Find all lowercase words that start with a-d
pattern = r"\b[a-d][a-z]*\b"
text = "apple banana cherry date elephant fig grape kiwi lemon mango orange"
matches = re.findall(pattern, text)
print(matches)

Here, \b represents a word boundary (more on this later), [a-d] matches any lowercase letter from a to d, and [a-z]* matches zero or more lowercase letters.

Output >>> ['apple', 'banana', 'cherry', 'date']

Quantifiers: Specifying Repetition

Often, you’ll want to match a pattern that repeats. Quantifiers let you specify how many times a character or group should appear. Let’s find all phone numbers, whether they use hyphens or not:

text = "Phone numbers: 555-1234, 555-5678, 5551234"
pattern = r"\b\d3-?\d4\b"
matches = re.findall(pattern, text)
print(matches) 

This gives the following:

Output >>> ['555-1234', '555-5678', '5551234']

Breaking down this pattern:

• \b ensures we’re at a word boundary
• \d3 matches exactly 3 digits
• -? matches zero or one hyphen (the ? makes the hyphen optional)
• \d4 matches exactly 4 digits
• \b ensures we’re at another word boundary

This is much more elegant than writing multiple patterns or complex string operations to handle different formats.

3. Anchors: Finding Patterns at Specific Positions

 
Sometimes you only want to find patterns at specific positions in the text. Anchors help with this:

text = "Python is popular in data science."

# ^ anchors to the start of the string
start_matches = re.findall(r"^Python", text)
print(start_matches)

# $ anchors to the end of the string
end_matches = re.findall(r"science\.$", text)
print(end_matches) 

This outputs:

Anchors don’t match characters; they match positions. This is useful for validating formats like email addresses, where specific elements must appear at the beginning or end.

4. Capturing Groups: Extracting Specific Parts

 
Often in data science, you don’t just want to find patterns—you want to extract specific parts of those patterns. Capturing groups, created with parentheses, let you do this:

text = "Dates: 2023-10-15, 2022-05-22"
pattern = r"(\d4)-(\d2)-(\d2)"

# findall returns tuples of the captured groups
matches = re.findall(pattern, text)
print(matches) 

# You can use these to create structured data
for year, month, day in matches:
    print(f"Year: year, Month: month, Day: day")

Here’s the output:

[('2023', '10', '15'), ('2022', '05', '22')]
Year: 2023, Month: 10, Day: 15
Year: 2022, Month: 05, Day: 22

This is especially helpful in extracting structured information from unstructured text, a common task in data science.

5. Named Groups: Making Your Regex More Readable

 
For complex patterns, remembering what each group captures can be challenging. Named groups solve this:

text = "Contact: john.doe@example.com"
pattern = r"(?P[\w.]+)@(?P[\w.]+)"

match_ = re.search(pattern, text)
if match_:
    print(f"Username: match.group('username')")
    print(f"Domain: match.group('domain')")

This gives:

Username: john.doe
Domain: example.com

Named groups make your regex more self-documenting and easier to maintain.

Working with Real Data: Practical Examples

 
Let’s see how regex applies to common data science tasks.

Example 1: Cleaning Messy Data

Suppose you have a dataset with inconsistent product codes:

product_codes = [
    "PROD-123",
    "Product 456",
    "prod_789",
    "PR-101",
    "p-202"
]

You want to standardize these to extract just the numeric part:

cleaned_codes = []
for code in product_codes:
    # Extract just the numeric portion
    match = re.search(r"\d+", code)
    if match:
        cleaned_codes.append(match.group())

print(cleaned_codes)  

Output:

['123', '456', '789', '101', '202']

This is much cleaner than writing multiple string operations to handle different formats.

Example 2: Extracting Information from Text

Imagine you have customer service logs and need to extract information:

log = "ISSUE #1234 [2023-10-15] Customer reported app crash on iPhone 12, iOS 15.2"

You can extract structured data with regex:

# Extract issue number, date, device, and OS version
pattern = r"ISSUE #(\d+) \[(\d4-\d2-\d2)\].*?(iPhone \d+).*?(iOS \d+\.\d+)"
match = re.search(pattern, log)

if match:
    issue_num, date, device, ios_version = match.groups()
    print(f"Issue: issue_num")
    print(f"Date: date")
    print(f"Device: device")
    print(f"iOS Version: ios_version")

Output:

Issue: 1234
Date: 2023-10-15
Device: iPhone 12
iOS Version: iOS 15.2

Example 3: Data Validation

Regular expressions are useful for validating data formats:

def validate_email(email):
    """Validate email format with explanation of what makes it valid or invalid."""
    pattern = r"^[\w.%+-]+@[A-Za-z0-9.-]+\.[A-Za-z]2,$"
    
    if not re.match(pattern, email):
        # Let's check specific issues
        if '@' not in email:
            return False, "Missing @ symbol"
        
        username, domain = email.split('@', 1)
        if not username:
            return False, "Username is empty"
        
        if '.' not in domain:
            return False, "Invalid domain (missing top-level domain)"
        
        return False, "Invalid email format"
    
    return True, "Valid email"

Now test with different emails:

# Test with different emails
emails = ["user@example.com", "invalid@.com", "no_at_sign.com", "user@example.co.uk"]
for email in emails:
    valid, reason = validate_email(email)
    print(f"email: reason")

Output:

user@example.com: Valid email
invalid@.com: Invalid email format
no_at_sign.com: Missing @ symbol
user@example.co.uk: Valid email

This function not only validates emails but explains what makes them valid or invalid, which is more useful than a simple true/false result.

Rather than just listing patterns, let’s understand the components that make them work:

Email Validation

pattern = r"^[\w.%+-]+@[A-Za-z0-9.-]+\.[A-Za-z]2,$"

Breaking this down:

• ^ ensures we start at the beginning of the string
• [\w.%+-]+ matches one or more word characters, dots, percent signs, plus signs, or hyphens (common username characters)
• @ matches the literal @ symbol
• [A-Za-z0-9.-]+ matches one or more letters, numbers, dots, or hyphens (domain name)
• \. matches a literal dot
• [A-Za-z]2, matches two or more letters (top-level domain)
• $ ensures we end at the end of the string

This pattern allows for valid emails while rejecting invalid formats.

Date Extraction

pattern = r"\b(\d4)-(\d2)-(\d2)\b"

This pattern matches ISO dates (YYYY-MM-DD):

• \b ensures we’re at a word boundary
• (\d4) captures exactly 4 digits for the year
• – matches the literal hyphen
• (\d2) captures exactly 2 digits for the month
• – matches the literal hyphen
• (\d2) captures exactly 2 digits for the day
• \b ensures we’re at a word boundary

Understanding this structure lets you adapt it for other date formats like MM/DD/YYYY.

Advanced Techniques: Beyond Basic Regex

 
As you become more comfortable with regex, you’ll encounter situations where basic patterns fall short. Here are some advanced techniques:

Lookaheads and Lookbehinds

These are “zero-width assertions” that check if a pattern exists without including it in the match:

# Password validation 
password = "Password123"
has_uppercase = bool(re.search(r"(?=.*[A-Z])", password))
has_lowercase = bool(re.search(r"(?=.*[a-z])", password))
has_digit = bool(re.search(r"(?=.*\d)", password))
is_long_enough = len(password) >= 8

if all([has_uppercase, has_lowercase, has_digit, is_long_enough]):
    print("Password meets requirements")
else:
    print("Password does not meet all requirements")

Output:

Password meets requirements

The lookahead (?=.*[A-Z]) checks if there’s an uppercase letter anywhere in the string without actually capturing it.

Non-Greedy Matching

Quantifiers are “greedy” by default. Meaning they match as much as possible. Adding a ? after a quantifier makes it “non-greedy”:

text = "
First content
Second content
"

# Greedy matching (default)
greedy = re.findall(r"
(.*)
", text)
print(f"Greedy: greedy")  

# Non-greedy matching
non_greedy = re.findall(r"
(.*?)
", text)
print(f"Non-greedy: non_greedy")

Output:

Second content’]
Non-greedy: [‘First content’, ‘Second content’]

Understanding the difference between greedy and non-greedy matching is necessary for parsing nested structures like HTML or JSON.

Learning and Debugging Regex

 
When you’re learning regular expressions:

1. Start with literal matching: Match exact strings before adding complexity
2. Add character classes: Learn to match categories of characters
3. Master quantifiers: Understand repetition patterns
4. Use capturing groups: Extract structured data
5. Learn anchors and boundaries: Control where patterns match
6. Explore advanced techniques: Lookaheads, non-greedy matching, etc.

The key is to keep learning—start simple and gradually move to others as needed.

When your regex isn’t working as expected:

1. Break it down: Test simpler versions of your pattern to isolate the issue
2. Visualize it: Use tools like regex101.com to see how your pattern matches step by step
3. Test with sample data: Create small test cases that cover different scenarios

For example, if you’re trying to match phone numbers but your pattern isn’t working, try matching just the area code first, then add more components gradually.

Wrapping Up

 
Regular expressions are a powerful tool for text processing in data science. They allow you to:

1. Extract structured information from unstructured text
2. Clean and standardize inconsistent data formats
3. Validate data against specific patterns
4. Transform text through sophisticated search and replace operations

Remember that regex is a skill that develops over time. Don’t try to memorize every metacharacter and technique—instead, focus on understanding the underlying principles and practice regularly with real-world data problems.
 
 

Bala Priya C is a developer and technical writer from India. She likes working at the intersection of math, programming, data science, and content creation. Her areas of interest and expertise include DevOps, data science, and natural language processing. She enjoys reading, writing, coding, and coffee! Currently, she’s working on learning and sharing her knowledge with the developer community by authoring tutorials, how-to guides, opinion pieces, and more. Bala also creates engaging resource overviews and coding tutorials.