10 Python Tips Every Data Scientist Should Know

As a data scientist working with Python daily, I’ve learned several techniques that dramatically improved my productivity. Here are 10 essential tips that every data scientist should master.

1. Use List Comprehensions for Better Performance

Instead of traditional loops, list comprehensions are faster and more Pythonic:

# Bad - Traditional loop
squares = []
for i in range(1000):
    squares.append(i**2)

# Good - List comprehension
squares = [i**2 for i in range(1000)]

Performance gain: ~30% faster for large datasets

2. Leverage Pandas’ .query() for Readable Filtering

import pandas as pd

# Instead of this
df_filtered = df[(df['age'] > 25) & (df['salary'] > 50000)]

# Use this - more readable
df_filtered = df.query('age > 25 and salary > 50000')

3. Use .pipe() for Chainable Operations

result = (df
    .pipe(remove_outliers)
    .pipe(normalize_columns)
    .pipe(add_features)
)

This makes your data pipeline clear and maintainable.

4. Memory Optimization with Data Types

# Check memory usage
print(df.memory_usage(deep=True))

# Optimize dtypes
df['category_col'] = df['category_col'].astype('category')
df['int_col'] = df['int_col'].astype('int32')  # instead of int64

Result: Can reduce memory usage by 50-80%

5. Use functools.lru_cache for Expensive Functions

from functools import lru_cache

@lru_cache(maxsize=128)
def expensive_calculation(n):
    # Your complex computation
    return result

Perfect for recursive functions or repeated calculations.

6. Parallel Processing with concurrent.futures

from concurrent.futures import ProcessPoolExecutor
import pandas as pd

def process_chunk(chunk):
    # Your processing logic
    return chunk.apply(some_function)

with ProcessPoolExecutor() as executor:
    results = list(executor.map(process_chunk, df_chunks))

7. Use pathlib Instead of os.path

from pathlib import Path

# Modern, clean way
data_dir = Path('data')
csv_files = list(data_dir.glob('*.csv'))

# Works cross-platform automatically
output_path = data_dir / 'processed' / 'output.csv'

8. Context Managers for Resource Management

from contextlib import contextmanager

@contextmanager
def timer(name):
    start = time.time()
    yield
    print(f"{name} took {time.time() - start:.2f}s")

with timer("Data loading"):
    df = pd.read_csv('large_file.csv')

9. Use .loc and .iloc Explicitly

# Ambiguous - can cause SettingWithCopyWarning
df[df['age'] > 25]['salary'] = 60000

# Clear and correct
df.loc[df['age'] > 25, 'salary'] = 60000

10. Leverage NumPy’s Vectorization

import numpy as np

# Slow - Python loop
result = [x**2 + 2*x + 1 for x in data]

# Fast - Vectorized
result = data**2 + 2*data + 1

Performance: 10-100x faster for large arrays

Bonus Tip: Use Type Hints

from typing import List, Dict, Optional
import pandas as pd

def process_data(df: pd.DataFrame, 
                 columns: List[str], 
                 threshold: Optional[float] = None) -> pd.DataFrame:
    # Your code here
    return df

This makes your code self-documenting and catches errors early.

Conclusion

These tips have saved me countless hours in my data science projects. Start incorporating them into your workflow, and you’ll see immediate improvements in code quality, performance, and maintainability.

What’s your favorite Python tip? Let me know in the comments below!

---

Tags: #Python #DataScience #Programming #BestPractices #Productivity