Data Science Project Workflow: From Problem to Production

After completing dozens of data science projects, I’ve developed a systematic workflow that ensures projects deliver value and reach production. Here’s my complete end-to-end process.

Phase 1: Problem Definition (Week 1)

1.1 Understand the Business Problem

Before writing any code, I spend time understanding:

Key Questions:

• What business metric are we trying to improve?
• What does success look like? (Quantify it!)
• Who are the stakeholders?
• What’s the expected ROI?

Example:

Problem: Customer churn is 25% annually
Goal: Reduce churn to 15% within 6 months
Success Metric: 40% reduction in churn rate
ROI: $2M annual savings

1.2 Define the ML Problem

Translate business to ML:

• Type: Classification, Regression, Clustering?
• Target variable: What are we predicting?
• Features: What data is available?
• Constraints: Latency, interpretability, cost?

1.3 Create Project Charter

# Project: Customer Churn Prediction

## Objective
Build ML model to predict customer churn 30 days in advance

## Success Criteria
- Precision > 80% (minimize false alarms)
- Recall > 70% (catch most churners)
- Inference time < 100ms

## Timeline
- Phase 1 (EDA): 2 weeks
- Phase 2 (Modeling): 3 weeks
- Phase 3 (Deployment): 2 weeks

## Resources
- Data: CRM database, usage logs
- Team: 1 DS, 1 MLE, 1 Product Manager
- Infrastructure: AWS SageMaker

Phase 2: Data Exploration & Analysis (Week 2-3)

2.1 Data Collection

import pandas as pd
import numpy as np

# Load data
customers = pd.read_sql("SELECT * FROM customers", conn)
usage = pd.read_sql("SELECT * FROM usage_logs", conn)
transactions = pd.read_sql("SELECT * FROM transactions", conn)

# Merge datasets
df = customers.merge(usage, on='customer_id', how='left')
df = df.merge(transactions, on='customer_id', how='left')

print(f"Total records: {len(df)}")
print(f"Features: {len(df.columns)}")
print(f"Churn rate: {df['churned'].mean():.2%}")

2.2 EDA Checklist

def perform_eda(df):
    """Comprehensive EDA"""
    
    # 1. Basic info
    print("=== Dataset Info ===")
    print(df.info())
    
    # 2. Missing values
    print("\n=== Missing Values ===")
    missing = df.isnull().sum()
    print(missing[missing > 0].sort_values(ascending=False))
    
    # 3. Target distribution
    print("\n=== Target Distribution ===")
    print(df['churned'].value_counts(normalize=True))
    
    # 4. Numeric features
    print("\n=== Numeric Features ===")
    print(df.describe())
    
    # 5. Correlations
    print("\n=== Top Correlations with Target ===")
    correlations = df.corr()['churned'].sort_values(ascending=False)
    print(correlations.head(10))
    
    # 6. Outliers
    print("\n=== Outliers Detection ===")
    numeric_cols = df.select_dtypes(include=[np.number]).columns
    for col in numeric_cols:
        Q1 = df[col].quantile(0.25)
        Q3 = df[col].quantile(0.75)
        IQR = Q3 - Q1
        outliers = ((df[col] < (Q1 - 1.5 * IQR)) | (df[col] > (Q3 + 1.5 * IQR))).sum()
        if outliers > 0:
            print(f"{col}: {outliers} outliers ({outliers/len(df)*100:.1f}%)")

perform_eda(df)

2.3 Visualization

import matplotlib.pyplot as plt
import seaborn as sns

# Set style
sns.set_style("whitegrid")
plt.rcParams['figure.figsize'] = (12, 6)

# 1. Target distribution
fig, axes = plt.subplots(1, 2, figsize=(14, 5))
df['churned'].value_counts().plot(kind='bar', ax=axes[0])
axes[0].set_title('Churn Distribution')

# 2. Feature importance (correlation)
top_features = correlations.head(10).index
df[top_features].corr()['churned'].plot(kind='barh', ax=axes[1])
axes[1].set_title('Top Correlated Features')

plt.tight_layout()
plt.savefig('eda_summary.png', dpi=300)

Phase 3: Feature Engineering (Week 3-4)

3.1 Create Features

def create_features(df):
    """Feature engineering pipeline"""
    
    # Temporal features
    df['account_age_days'] = (pd.Timestamp.now() - df['signup_date']).dt.days
    df['last_activity_days'] = (pd.Timestamp.now() - df['last_login']).dt.days
    df['days_since_purchase'] = (pd.Timestamp.now() - df['last_purchase_date']).dt.days
    
    # Engagement features
    df['login_frequency'] = df['total_logins'] / df['account_age_days']
    df['purchase_frequency'] = df['total_purchases'] / df['account_age_days']
    df['avg_session_duration'] = df['total_session_time'] / df['total_logins']
    
    # Monetary features
    df['total_revenue'] = df['total_purchases'] * df['avg_order_value']
    df['revenue_per_day'] = df['total_revenue'] / df['account_age_days']
    
    # Behavioral features
    df['support_ticket_rate'] = df['support_tickets'] / df['account_age_days']
    df['complaint_rate'] = df['complaints'] / df['total_interactions']
    
    # Recency features
    df['is_active_30d'] = (df['last_activity_days'] <= 30).astype(int)
    df['is_paying_customer'] = (df['total_revenue'] > 0).astype(int)
    
    return df

df = create_features(df)

3.2 Feature Selection

from sklearn.ensemble import RandomForestClassifier
from sklearn.feature_selection import SelectKBest, mutual_info_classif

# Method 1: Tree-based importance
rf = RandomForestClassifier(n_estimators=100, random_state=42)
rf.fit(X_train, y_train)

feature_importance = pd.DataFrame({
    'feature': X_train.columns,
    'importance': rf.feature_importances_
}).sort_values('importance', ascending=False)

print("Top 20 Features:")
print(feature_importance.head(20))

# Method 2: Mutual information
mi_scores = mutual_info_classif(X_train, y_train)
mi_scores = pd.Series(mi_scores, index=X_train.columns).sort_values(ascending=False)

# Select top features
top_features = feature_importance.head(50)['feature'].tolist()
X_train_selected = X_train[top_features]
X_test_selected = X_test[top_features]

Phase 4: Model Development (Week 4-6)

4.1 Baseline Model

from sklearn.dummy import DummyClassifier

# Always start with a baseline
baseline = DummyClassifier(strategy='most_frequent')
baseline.fit(X_train, y_train)

baseline_score = baseline.score(X_test, y_test)
print(f"Baseline Accuracy: {baseline_score:.3f}")

4.2 Experiment Tracking

import mlflow
from sklearn.linear_model import LogisticRegression
from sklearn.ensemble import RandomForestClassifier, GradientBoostingClassifier
from xgboost import XGBClassifier
from sklearn.metrics import accuracy_score, precision_score, recall_score, f1_score, roc_auc_score

def train_and_log_model(model, model_name, X_train, y_train, X_test, y_test):
    """Train model and log to MLflow"""
    
    with mlflow.start_run(run_name=model_name):
        # Train
        model.fit(X_train, y_train)
        
        # Predict
        y_pred = model.predict(X_test)
        y_pred_proba = model.predict_proba(X_test)[:, 1]
        
        # Metrics
        metrics = {
            'accuracy': accuracy_score(y_test, y_pred),
            'precision': precision_score(y_test, y_pred),
            'recall': recall_score(y_test, y_pred),
            'f1': f1_score(y_test, y_pred),
            'roc_auc': roc_auc_score(y_test, y_pred_proba)
        }
        
        # Log
        mlflow.log_params(model.get_params())
        mlflow.log_metrics(metrics)
        mlflow.sklearn.log_model(model, "model")
        
        print(f"\n{model_name} Results:")
        for metric, value in metrics.items():
            print(f"{metric}: {value:.3f}")
        
        return model, metrics

# Train multiple models
models = {
    'Logistic Regression': LogisticRegression(max_iter=1000),
    'Random Forest': RandomForestClassifier(n_estimators=100),
    'Gradient Boosting': GradientBoostingClassifier(n_estimators=100),
    'XGBoost': XGBClassifier(n_estimators=100, use_label_encoder=False)
}

results = {}
for name, model in models.items():
    results[name] = train_and_log_model(model, name, X_train, y_train, X_test, y_test)

4.3 Hyperparameter Tuning

from sklearn.model_selection import GridSearchCV

# Best performing model from above
param_grid = {
    'n_estimators': [100, 200, 300],
    'max_depth': [5, 10, 15, 20],
    'min_samples_split': [2, 5, 10],
    'min_samples_leaf': [1, 2, 4]
}

grid_search = GridSearchCV(
    RandomForestClassifier(),
    param_grid,
    cv=5,
    scoring='f1',
    n_jobs=-1,
    verbose=1
)

grid_search.fit(X_train, y_train)

print(f"Best Parameters: {grid_search.best_params_}")
print(f"Best F1 Score: {grid_search.best_score_:.3f}")

best_model = grid_search.best_estimator_

Phase 5: Model Evaluation (Week 6)

5.1 Comprehensive Evaluation

from sklearn.metrics import classification_report, confusion_matrix
import matplotlib.pyplot as plt

def evaluate_model(model, X_test, y_test):
    """Comprehensive model evaluation"""
    
    y_pred = model.predict(X_test)
    y_pred_proba = model.predict_proba(X_test)[:, 1]
    
    # Classification report
    print("=== Classification Report ===")
    print(classification_report(y_test, y_pred))
    
    # Confusion matrix
    cm = confusion_matrix(y_test, y_pred)
    plt.figure(figsize=(8, 6))
    sns.heatmap(cm, annot=True, fmt='d', cmap='Blues')
    plt.title('Confusion Matrix')
    plt.ylabel('True Label')
    plt.xlabel('Predicted Label')
    plt.savefig('confusion_matrix.png')
    
    # ROC curve
    from sklearn.metrics import roc_curve, auc
    fpr, tpr, _ = roc_curve(y_test, y_pred_proba)
    roc_auc = auc(fpr, tpr)
    
    plt.figure(figsize=(8, 6))
    plt.plot(fpr, tpr, label=f'ROC curve (AUC = {roc_auc:.2f})')
    plt.plot([0, 1], [0, 1], 'k--')
    plt.xlabel('False Positive Rate')
    plt.ylabel('True Positive Rate')
    plt.title('ROC Curve')
    plt.legend()
    plt.savefig('roc_curve.png')

evaluate_model(best_model, X_test, y_test)

Phase 6: Deployment (Week 7-8)

6.1 Create API

from fastapi import FastAPI
from pydantic import BaseModel
import joblib

app = FastAPI(title="Churn Prediction API")

# Load model
model = joblib.load('churn_model.pkl')
preprocessor = joblib.load('preprocessor.pkl')

class Customer(BaseModel):
    customer_id: str
    account_age_days: int
    login_frequency: float
    purchase_frequency: float
    support_ticket_rate: float
    # ... other features

@app.post("/predict")
async def predict_churn(customer: Customer):
    # Preprocess
    features = preprocessor.transform([customer.dict()])
    
    # Predict
    probability = model.predict_proba(features)[0][1]
    prediction = "Churn" if probability > 0.5 else "Retain"
    
    return {
        "customer_id": customer.customer_id,
        "churn_probability": float(probability),
        "prediction": prediction,
        "confidence": float(max(probability, 1-probability))
    }

6.2 Docker Deployment

FROM python:3.9-slim

WORKDIR /app

COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

COPY . .

CMD ["uvicorn", "main:app", "--host", "0.0.0.0", "--port", "8000"]

Project Structure

churn-prediction/
├── data/
│   ├── raw/
│   ├── processed/
│   └── features/
├── notebooks/
│   ├── 01_eda.ipynb
│   ├── 02_feature_engineering.ipynb
│   └── 03_modeling.ipynb
├── src/
│   ├── data/
│   │   ├── ingestion.py
│   │   └── preprocessing.py
│   ├── features/
│   │   └── build_features.py
│   ├── models/
│   │   ├── train.py
│   │   └── predict.py
│   └── api/
│       └── main.py
├── tests/
│   ├── test_data.py
│   ├── test_features.py
│   └── test_models.py
├── config/
│   └── config.yaml
├── requirements.txt
├── Dockerfile
└── README.md

Key Lessons Learned

1. Spend 60% time on data, 40% on models - Better data beats better algorithms
2. Start simple, iterate - Baseline → Linear → Trees → Ensembles
3. Track everything - Use MLflow/Weights & Biases from day 1
4. Test assumptions - Data distribution, feature correlations, model assumptions
5. Think production first - Design for deployment, not just notebooks
6. Communicate early and often - Show progress weekly to stakeholders

Conclusion

This workflow has helped me deliver 20+ successful data science projects. The key is being systematic, documenting everything, and always keeping the end goal (production deployment) in mind.

What’s your data science workflow? I’d love to hear your approach!

---

Tags: #DataScience #Workflow #MLOps #ProjectManagement #BestPractices