Kubernetes Best Practices for Spring Boot Microservices

The Context: 100+ Spring Boot Services on GKE

When we deployed our mutual fund platform to Google Kubernetes Engine (GKE), we had:

100+ Spring Boot microservices (Java 21, Spring Boot 3.3.5)
10,000+ concurrent users at peak
1M+ transactions daily
99.9% uptime SLA (financial system requirement)

After 12 months in production, we’ve learned what works and what doesn’t. Here are the battle-tested practices that helped us achieve 99.94% actual uptime.

1. Health Checks: More Than Just `/actuator/health`

❌ Common Mistake: Single Health Endpoint

# ❌ BAD: Using same endpoint for liveness and readiness
livenessProbe:
  httpGet:
    path: /actuator/health
    port: 8080
readinessProbe:
  httpGet:
    path: /actuator/health
    port: 8080

Problem: If the database is down, the app reports unhealthy, Kubernetes restarts it, but the database is still down → restart loop hell.

✅ Best Practice: Separate Liveness and Readiness

Spring Boot Actuator 2.3+ supports separate endpoints:

# ✅ GOOD: Different probes for different purposes
livenessProbe:
  httpGet:
    path: /actuator/health/liveness
    port: 8080
  initialDelaySeconds: 60
  periodSeconds: 10
  failureThreshold: 3

readinessProbe:
  httpGet:
    path: /actuator/health/readiness
    port: 8080
  initialDelaySeconds: 30
  periodSeconds: 5
  failureThreshold: 2

application.yml:

management:
  endpoint:
    health:
      probes:
        enabled: true  # Enables /liveness and /readiness
  health:
    livenessState:
      enabled: true
    readinessState:
      enabled: true

What each checks:

Liveness: “Is the app running?” (crashes, deadlocks, out-of-memory)
Readiness: “Can the app handle traffic?” (database connection, external API availability)

Custom Health Indicators

@Component
public class ExternalApiHealthIndicator implements HealthIndicator {
    
    @Autowired
    private WebClient camsApiClient;
    
    @Override
    public Health health() {
        try {
            camsApiClient.get()
                .uri("/health")
                .retrieve()
                .toBodilessEntity()
                .block(Duration.ofSeconds(2));
            
            return Health.up()
                .withDetail("cams-api", "reachable")
                .build();
        } catch (Exception e) {
            return Health.down()
                .withDetail("cams-api", "unreachable")
                .withDetail("error", e.getMessage())
                .build();
        }
    }
}

Configuration:

management:
  health:
    defaults:
      enabled: false  # Disable all by default
    liveness:
      enabled: true
    readiness:
      enabled: true
    db:
      enabled: true  # Include database in readiness
    diskSpace:
      enabled: true  # Include disk in liveness

Result: When CAMS API is down, pods become “not ready” (stop receiving traffic) but don’t restart.

2. Resource Management: Requests vs Limits

❌ Common Mistake: No Resource Definitions

# ❌ BAD: No resource constraints
containers:
- name: portfolio-service
  image: gcr.io/giftcity/portfolio-service:v1.0.0
  # No resources defined → unbounded memory usage

Problem: One rogue service can consume all node resources, starving other pods.

✅ Best Practice: Set Requests and Limits

# ✅ GOOD: Properly sized resources
containers:
- name: portfolio-service
  image: gcr.io/giftcity/portfolio-service:v1.0.0
  resources:
    requests:
      memory: "512Mi"   # Guaranteed memory
      cpu: "500m"       # Guaranteed CPU (0.5 cores)
    limits:
      memory: "1Gi"     # Max memory before OOMKill
      cpu: "1000m"      # Max CPU (1 core)

How to determine the right values?

Run load tests and monitor resource usage:

kubectl top pod portfolio-service-abc123

Set requests = P95 usage (what you typically need):

Memory P95: 400Mi → request: 512Mi (add 25% buffer)
CPU P95: 0.4 cores → request: 500m (add 25% buffer)

Set limits = 2x requests (allow burst):

Memory limit: 1Gi (2x 512Mi)
CPU limit: 1000m (2x 500m)

JVM Memory Configuration

Spring Boot apps need JVM tuning:

env:
- name: JAVA_OPTS
  value: "-Xms512m -Xmx768m -XX:MaxMetaspaceSize=256m"

Calculation:

Container memory limit: 1Gi (1024Mi)
JVM heap (-Xmx): 768Mi (75%)
Metaspace: 256Mi (25%)
Total: 1024Mi

Why not 100%? JVM needs memory for:

Thread stacks
Native memory
Off-heap caches (Netty buffers, etc.)

Production config we use:

env:
- name: JAVA_OPTS
  value: >
    -Xms512m 
    -Xmx768m 
    -XX:MaxMetaspaceSize=256m
    -XX:+UseG1GC
    -XX:MaxGCPauseMillis=200
    -XX:+HeapDumpOnOutOfMemoryError
    -XX:HeapDumpPath=/tmp/heapdump.hprof

3. Graceful Shutdown: Don’t Drop Requests

❌ Common Mistake: Immediate Shutdown

When Kubernetes sends SIGTERM, Spring Boot has 30 seconds (default) to shut down. Without configuration, it immediately stops accepting new requests → dropped connections.

✅ Best Practice: Graceful Shutdown with Drain Period

# application.yml
server:
  shutdown: graceful  # Wait for active requests to complete

spring:
  lifecycle:
    timeout-per-shutdown-phase: 30s  # Max wait time

Kubernetes config:

lifecycle:
  preStop:
    exec:
      command: ["sh", "-c", "sleep 10"]  # Give load balancer time to update
terminationGracePeriodSeconds: 40  # 10s sleep + 30s graceful shutdown

What happens during shutdown:

Kubernetes sends SIGTERM to pod
preStop hook: Sleep 10 seconds (let kube-proxy update iptables)
Spring Boot stops accepting new requests
Spring Boot waits up to 30s for active requests to finish
If requests still active after 30s, force shutdown
Container exits

Result: Zero dropped requests during deployments.

4. Horizontal Pod Autoscaling (HPA)

✅ CPU-Based Autoscaling

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: portfolio-service-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: portfolio-service
  minReplicas: 5
  maxReplicas: 50
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 70  # Scale when avg CPU > 70%
  - type: Resource
    resource:
      name: memory
      target:
        type: Utilization
        averageUtilization: 80
  behavior:
    scaleUp:
      stabilizationWindowSeconds: 60  # Wait 60s before scaling up
      policies:
      - type: Percent
        value: 50  # Scale up by max 50% of current replicas
        periodSeconds: 60
    scaleDown:
      stabilizationWindowSeconds: 300  # Wait 5 min before scaling down
      policies:
      - type: Pods
        value: 2  # Scale down by max 2 pods at a time
        periodSeconds: 60

Key settings:

minReplicas: 5 → Always have 5 pods (handles baseline traffic)
maxReplicas: 50 → Never exceed 50 pods (cost control)
CPU threshold: 70% → Scale up when average CPU > 70%
Stabilization: Prevent flapping (rapid scale up/down)

✅ Custom Metrics Autoscaling

CPU/memory don’t always reflect application load. We also scale based on queue depth:

metrics:
- type: External
  external:
    metric:
      name: pubsub|topic|num_undelivered_messages
      selector:
        matchLabels:
          resource.type: pubsub_topic
          resource.labels.topic_id: order-processing
    target:
      type: AverageValue
      averageValue: "100"  # Scale when 100+ unacked messages

Result: Scale workers based on actual backlog, not just CPU.

5. Configuration Management: ConfigMaps vs Secrets

✅ ConfigMap for Non-Sensitive Data

apiVersion: v1
kind: ConfigMap
metadata:
  name: portfolio-service-config
data:
  application.yml: |
    spring:
      application:
        name: portfolio-service
      datasource:
        url: jdbc:postgresql://10.x.x.x:5432/giftcity
        hikari:
          maximum-pool-size: 20
    
    external-apis:
      cams:
        url: https://cams-api.example.com
        timeout: 2000
      nav:
        url: https://nav-service:8080
        timeout: 1000

Mount as file:

containers:
- name: portfolio-service
  volumeMounts:
  - name: config
    mountPath: /config
    readOnly: true
volumes:
- name: config
  configMap:
    name: portfolio-service-config

Spring Boot picks it up automatically:

spring:
  config:
    import: file:/config/application.yml

✅ Secrets for Credentials

apiVersion: v1
kind: Secret
metadata:
  name: db-credentials
type: Opaque
data:
  username: cG9zdGdyZXM=  # base64 encoded
  password: c3VwZXJzZWNyZXQ=

Use as environment variables:

containers:
- name: portfolio-service
  env:
  - name: DB_USERNAME
    valueFrom:
      secretKeyRef:
        name: db-credentials
        key: username
  - name: DB_PASSWORD
    valueFrom:
      secretKeyRef:
        name: db-credentials
        key: password

Spring Boot config:

spring:
  datasource:
    username: ${DB_USERNAME}
    password: ${DB_PASSWORD}

✅ Google Secret Manager (Production)

For production, we use Google Secret Manager instead of Kubernetes Secrets:

@Configuration
public class SecretConfig {
    
    @Bean
    public SecretManagerServiceClient secretManagerClient() throws IOException {
        return SecretManagerServiceClient.create();
    }
    
    @Bean
    public String dbPassword(SecretManagerServiceClient client) {
        SecretVersionName secretName = SecretVersionName.of(
            "giftcity-prod", "db-password", "latest"
        );
        return client.accessSecretVersion(secretName)
            .getPayload()
            .getData()
            .toStringUtf8();
    }
}

Why? Centralized secret rotation, audit logs, versioning.

6. Logging: Structured JSON to Stdout

✅ Structured Logging (Logback)

<!-- logback-spring.xml -->
<configuration>
    <appender name="CONSOLE" class="ch.qos.logback.core.ConsoleAppender">
        <encoder class="net.logstash.logback.encoder.LogstashEncoder">
            <customFields>
                {
                    "service": "portfolio-service",
                    "version": "${APP_VERSION}",
                    "environment": "${SPRING_PROFILES_ACTIVE}"
                }
            </customFields>
            <includeContext>false</includeContext>
            <fieldNames>
                <timestamp>timestamp</timestamp>
                <version>[ignore]</version>
                <levelValue>[ignore]</levelValue>
            </fieldNames>
        </encoder>
    </appender>
    
    <root level="INFO">
        <appender-ref ref="CONSOLE" />
    </root>
</configuration>

Output (JSON):

{
  "timestamp": "2026-04-12T10:30:45.123Z",
  "level": "INFO",
  "service": "portfolio-service",
  "version": "v1.2.0",
  "environment": "production",
  "thread": "reactor-http-nio-2",
  "logger": "com.giftcity.portfolio.PortfolioService",
  "message": "Portfolio fetched for investor INV-12345",
  "investor_id": "INV-12345",
  "response_time_ms": 45
}

Benefits:

Cloud Logging integration: GKE automatically indexes JSON fields
Searchable: Query by investor_id, response_time_ms, etc.
Structured: No regex parsing needed

7. Observability: The Three Pillars

✅ Metrics (Prometheus)

# application.yml
management:
  endpoints:
    web:
      exposure:
        include: health,info,metrics,prometheus
  metrics:
    export:
      prometheus:
        enabled: true
    tags:
      application: ${spring.application.name}
      environment: ${SPRING_PROFILES_ACTIVE}

ServiceMonitor (Prometheus Operator):

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: portfolio-service
spec:
  selector:
    matchLabels:
      app: portfolio-service
  endpoints:
  - port: http
    path: /actuator/prometheus
    interval: 15s

Grafana dashboard queries:

# Request rate
rate(http_server_requests_seconds_count{application="portfolio-service"}[1m])

# P95 latency
histogram_quantile(0.95, 
  rate(http_server_requests_seconds_bucket{application="portfolio-service"}[5m])
)

# Error rate
rate(http_server_requests_seconds_count{application="portfolio-service",status=~"5.."}[1m])

✅ Distributed Tracing (OpenTelemetry)

<dependency>
    <groupId>io.opentelemetry.instrumentation</groupId>
    <artifactId>opentelemetry-spring-boot-starter</artifactId>
    <version>2.2.0</version>
</dependency>

# application.yml
otel:
  exporter:
    otlp:
      endpoint: http://otel-collector:4317
  service:
    name: portfolio-service
  traces:
    sampler:
      probability: 0.1  # Sample 10% of traces (reduce cost)

Result: Full request trace across all microservices.

8. Pod Disruption Budgets (PDB)

Prevents Kubernetes from evicting too many pods during node maintenance:

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: portfolio-service-pdb
spec:
  minAvailable: 3  # Always keep 3 pods running
  selector:
    matchLabels:
      app: portfolio-service

Why? During node upgrades, Kubernetes drains nodes. Without PDB, it might evict all pods at once → downtime.

9. Deployment Strategy: Rolling Updates

spec:
  replicas: 10
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 2        # Create 2 extra pods during rollout
      maxUnavailable: 1  # Max 1 pod can be unavailable

Rollout process:

Create 2 new pods (total: 12)
Wait for new pods to become ready
Terminate 1 old pod (total: 11)
Create 1 new pod (total: 12)
Repeat until all old pods replaced

Result: Zero downtime deployments.

10. Security: Non-Root User

# Dockerfile
FROM eclipse-temurin:21-jre

# Create non-root user
RUN groupadd -r spring && useradd -r -g spring spring

WORKDIR /app
COPY target/portfolio-service.jar app.jar

# Change ownership
RUN chown -R spring:spring /app

# Run as non-root
USER spring

ENTRYPOINT ["java", "-jar", "app.jar"]

Kubernetes SecurityContext:

securityContext:
  runAsNonRoot: true
  runAsUser: 1000
  allowPrivilegeEscalation: false
  readOnlyRootFilesystem: true
  capabilities:
    drop:
    - ALL

Results: Production Metrics

After implementing these practices across 100+ services:

Metric	Before	After	Improvement
Uptime	99.7%	99.94%	0.24% ↑
P95 Deployment Time	8 min	3 min	63% faster
Dropped Requests/Deploy	50-100	0	100% better
OOMKills/month	15-20	0-1	95% fewer
Restart Loops	5-10/month	0	100% eliminated
Resource Utilization	45%	65%	Better bin packing

Cost impact: Better resource utilization saved ~₹80K/month in GKE costs.

Quick Reference: Complete Deployment YAML

apiVersion: apps/v1
kind: Deployment
metadata:
  name: portfolio-service
  namespace: giftcity-prod
spec:
  replicas: 5
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 2
      maxUnavailable: 1
  selector:
    matchLabels:
      app: portfolio-service
  template:
    metadata:
      labels:
        app: portfolio-service
        version: v1.2.0
    spec:
      containers:
      - name: portfolio-service
        image: gcr.io/giftcity/portfolio-service:v1.2.0
        ports:
        - containerPort: 8080
          name: http
        env:
        - name: SPRING_PROFILES_ACTIVE
          value: "production"
        - name: JAVA_OPTS
          value: "-Xms512m -Xmx768m -XX:MaxMetaspaceSize=256m"
        resources:
          requests:
            memory: "512Mi"
            cpu: "500m"
          limits:
            memory: "1Gi"
            cpu: "1000m"
        livenessProbe:
          httpGet:
            path: /actuator/health/liveness
            port: 8080
          initialDelaySeconds: 60
          periodSeconds: 10
          failureThreshold: 3
        readinessProbe:
          httpGet:
            path: /actuator/health/readiness
            port: 8080
          initialDelaySeconds: 30
          periodSeconds: 5
          failureThreshold: 2
        lifecycle:
          preStop:
            exec:
              command: ["sh", "-c", "sleep 10"]
        volumeMounts:
        - name: config
          mountPath: /config
          readOnly: true
        securityContext:
          runAsNonRoot: true
          runAsUser: 1000
          allowPrivilegeEscalation: false
          readOnlyRootFilesystem: true
      terminationGracePeriodSeconds: 40
      volumes:
      - name: config
        configMap:
          name: portfolio-service-config
---
apiVersion: v1
kind: Service
metadata:
  name: portfolio-service
spec:
  selector:
    app: portfolio-service
  ports:
  - protocol: TCP
    port: 80
    targetPort: 8080
---
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: portfolio-service-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: portfolio-service
  minReplicas: 5
  maxReplicas: 50
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 70
---
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: portfolio-service-pdb
spec:
  minAvailable: 3
  selector:
    matchLabels:
      app: portfolio-service

Lessons Learned

✅ Separate liveness and readiness: Prevented restart loops during database outages
✅ Graceful shutdown: Zero dropped requests during 500+ deployments
✅ Resource limits: Eliminated OOMKills and improved bin packing
✅ Structured logging: Cut debugging time from hours to minutes
✅ Pod Disruption Budgets: Maintained availability during node maintenance

Resources

Questions about Kubernetes deployments? Contact me or connect on LinkedIn.

Related Posts:

Building a Mutual Fund Platform: Architecture Deep-Dive
Spring WebFlux: When and Why to Go Reactive
Coming soon: “PostgreSQL Performance Tuning for 1M+ Records”

Kubernetes Best Practices for Spring Boot Microservices

The Context: 100+ Spring Boot Services on GKE

1. Health Checks: More Than Just `/actuator/health`

❌ Common Mistake: Single Health Endpoint

✅ Best Practice: Separate Liveness and Readiness

Custom Health Indicators

2. Resource Management: Requests vs Limits

❌ Common Mistake: No Resource Definitions

✅ Best Practice: Set Requests and Limits

JVM Memory Configuration

3. Graceful Shutdown: Don’t Drop Requests

❌ Common Mistake: Immediate Shutdown

✅ Best Practice: Graceful Shutdown with Drain Period

4. Horizontal Pod Autoscaling (HPA)

✅ CPU-Based Autoscaling

✅ Custom Metrics Autoscaling

5. Configuration Management: ConfigMaps vs Secrets

✅ ConfigMap for Non-Sensitive Data

✅ Secrets for Credentials

✅ Google Secret Manager (Production)

6. Logging: Structured JSON to Stdout

✅ Structured Logging (Logback)

7. Observability: The Three Pillars

✅ Metrics (Prometheus)

✅ Distributed Tracing (OpenTelemetry)

8. Pod Disruption Budgets (PDB)

9. Deployment Strategy: Rolling Updates

10. Security: Non-Root User

Results: Production Metrics

Quick Reference: Complete Deployment YAML

Lessons Learned

Resources

Topics

Share this article

Related Articles

gRPC vs REST: Real-World Performance Comparison

gRPC vs REST: Real-World Performance Comparison at Scale

Real-Time ML Model Monitoring: Detecting Drift Before It Hurts

The Context: 100+ Spring Boot Services on GKE

1. Health Checks: More Than Just /actuator/health

❌ Common Mistake: Single Health Endpoint

✅ Best Practice: Separate Liveness and Readiness

Custom Health Indicators

2. Resource Management: Requests vs Limits

❌ Common Mistake: No Resource Definitions

✅ Best Practice: Set Requests and Limits

JVM Memory Configuration

3. Graceful Shutdown: Don’t Drop Requests

❌ Common Mistake: Immediate Shutdown

✅ Best Practice: Graceful Shutdown with Drain Period

4. Horizontal Pod Autoscaling (HPA)

✅ CPU-Based Autoscaling

✅ Custom Metrics Autoscaling

5. Configuration Management: ConfigMaps vs Secrets

✅ ConfigMap for Non-Sensitive Data

✅ Secrets for Credentials

✅ Google Secret Manager (Production)

6. Logging: Structured JSON to Stdout

✅ Structured Logging (Logback)

7. Observability: The Three Pillars

✅ Metrics (Prometheus)

✅ Distributed Tracing (OpenTelemetry)

8. Pod Disruption Budgets (PDB)

9. Deployment Strategy: Rolling Updates

10. Security: Non-Root User

Results: Production Metrics

Quick Reference: Complete Deployment YAML

Lessons Learned

Resources

Topics

Share this article

Related Articles

gRPC vs REST: Real-World Performance Comparison

gRPC vs REST: Real-World Performance Comparison at Scale

Real-Time ML Model Monitoring: Detecting Drift Before It Hurts

1. Health Checks: More Than Just `/actuator/health`